search

spring-attic / hdfs

Apache License 2.0
5 stars 6 forks source link

hdfs-dataset and graceful shutdown #3

Open bakabo opened 7 years ago

bakabo commented 7 years ago

Issue: There is a logical difference between SIGTERM and the /shutdown endpoint. But doesn't appear to be a difference between SIGKILL vs SIGTERM. Attached are logs that show a scenario with 2 different kill sequences.

Reproducable steps. Scenario 1:

Start the hdfs-datasets-sink Send a message to cache in the MessageStore Send /shutdown request Scenario 2:

Start the hdfs-datasets-sink Send a message to cache in the MessageStore kill -15 PID The first scenario will gracefully shutdown. Meaning the data in the MessageStore will be flushed to disk. The second scenario will not gracefully shutdown. The hook to HDFS will be closed before the MessageStore can flush the cache, resulting in an exception being thrown.

"Caused by: java.io.IOException: Filesystem closed" Configurations. hdfs.dataset: fs-uri: hdfs://localhost batch-size: 3 idle-timeout: 10000 format: parquet

endpoints.shutdown: sensitive: false enabled: true

Versions: org.springframework.cloud.stream.app:spring-cloud-starter-stream-sink-hdfs-dataset:1.1.1.BUILD-SNAPSHOT org.springframework.integration:spring-integration-kafka:2.1.0.RELEASE

org.springframework.cloud:spring-cloud-stream org.springframework.cloud:spring-cloud-stream-schema org.springframework.cloud:spring-cloud-stream-binder-kafka org.springframework.cloud:spring-cloud-starter-consul-all

mavenBom 'org.springframework.cloud:spring-cloud-dependencies:Camden.SR5'

sink_kill_sigterm.txt sink_shutdown_endpoint_graceful.txt

trisberg commented 7 years ago

Have you tried setting spring.hadoop.config.fs.automatic.close to false? That should prevent the HDFS JVM shutdown hook from executing.

bakabo commented 7 years ago

That did the trick. Thank you very much.

bakabo commented 7 years ago

Has it already been discussed on whether to make that setting a default?

trisberg commented 7 years ago

We might want to do that. Have to check if the message store relies on a shutdown hook where we don't control the order of the execution of the hooks. I recall this being an issue in a different scenario, @jvalkeal might remember more. I'll reopen this for now so we can decide whether to make this a default config or add it to the docs.

PedroAlvarado commented 7 years ago

Allow me to second setting spring.hadoop.config.fs.automatic.close=false as default to at least have consistency relative to /shutdown and mitigate in-flight data loss.

jvalkeal commented 7 years ago

I think message store will just go down with app context. There's been numerous tweaks in Spring INT over the years about its own lifecycle what comes for order of start/stop.

I would not change spring.hadoop.config.fs.automatic.close from defaults(well maybe for this component) in a lower level libs as not closing automatically will potentially create memory leaks. I think automatic close did something else as well but jvm hook from it is a nasty one.