search

digitalocean / firebolt

Golang framework for streaming ETL, observability data pipeline, and event processing apps
Other
698 stars 50 forks source link

App can't be be normaly sopped if source wait to reload #22

Open yellow-sky opened 3 years ago

yellow-sky commented 3 years ago
  1. Source can't connect to kafka\other resource - executor wait 10 sedonds (https://github.com/digitalocean/firebolt/blob/223d445912e74ffe1d039584a89df03a222ae954/executor/executor.go#L264)
  2. During this period of time, a shutdown message is received by app (https://github.com/digitalocean/firebolt/blob/223d445912e74ffe1d039584a89df03a222ae954/executor/executor.go#L271)
  3. Executor try to shutdown source, but old instance (https://github.com/digitalocean/firebolt/blob/223d445912e74ffe1d039584a89df03a222ae954/executor/executor.go#L274)
  4. Executor continues to execute an infinite loop (https://github.com/digitalocean/firebolt/blob/223d445912e74ffe1d039584a89df03a222ae954/executor/executor.go#L250)
  5. New source instance doesn't know anything about studown signal
  6. Go to step 4 => App does not respond to system shutdown signals
jnadler commented 3 years ago

Thanks for the bug report @yellow-sky - I can see how this is a problem if a firebolt app starts while kafka is unavailable. I'll work on fixing this when I can, or if you need it fixed quickly we'd be happy to consider a pull request!