OceanDataTools / openrvdas

An open source data acquisition system designed for use on research vessels and other scientific installations
http://openrvdas.org
Other
39 stars 20 forks source link

Loggers are getting stopped silently after reloading stale config #392

Closed veggiemike closed 3 months ago

veggiemike commented 3 months ago

I saw this happen the other day, but couldn’t figure out what I’d done to trigger it… If you modify just one logger in the yaml config, then push the yellow “Definition is stale. Reload” button, all the loggers are still running and the one you modified hasn’t seen it’s changes yet. If at that point you stop and restart JUST the logger you actually modified, you end up with ONLY that logger running. For some reason, when you stop the one logger, OpenRVDAS stops ALL the loggers but displays them all as still running in the web UI (happy little green buttons)… so that’s bad.

The quick fix is to push the “cruise mode” button (which has turned into an empty label) and select “all-on” to have it start them all back up.

For now, the safe way to refresh just a single logger seems to be: 1) modify the config 2) Reload the stale config 3) open the cruise mode dialog 4) select “all-logging” and “Change Mode”.

And that seems like it can tell that only the one logger really needs to restart, at least I think that's what I'm seeing.

This is happening to me while running v1.9.0 with about a dozen loggers reading UDP streams and writing them all to disk after applying timestamps. On Ubuntu 22.04.

davidpablocohn commented 3 months ago

Hrrmmmmm... As you've observed, that shouldn't be happening - thanks for catching/documenting that!

I'm neck deep in farm stuff tomorrow, but will put on my code snorkel on Friday and try diving into this.

On Tue, May 21, 2024 at 5:00 PM Michael D Labriola @.***> wrote:

I saw this happen the other day, but couldn’t figure out what I’d done to trigger it… If you modify just one logger in the yaml config, then push the yellow “Definition is stale. Reload” button, all the loggers are still running and the one you modified hasn’t seen it’s changes yet. If at that point you stop and restart JUST the logger you actually modified, you end up with ONLY that logger running. For some reason, when you stop the one logger, OpenRVDAS stops ALL the loggers but displays them all as still running in the web UI (happy little green buttons)… so that’s bad.

The quick fix is to push the “cruise mode” button (which has turned into an empty label) and select “all-on” to have it start them all back up.

For now, the safe way to refresh just a single logger seems to be:

  1. modify the config
  2. Reload the stale config
  3. open the cruise mode dialog
  4. select “all-logging” and “Change Mode”.

And that seems like it can tell that only the one logger really needs to restart, at least I think that's what I'm seeing.

This is happening to me while running v1.9.0 with about a dozen loggers reading UDP streams and writing them all to disk after applying timestamps. On Ubuntu 22.04.

— Reply to this email directly, view it on GitHub https://github.com/OceanDataTools/openrvdas/issues/392, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFO7V3XYNDXIZKNJX3T27O3ZDPNYJAVCNFSM6AAAAABICRXHS2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGMYDSMZQG42TINY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

davidpablocohn commented 3 months ago

'k - I can reliably reproduce this and see what's going on. A combination of two errors: naivety and sloppiness in my original implementation.

Error 1: For some reason I reused the simple "load_configuration" api call for reload, which reverts all loggers to their default mode configuration ("off"). After hitting "reload", if you pop up any logger's status box, it'll say it's "off", because the default mode, at least for the sample NBP cruise, is everything off.

Error 2: but after the reload, the logger manager somehow misses the signal that things have changed in the database, and doesn't pick it up until you make another change, such as turning a logger on again. Then it realizes that everything should have (according to the DB) been off, and Makes It So.

Going to tackle these one at a time, first one (the easy one) first.

veggiemike commented 3 months ago

Glad to hear it's reproducible! I was a little worried it would end up being some fluky thing that only happened to me. ;-)

davidpablocohn commented 3 months ago

Fix now pushed to dev. Whew - that took some digging, but exposed an ooooold bug. Thank you for flagging that one!

veggiemike commented 3 months ago

Outstanding! Thanks so much, Pablo!

On Mon, May 27, 2024, 10:31 David Pablo Cohn @.***> wrote:

Fix now pushed to dev. Whew - that took some digging, but exposed an ooooold bug. Thank you for flagging that one!

— Reply to this email directly, view it on GitHub https://github.com/OceanDataTools/openrvdas/issues/392#issuecomment-2133603744, or unsubscribe https://github.com/notifications/unsubscribe-auth/AROPDNDP45OSIGYVCCYDKKDZEM7TZAVCNFSM6AAAAABICRXHS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZTGYYDGNZUGQ . You are receiving this because you authored the thread.Message ID: @.***>

davidpablocohn commented 3 months ago

Of course, please let me know if the problem persists. I'm sure there are more bugs lurking in that ugly bit of django_server_api code.

On Mon, May 27, 2024 at 8:19 AM Michael D Labriola @.***> wrote:

Outstanding! Thanks so much, Pablo!

On Mon, May 27, 2024, 10:31 David Pablo Cohn @.***> wrote:

Fix now pushed to dev. Whew - that took some digging, but exposed an ooooold bug. Thank you for flagging that one!

— Reply to this email directly, view it on GitHub < https://github.com/OceanDataTools/openrvdas/issues/392#issuecomment-2133603744>,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/AROPDNDP45OSIGYVCCYDKKDZEM7TZAVCNFSM6AAAAABICRXHS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZTGYYDGNZUGQ>

. You are receiving this because you authored the thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/OceanDataTools/openrvdas/issues/392#issuecomment-2133684609, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFO7V3UWMJAB5XG2UJFK5FLZENFHDAVCNFSM6AAAAABICRXHS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZTGY4DINRQHE . You are receiving this because you commented.Message ID: @.***>

davidpablocohn commented 3 months ago

Pushed to master as v1.10.1