Open davepacheco opened 2 years ago
I've only been in code adjacent to this recently, but I suspect logs like this:
[2022-06-01T23:09:06.269248009Z] WARN: SledAgent/StorageManager/2909 on sock: failed to notify nexus about datasets, will retry in 528.435574ms
[2022-06-01T23:09:06.269491647Z] WARN: SledAgent/StorageManager/2909 on sock: failed to notify nexus about datasets, will retry in 262.229898ms
are two independent dataset retry loops running, not one with a weirdly-decreasing interval. Big :+1: on including more context here to clarify, though!
While setting up the stack a few weeks ago, I found the Sled Agent logging pretty confusing about the current state of things.
The first thing to know is that Sled Agent startup makes calls out to Nexus, but the first time it starts up, Nexus isn't running yet. Eventually, it will deploy the first Nexus instance.
I'll include the full log here for reference:
What I found most confusing are those messages at the end:
As I looked at this, it was five minutes later:
I think there are a few problems here:
This might be a bug in the behavior, not just the log. If not, maybe we could be more clear about what's being retried?