Loading and synchronizing Runners failed! Retrying...

[sentry-io[bot]]

Sentry Issue: POSEIDON-4R

Loading and synchronizing Runners failed! Retrying...

[mpass99]

There are two types of errors grouped in the Sentry Issue:

• Loading Environments failed
  • During the loading of the environments before synchronizing the runners
  • Mostly with the underlying error dial tcp [ip]:4646: connect: connection refused
  • We might check if this error correlates with reboots of Nomad Servers
  • We might check if our DNS-Fail-Over works
• Loading and synchronizing Runners failed
  • Also the connection refused error from above
  • But more prominent is an unexpected EOF error.
  • We should investigate why Nomad aborted the connection. Deployment? Restart? ...

We might separate the two types into distinct Sentry issues 🤷

[MrSerth]

I didn't notice there are two issues grouped together; we should probably use the new approach introduced in #643 to "ungroup" them.

That having said, I agree that some might be infrastructure-related (DNS) and others might be caused by unattended-upgrades. Specifically:

• Loading and synchronizing Runners failed! Retrying...
  • Only occurs during the night, indicating it could be related to unattended-ugprades
• Loading Environments failed
  • Those errors are usually grouped, with many events occurring at a similar time (and then no event at all for a longer period)
  • The events occur during regular use (not during the night)
  • I didn't found any link to a specific cause; we might need to dig deeper into the logs for this one (was CodeOcean triggering to synchronize an environment, or is it caused by a deployment / restart)?

[mpass99]

Loading Environments failed - was CodeOcean triggering to synchronize an environment, or is it caused by a deployment / restart

In general, we can differentiate the two errors as follows:

• Loading and synchronizing Runners failed! Retrying...
  • Failure of the Nomad Event Stream connection
  • Mostly short-term errors, e.g. deployment/restarts
• Loading Environments failed
  • Failure to connect to Nomad i.e. loading jobs
  • Mostly infrastructure issues
  • Checking the last occurrences (12th June), the error is highly accompanied by other errors, e.g. from the influxdbclient not being able to connect to the InfluxDB server.

[MrSerth]

This makes sense, thank you! Ideally, we copy this over to Sentry (when the issue re-appears there) to find this analysis immediately.

[sentry-io[bot]]

Sentry Issue for Loading and synchronizing Runners failed!: POSEIDON-71