pietroalbini
commented
2 years ago The code change looks good to me, r=me depending on what you want to do with the message below.
In the future, we will likely also want some tracking/reporting of these failures such that hard errors across all agents lead to alerting the infra team, but this may be best done through Prometheus metrics rather than GitHub "end-user" visible comments (which are likely not the best way to reach infra team members regardless).
To store the error messages (and avoid them being drowned in so many other log messages) we could use Sentry, as it has nice features like deduplication.
In the meantime, could you add a simple counter metric to count how many of those failures we detect (so that we can setup some prometheus alerts)?
Mark-Simulacrum
bors
Previously, any agent reporting an error would cause an experiment to come to a hard stop. This is noisy (leads to a user-visible report) and, at least lately, not actually representative of a terminal failure. Instead, we take the approach of re-queueing any crates assigned to that agent and logging an error on the server host specifying which agent failed.
As the error message is now not typically posted publicly (and no longer needs to be short for being in a GitHub comment), we will also have more opportunity to add extra metadata over time to that message (in future PRs) enabling diagnosing and fixing these errors.
In the future, we will likely also want some tracking/reporting of these failures such that hard errors across all agents lead to alerting the infra team, but this may be best done through Prometheus metrics rather than GitHub "end-user" visible comments (which are likely not the best way to reach infra team members regardless).
r? @pietroalbini