Open davepacheco opened 2 months ago
The Unknown diesel error creating VpcRouter called \\"system\\
error is coming from here:
https://github.com/oxidecomputer/omicron/blob/e514bc8262f404f1d28b7b5e3ef39712dccb81f2/nexus/db-queries/src/db/error.rs#L248
and the cause is a NotFound
error that's raised when we use something like get_result
to execute the query, which we are doing here:
https://github.com/oxidecomputer/omicron/blob/e514bc8262f404f1d28b7b5e3ef39712dccb81f2/nexus/db-queries/src/db/datastore/vpc.rs#L1090
The stack is:
PopulateBuiltinVpcs
's implementation calls datastore.load_builtin_vpcs(opctx)
self.vpc_create_router()
.get_result()
.It's a little surprising for an INSERT
to return NotFound
. But we are expecting a row because we're using get_result_async()
. And we're expecting that row because we're using INSERT ... RETURNING ...
, which returns the newly-inserted row. That we got 0 rows here tells us that we didn't insert it. That can only happen for this query if there was another row with the same id. This is a totally normal and expected condition if some other Nexus already went through the populate step. Now, we already have tests for populators being idempotent, so my guess is that normally if this has already been inserted, then we won't make it this far into load_builtin_vpcs()
, and what actually happened here was a race. I'll look for more evidence of that.
Yeah, I think we have Nexus instances racing. Here's the point in three different Nexus logs where we started this populator:
The first of these Nexus instances went on to the next populate step, so it must have succeeded. The second and third ones both gave up.
The timeline is:
2024-06-28T22:10:18.480Z Nexus 1: "attempting to create built-in VPCs"
2024-06-28T22:10:18.504Z Nexus 2: "attempting to create built-in VPCs"
2024-06-28T22:10:18.510Z Nexus 3: "attempting to create built-in VPCs"
2024-06-28T22:10:18.584Z Nexus 2 reports the "not found" error
2024-06-28T22:10:18.588Z Nexus 3 reports the "not found" error
2024-06-28T22:10:18.664Z Nexus 1: "attempting to create built-in VPC firewall rules", which implies it had finished creating the built-in VPCs
This is not a smoking gun (I don't think that's possible without more fine-grained logging from the VPC populate steps) but it's consistent with the explanation that these raced with each other.
In this case, prior to #5962, the impact would be:
I think there are three potential takeaways here:
RETURNING
, just always fetch the row (which we may or may not have just inserted). We might also be able to do something like INSERT INTO ... ON CONFLICT SET id = id RETURNING *
so that the row technically gets updated by the INSERT (even though the data doesn't actually change). I think we might do this in some other places.I found another example where we do the ON CONFLICT SET id = id
for this exact reason (I think! There's not much explanation in the comment.):
https://github.com/oxidecomputer/omicron/blob/653964be121425c55955a6e4f58d9e1b8cda83da/nexus/db-queries/src/db/datastore/rack.rs#L236-L237
Here's the CI failure: https://github.com/oxidecomputer/omicron/pull/5962/checks?check_run_id=26826135569
This PR is non-trivial and it's conceivable that my changes introduced this but I don't yet see how. The helios-deploy job failed in the usual way (timing out after 600s trying to log in):
That TLS error can reflect that there are no certificates, and so a problem with the "external endpoints" task. I went to the Nexus logs to look for errors and found that one Nexus failed some of the first-time-setup steps:
One possibility here is that before my PR, we would still have run the background tasks that eventually configure TLS, whereas now we won't. But at best the old code would have been papering over this problem.