neonvm-controller: improve node failure reaction speed

[Omrigan]

Multiple commits, their descriptions:

• neonvm: recreate the pod for VM even if the older pod still exists

  We have previously assumed running multiple pods for one VM is dangerous.

  We now believe it should be fine, even if the old pod still tries to continue operating. What would happen is that the new pod would take over the safekeepers quorum, leaving the old one to disconnect and shutdown.

• neonvm: check if deletion timestamp is in the past

  If we are passed the mark of the deletion timestamp, it means the deletion is stuck, and we should consider the pod failed anyway.

  Possible reasons for this are:

  1. Node is down.
  2. Pod is stuck pulling the image from the container registry.

• neonvm: add explicit "node not ready" tolerations with 30s grace period

  By default they are 300s (5m), which is way too long.

Part of the https://github.com/neondatabase/cloud/issues/14114

[sharnoff]

@Omrigan IIUC this PR needs rebasing?

[Omrigan]

2. If you plan to rebase-and-merge, could you edit the commit titles to add the PR number before merging? (i.e. appending `(#1055)` so it looks similar to squash-and-merge)

I was thinking maybe it we don't really need it?

From the history, if you click on commit, then the commit will have a PR link.

So editing the commits saves us 1 click.

[sharnoff]

I was thinking maybe it we don't really need it?

From the history, if you click on commit, then the commit will have a PR link.

It's very useful to have that information available when interacting with git locally

[github-actions[bot]]

Merging this branch will increase overall coverage

Impacted Packages	Coverage Δ	:robot:
github.com/neondatabase/autoscaling/neonvm-controller/cmd	0.00% (ø)
github.com/neondatabase/autoscaling/pkg/neonvm/controllers	11.79% (+0.28%)	:thumbsup:
github.com/neondatabase/autoscaling/pkg/neonvm/controllers/functests	0.00% (ø)

---

Coverage by file

### Changed files (no unit tests) | Changed File | Coverage Δ | Total | Covered | Missed | :robot: | |--------------|------------|-------|---------|--------|---------| | github.com/neondatabase/autoscaling/neonvm-controller/cmd/main.go | 0.00% (ø) | 132 (+2) | 0 | 132 (+2) | | | github.com/neondatabase/autoscaling/pkg/neonvm/controllers/config.go | 0.00% (ø) | 0 | 0 | 0 | | | github.com/neondatabase/autoscaling/pkg/neonvm/controllers/vm_controller.go | 25.87% (**+0.64%**) | 661 (-1) | 171 (+4) | 490 (-5) | :thumbsup: | _Please note that the "Total", "Covered", and "Missed" counts above refer to ***code statements*** instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code._ ### Changed unit test files - github.com/neondatabase/autoscaling/pkg/neonvm/controllers/functests/vm_controller_test.go - github.com/neondatabase/autoscaling/pkg/neonvm/controllers/vm_controller_unit_test.go

HTML Report

Click to open