search

DBCDK / morph

NixOS deployment tool
MIT License
839 stars 63 forks source link

Issues with repairing broken path in server #191

Open arximboldi opened 2 years ago

arximboldi commented 2 years ago

Hi!

Something very strange has happened in one of my deployments. Even though I did not get any errors when using morph deploy switch, in the server, one of the path /nix/store/<asdasd>-some-package/... (a large one) is missing quite a few packages.

Running nix-store --verify --check-contents on the server shows that this in an issue.

However running nix-store --repair-path is not working, because the server I guess does not know how to redownload the package.

What is the recommended way to solve this situation? I am tempted to solve it manually but that's messy bussiness. I am looking for an option in morph itself to repair or force push a path but I can't find it...

arximboldi commented 2 years ago

Hmmm, worked around it by running:

sudo nix-store --delete --ignore-liveness /nix/store/...faulty-path...
srhb commented 2 years ago

That is a good way to do it. You should try and verify that the disk in question is healthy -- morph exclusively uses nix(-copy-closure) to transfer store paths, and nix should not break store paths en-route to registration.

arximboldi commented 2 years ago

I think it may have happened because of running deploy switch locally while CI was running deploy dry-activate simultaneously. Could that be the case?

srhb commented 2 years ago

It absolutely should not be the case. Everything goes through the nix-daemon (on a normal nixos system) and any number of clients should be able to realise paths at the same time. If you're able to reproduce corrupting store paths like this and your hardware is fine, that sounds like a quite serious bug (but one I have not witnessed.)

arximboldi commented 2 years ago

Hmmm, interesting. Really not sure what happened. The dry-activate on Github Actions seemed to have hanged, while a local deployed succeeded. Maybe the dry-activate never got to finish, maybe it hanged mid submission of that path. I'll keep my eyes open in case it happens again with a more clarity about how.

arximboldi commented 2 years ago

And thank you very much for the quick responses!