search

tazjin / nixery

Container registry which transparently builds images using the Nix package manager. Canonical repository is https://cs.tvl.fyi/depot/-/tree/tools/nixery
https://nixery.dev/
Apache License 2.0
1.8k stars 67 forks source link

nixery.dev registry is down #138

Closed mkmik closed 2 years ago

mkmik commented 2 years ago

FYI:

$ docker pull nixery.dev/shell
Using default tag: latest
Error response from daemon: received unexpected HTTP status: 500 Internal Server Error
tazjin commented 2 years ago

There seem to be some errors after the last channel bump. Nixery's logs are full of Nix evaluation errors. Taking a look.

tazjin commented 2 years ago 
<tazjin> [nix] while evaluating anonymous function at /nix/store/lz3s922vzs2fvb16p550w3wpga409kzm-source/third_party/overlays/emacs.nix:2:1, called from /nix/store/lz3s922vzs2fvb16p550w3wpga409kzm-source/nix/readTree/default.nix:67:10:
<tazjin> [nix] creating directory '//.cache': Read-only file system

This was the error previously causing the failures, after moving to the latest bump things seem to work fine. I suspect that a recent nixpkgs bump caused some incompatibility with this particular overlay.

Notably, monitoring detected this but didn't alert - going to take a look at that.

mkmik commented 2 years ago

@tazjin

This was fixed but now it (or something similar) happened again:

  Warning  Failed     21s (x2 over 77s)  kubelet            Failed to pull image "nixery.dev/shell/findutils/gnugrep/gnutar/hexdump/less/mount/procps/curl/grpcurl/binutils/gdb/strace/rustc/linuxpackages.perf/perf-tools/gzip/zstd": rpc error: code = Unknown desc = Error response from daemon: received unexpected HTTP status: 500 Internal Server Error
tazjin commented 2 years ago

It's possible that you ran a command in the middle of a restart (which only take a few seconds); with the setup we have right now the number of restarts is unfortunately pretty high.

We have a tracking bug for this in the TVL infrastructure and I'll attempt to prioritise it soon.

mkmik commented 2 years ago

Thanks

tazjin commented 2 years ago

Quick update: As of cl/4064 the frequency of config changes for Nixery is going to be reduced significantly, which should make occurrences of this much less frequent.

Do note however that nixery.dev is provided without any SLO at the moment.

mkmik commented 2 years ago

Do note however that nixery.dev is provided without any SLO at the moment.

Thanks for running it!

(I have it in my backlog to self-host it somewhere, call me lazy but I'm just waiting until the self-hosting docs are more fleshed out :-) )

tazjin commented 2 years ago

Do you use NixOS? At TVL we have a module for it which you might be able to adapt pretty easily: https://cs.tvl.fyi/depot/-/blob/ops/modules/nixery.nix

mkmik commented 2 years ago

No I don't use nixos yet; always wanted to find a good excuse to play with it; I guess I just found one! :-)