Open phaer opened 1 year ago
[...] and ran into the problem that --build-on-target stops working as colmena wouldn't copy rekeyed secret.
Oh good point, I haven't actually run into this myself since I usually build on my local machine.
Curios if others here found better workarounds than me?
Copying all secret derivations to your own substituter would also work, but that may or may not be desirable depending on your infrastructure setup.
While doing so, I found that it's hard to know which derivation belongs to which host in the output of nix run .#rekey -- --show-out-paths`. Would a PR for that be welcome?
Certainly, contributions are always welcome! :D
I can think of either extending the .#rekey
CLI to be able to output hostnames, or maybe even better by exposing the rekey derivation in the module directly as a readOnly option age.rekey.drv
or something similar to that. How would your desired interface look like?
UPDATE: The resulting derivation can now be targeted via age.rekey.derivation
.
I think I'm running into a similar issue when deploying from x86_64-linux
to aarch64-linux
with building on the remote.
Is there a workaround for this? I'm not really sure on how to fix that problem
@NyCodeGHG Your issue is probably the following: Your local system must build the derivation containing the rekeyed secrets for your target host, since it's the only system that has access to your secret key. But initiating the true build process on the remote means that the remote doesn't know about your local system at all, so naturally it cannot fetch the rekeyed secrets in any way.
There are several workarounds/solutions to this:
boot.binfmt.emulatedSystems = ["aarch64-linux"];
on your local system and prevent building on the aarch64 machine in the first place. This may actually be faster than letting the remote do the building if the remote is very slow (e.g. RaspberryPi 2/3)nix copy .#nixosConfigurations.yourHost.config.age.rekey.derivation ssh://yourHost
whenever necessary.There may also exist other solutions that I haven't thought about yet.
Also please beware that manual copying can lead to caveats when a rekeyed derivation is rebuild without changing it's hash. This may happen if you choose to use a dummy secret value and later rekeying again to use the true secrets. In that case any existing derivations must be replaced in the store, which agenix-rekey does automatically for your local machine but it can't do that for your target machine since it doesn't know about them. This means in certain rare situations you might need to delete and re-copy the derivation to your remote.
I seem to mostly stick to options 1/2 on my local infrastructure. I'll build locally what I can, and if I need more build power I add the remotes as builders to my local machine.
@phaer @NyCodeGHG There's been a development in this regard, I've introduced a new local storage mode for rekeyed secrets. When using this new local storage mode, all rekeyed secrets will be stored locally in your flake's repository which removes the requirement to build a derivation at all, meaning there should be no more shenanigans with remote builders or CI/CD. You can refer to the Storage Modes in the readme if you want to try this :)
Hello,
Thanks for this interesting project & and your general work in the Nix ecosystem!
I am deploying from
aarch64-darwin
tox86_64-linux
with colmena andforceRekeyOnSystem = "aarch64-darwin"
, and ran into the problem that--build-on-target
stops working as colmena wouldn't copy rekeyed secret.Curios if others here found better workarounds than me? It works if I either use a remote-builder and leave out ´--build-on-target
or if I copy the secrets manually via
nix copy. While doing so, I found that it's hard to know which derivation belongs to which host in the output of
nix run .#rekey -- --show-out-paths`. Would a PR for that be welcome?