search

CrazyChaoz / gradle-dot-nix

This flake can generate the full maven repo required to build a gradle app from gradle/verification-metadata.xml, all in the sandbox, without code generation.
MIT License
4 stars 2 forks source link

[Proposal] Adding possibility to use and authenticate to private repositories #1

Open eeedean opened 6 months ago

eeedean commented 6 months ago

Hi,

thanks for the effort of creating this tooling. It looks very promising!

What lacks to me is support for traditional enterprise infrastructure. Enterprise Infrastructure almost always requires usage of private repositories to be used in dependency resolution.

Therefore, I would appreciate:

CrazyChaoz commented 6 months ago

Ah, yes, sure. I totally forgot i just added the defaults and never made them configurable.

For the repo fetching it is currently just a Python

requests.get("https//repo-url/path/to/package", stream=True)

and I don't have a protected repo at hand to check how the requests for that look like exactly.

If you could provide me a sample request i can probably implement something rather quickly.

eeedean commented 6 months ago

For JFrog Artifactory it may look like that: 

curl -H "Authorization: Bearer $ARTIFACTORY_IDENTITY_TOKEN" -XGET https://jfrog.my-private.repo/artifactory/public/com/fasterxml/jackson/jackson-parent/2.13/jackson-parent-2.13.pom

How that does translate to Pythons API, I cannot tell unfortunately. However using an Authorization Header with a secret token is rather standard.

For secret management a user of your tooling might already use something like agenix. However, I'm rather new to the playing field, too. So maybe you or someone has a good idea, how to tackle the secret problem in this particular situation.

CrazyChaoz commented 6 months ago

Implementing that functionality shouldn't take too long, but I currently have 0 clue how to properly implement it without the credential leaking to the nix store and/or the git repo.

eeedean commented 6 months ago

I feel like, its feasible to leverage age Therefore encrypting a secret by a call to age -o my.secret.age -R $HOME/.ssh/id_ed25519.pub my.secret and decrypting it while build time with age -i $HOME/.ssh/id_ed25519 -d my.secret.age.

I'm not very experienced in using secret management in Nix-Builds, however, that's how I would go about it: Putting the secrets encrypted into the Git and encrypting them based on some static key or something.

I don't know, if it's feasible to use environment variables (having CI/CD in mind, which is usually configured by env), since there was some experimental NixOS-Feature configurable-impure-env, which might come handy into play.

mschwaig commented 6 months ago

I feel like, its feasible to leverage age Therefore encrypting a secret by a call to age -o my.secret.age -r $HOME/.ssh/id_ed25519.pub my.secret and decrypting it while build time with age -i $HOME/.ssh/id_ed25519 -d my.secret.age.

I'm not very experienced in using secret management in Nix-Builds, however, that's how I would go about it: Putting the secrets encrypted into the Git and encrypting them based on some static key or something.

I don't know, if it's feasible to use environment variables (having CI/CD in mind, which is usually configured by env), since there was some experimental NixOS-Feature configurable-impure-env, which might come handy into play.

I think you should avoid having secrets in the store at all, even if they are encrypted, because I don't like the idea of having to think that hard about the impact of leaked keys and secret revocation, but some people disagree and maybe there are situations where it makes sense.

That experimental feature looks like it could be a good solution for passing in an auth token. I will have to try this out myself at some point, since I have not had the chance yet. Another approach that is a bit worse from a technical and usability point of view, but might work with reasonable effort, is using the extra-sandbox-paths option to pass in a file with the token.

The things that you find online about overriding fetchurl with fetchurlBoot (https://nixos.wiki/wiki/Enterprise) are not a good fit, because the python script this project uses to fetch dependencies is basically a completely custom fetcher, and I don't know if you can adapt that fetcher do whatever fetchurlBoot does.

There are also a lot of dedicated projects that do secret management for Nix, and maybe one of those can help, but I have not looked into any of those myself yet.

EDIT: If you get a secret into the sandbox by encrypting it and adding it to the store, now you need to get the decryption key into the sandbox.

eeedean commented 6 months ago

EDIT: If you get a secret into the sandbox by encrypting it and adding it to the store, now you need to get the decryption key into the sandbox.

That may actually be the case due to sandbox limitations, that's right! I took the thought not far enough, as my only experience is from my personal system configuration. Thanks a lot for your input!

So far, I feel like supplying environment variables to the nix-daemon may be the way to go?

CrazyChaoz commented 6 months ago

To me it looks like --extra-sandbox-paths gives us exactly the behavior we want in regards to secrets.

Now the only question is how we want to structure the file(s)

There is the possibility of someone fetching via bearer token, but I have also seen here that a common way of setting authentication in maven is via username/password, and then there are other options described in the Maven docs that include other SCP settings, like public/private key based authentication.

Since we are currently not using the native mvn command, we would have to replicate that behaviour aswell, or just use mvn

Using mvn would give us the ability to specify all these parameters in a settings.xml, as most configs would normally, even with custom http params

It could maybe even simplify the whole fetcher code.

My current, potentially strange idea for the implementation would be: