Option to add registries

[doorisajar]

It would be convenient if there was a place to specify additional Julia package registries that need to be set up as part of the Dockerfile. Something like a dockerfile_add_registries called here might work. An additional_registries argument, which could just be an array of String, could be used to insert lines that look like this for each additional registry:

"julia -e 'using Pkg;Pkg.Registry.add(RegistrySpec(url = \"https://path.to.some.registry.git\"))'"

I only need one additional registry for my use case, but I can imagine a case where someone might want to add more than one -- say, the NREL registry and a private registry.

At the moment, I generate the Dockerfile and modify it by hand, which does successfully build.

[harris-chris]

Yes, you're right, that would be a good addition and prima facie, should be pretty easy to add. I think your suggestion as to how to do it is correct. I'll try to squeeze it in at some point in the next few days.

[harris-chris]

@doorisajar - just to check, I guess you're using a julia script (rather than a package) to create your lambda function? If a package, then this shouldn't be a problem anyway because the package's Project.toml should point directly to the dependencies used (including ones from external registries). If you're using a script, rather than a package, then the way that Jot handles julia scripts is to convert them to packages, and build them locally before moving that locally-built package to the docker image. So if your goal is just to be able to include packages from external registries in your julia scripts, I don't think we need to alter the Dockerfile, I think I just need to include the external registry in the local package build process. This should also then result in a faster lambda function, because the external dependency can be precompiled and preprocessed with PackageCompiler.jl. However I'd like to make sure I understand your use case correctly before I go down that route.

[doorisajar]

I'm actually using a package. However, the external registry it depends on is private and requires credentials, so that may be the underlying cause of my problem. I can test that out by trying a minimal package that depends on a public external registry and see if that works, I'll see if I can get that done tomorrow.

[harris-chris]

OK. So I guess you've already added the external registry dependencies into the package Project.toml, before even trying to turn it into a lambda function? And the package then works on your local setup, but not when turned into a docker image?

[doorisajar]

Wait, isn't registry information stored in ~/.julia/registries, rather than any specific Project.toml? There's no mention of Project.toml in the Registry docs and the Project.toml & Manifest.toml docs state that no repo-url is included for package dependencies that come from a registry rather than a specified URL.

What happens in my case is that packages from registries other than the General registry aren't found when the Dockerfile is run (either by Jot or manually), because only the General registry is searched (since no other registry is defined in ~/.julia/registries on the image).

[harris-chris]

Yes, your understanding is correct there. I've made these changes now - there's now a registry_urls keyword argument for get_responder, which takes a vector of strings. It would be nice to ultimately let the user pass in RegistrySpec instances rather than urls, since this allows additional registry types - I may try to make that change at some point. Please give the new version a go and let me know if it fixes your situation.

[doorisajar]

Yes, that works! It attempts to add the private registry. It fails due to not having credentials, but I expected that and don't think it's a huge problem -- I can edit the necessary command into the Dockerfile to obtain credentials, which are managed elsewhere.

Allowing users to pass an arbitrary vector of lines to add to the Dockerfile would address that, but I'm not sure if that really makes sense for Jot, and it gets into generalization problems -- how do we know where a user wants to insert each line, etc, etc.

Thanks for adding this!

[harris-chris]

Glad to hear it. Yes, I wondered the same myself, about letting the user enter arbitrary data into the dockerfile. An alternative might be to let the user pass a function which takes the dockerfile as a string (or vector of strings), and returns a new dockerfile string - this would let the user, eg, specify where in the dockerfile they want their additional lines to be added (since this is quite important), or add in conditional content - what do you think?

[doorisajar]

I could see that working well. It would certainly make things much more flexible for users with specialized build environments, for example.

[harris-chris]

OK, I've just added this feature in a new release - it's pretty simple, it just adds a dockerfile_update parameter to create_local_image and create_lambda_components. You can pass a function to this that takes a multi-line string (the original Dockerfile content) and returns a new multi-line string (the updated Dockerfile content). Let me know if it gives you any trouble.

[doorisajar]

Finally worked through some other things that were blocking me from trying this out. It works great! The Dockerfile gets edited exactly as expected.

It might also be useful to have a docker_build_args option to pass through to Docker at image build time. Right now, if someone edits the Dockerfile to expect some arguments, they don't have a way to provide them at build time. A vector of strings that gets turned into --build-arg arg1 --build-arg arg2 in docker build should work fine. I can probably figure out where that goes and make a PR for it pretty easily if you like.

[harris-chris]

Yes, that does sound like a sensible thing to have, and in theory pretty simple. I guess build_args::Vector{Pair} would probably be the logical type for the parameter. There's a get_dockerfile_build_cmd function that gets called from the external-facing create_local_image, and I think all that would be needed would be for that build_args parameter to be added to create_local_image, then passed on to get_dockerfile_build_cmd. If you'd like to try a pull request then that would be appreciated - I can then include it in the tests. If not I think I can add it this coming weekend anyhow.