Build .hex/cache.ets file offline for secure build applications

[sameluch]

Hello,

I am running into a build issue using Hex where I get the following error messages, even when all .tar dependencies are present in .hex/packages/hexpm:

(Mix) Unknown package << package name >> in lockfile (Mix) No package with name << package name >> (from: mix.exs) in registry

Due to secure build requirements, I must build with no network access. This appears to block the generation of the lockfile/registry file in question, cache.ets, when get.deps is called. Is there a way to generate the cache.ets in the correct format after loading up the .hex/packages/hexpm directory with the correct tar files while offline?

I was able to find that this cache file is created/modified by hex/lib/hex/registry/server.ex when a response is received from the remote repository to cache the hexpm package and version data, but this did not seem replicable without a network connection.

[ericmj]

You will generate the cache.ets when you fetch the .tar files. Can you use that?

[sameluch]

Unfortunately no. The .tar files are not retrieved via hex itself in this scenario as I have no outside internet access to reach hexpm. Instead the are hydrated via another source tarball and placed into the correct directory when building a package.

I'm looking for a way for hex to pick up and recognize these as cached versions of the dependencies (i.e. present in the cache.ets file) without the need to hit the hexpm repo to generate the cached package data.

[ericmj]

That's not currently possible and would require quite a bit of rework to make possible since we currently do the full dependency resolution even in offline mode and we cannot solve dependencies without the registry information in cache.ets. Contributions to make this work would of course be accepted.

In the mean time I would suggest to include cache.ets in the source tarball you use for building packages.

[sameluch]

Okay, thanks for looking into this. If I come up with something that isn't super hacky I'll be sure to let you know!

Is there specific template that the entries in the ETS follow? it may be possible to fudge the entries if there is a particular format it follows.

[ericmj]

These are all the keys: https://github.com/hexpm/hex/blob/main/lib/hex/registry/server.ex#L283-L290 and in this function you can find all the values we write: https://github.com/hexpm/hex/blob/main/lib/hex/registry/server.ex#L348-L382.

There's no guaranteed backwards compatibility but it's pretty unlikely we will change existing keys. If you also lock down the hex version it should be safe to use.

[ericmj]

We are still interested in this but closing because the issue is stale.