Forge module caching

[adrienthebo]

Forge modules should be locally cached after download. This will speed up cases where the same version of a Forge module is deployed to a single environment. This depends on GH-1 to be implemented so that there's better control over how modules are installed.

[jkroepke]

Hi!

do you know the state of this feature?

We are working with feature branches on our company, Deploy an new environment takes a lot of time..

[raphink]

@adrienthebo any news on this feature?

[adamancini]

I am also interested in this feature

[maxmanders]

💯

[maxmanders]

We're also moving to a Puppetfile based approach, mixing Forge modules, and GitHub hosted bespoke modules. We use many environments for development purposes, and the time to download forge modules for each new environment (typically tens per day) is less than ideal. GitHub r10k caching is great, but having something similar for downloaded Forge modules would be great.

[hybby]

:+1: for this feature from me. Any news?

[raphink]

What happened to https://github.com/puppetlabs/r10k/pull/573, which was merged last year? I can't find it in the code! @adrienthebo, @andersonmills ?

[jkroepke]

@raphink Its in the r10k-apibranch which is not merged into master.

[raphink]

@jkroepke yes I saw. And this r10k-api branch seems unmaintained and diverged from master. So my question remains: what happened to to that PR?

[andersonmills]

@raphink We're looking at this issue now. The first question to answer is whether the work to the r10k-api branch can be directly applied. I haven't even looked at the PR, yet, but this is in a set of "up next" work.

[raphink]

@andersonmills I'm afraid the APIs have changed quite a bit since then, but I do hope you're able to merge it.

[andersonmills]

@raphink Clearly the answer about "directly applied" is no, but I'm now looking into reimplementing it.

[jkroepke]

Just an idea for the caching: Work with an cache dir and symlinks, its way more faster then cache and extract the files somewhere.

[andersonmills]

@jkroepke Can you explain that a bit more. I'm not sure I follow.

[jkroepke]

@andersonmills Fine. I'll try it.

Instead download and extract forge module direkt to /etc/puppetlabs/code, r10k sould be save the extracted modules to a cache directory, like /var/cache/r10k/forge/<url of forge>/<full module name>/<version>. On a r10k deploy enviroment, r10k has to create symlinks in /etc/puppetlabs/code/environment/<environment>/modules/ for example: /etc/puppetlabs/code/environment/production/modules/puppetlabs-stdlib/ -> /var/cache/r10k/forge/forge.puppet.com/puppetlabs-stdlib/4.14.0/

r10k only needs to download the module, if the directory in the cache is not present. This implemenation should be really, really fast.

[raphink]

Yes, that sounds like a good idea @jkroepke

[andersonmills]

@jkroepke Ah, yes, that is similar to what we did for the r10k-api version, and along the same lines as what we are thinking this. There are reasons that simlinks into the live code directory won't work for us, but copying local files on disk from the cache is fast enough compared to the download that shouldn't be a big hit.

[jkroepke]

@andersonmills: Its possible to make it configurable? Let's choose what the user want.

[jarro2783]

Symlinks wouldn't work for us either, because we run r10k as a separate build step and package the result to be deployed. I would be very interested in seeing this; at the moment I'm considering switching all of our modules to git because forge modules are so slow.

[andersonmills]

Forge module caching has been enabled with https://github.com/puppetlabs/r10k/pull/729.

This just caches the downloaded tarball and the md5 of it locally. Because of issues around symlinking, that solution was not implemented.

[hybby]

Excellent, thank you so much @andersonmills! When is the next release of r10k anticipated?

[andersonmills]

The master branch is going to need to be a 3.0.0 release because of some breaking changes, unfortunately, and we have a couple of other things that need to get in, as they are breaking changes, too. Unfortunately, it's therefore hard to say. :/

[github-actions[bot]]

This issue has been marked stale because it has had no activity for 60 days. The Puppet Team is actively prioritizing existing bugs and new features, if this issue is still important to you please comment and we will add this to our backlog to complete. Otherwise, it will be closed in 7 days.