search

kindredgroup / puppet-forge-server

Private Puppet forge server supports local files and both v1 and v3 API proxies
69 stars 44 forks source link

Feature: Add cache to significantly speedup fetching from remote proxies #35

Closed cardil closed 8 years ago

cardil commented 8 years ago

I've added a cache that works in memory to significantly speedup subsequent remote fetches.

cardil commented 8 years ago

A bit strange. It works on my Ruby 1.9. I've double checked it.

Please restart Travis job, it may help.

asciicast

i11 commented 8 years ago

Hi,

Sorry for such late response. I got out of the loop totally. Travis job still fails with the same thing. Not sure why. I'll be testing a bit more today.

i11 commented 8 years ago

Btw, I saw you had a few more commits after this PR. Feel free to re-submit it.

i11 commented 8 years ago

I wonder if it could be the bundle issue. 1.7.x seems like a pretty old version. Could you please try updating .travis.yml with:

before_install:
  - gem update bundler
i11 commented 8 years ago

Just tried and it does the trick.

cardil commented 8 years ago

I would like it work without need for upgrading bundler.

before_install:
  - gem update bundler

I will look at it when I have a bit of time

i11 commented 8 years ago

Oh and thanks a lot! This is great stuff.

i11 commented 8 years ago

Sure thing

i11 commented 8 years ago

I'm merging this. I'll add the gem update command for now as it doesn't go through for all PRs now. Please feel very welcome to look into it when you have time.

i11 commented 8 years ago

Quick though. It would be nice to have an endpoint for invalidating the cache.

i11 commented 8 years ago

Now when I think about it caching all requests has heavy negative effects. For example after the module deployment TTL will be in the way for retrieving it. puppet module install will do /v3/releases?module=<module name> and it will be most probably already cached, so it won't find the new version. Hence search and release requests should have very short ttl or not cached at all. Already there things start to get complicated and squid or varnish feels much better choice.

i11 commented 8 years ago

Module files on the other hand most probably will be sitting in RAM anyway as OS file cache, so benefits of explicit caching kind of disappears...

cardil commented 8 years ago

Endpoint for cache invalidation would be nice.

About caching: I have done in with two levels. First exist in ram and last by default 15min. So after release cached time is max 15min. I think it is acceptable. If you don't like this you have option to completely turn the cache down by settings either ttl or size to 0. Try to execute librarian-puppet update couple times with and without ram cache. The difference is huge. The bigger your Puppetfile gets the more benefits you would have from this cache.

And there is cache level 2 for requests that can't change. That is module tar.gz and meta information about specific version. I also ecurage to test with ram cache turned off and cleared persistent cache and not. In my case fetching single meta information about specific version takes about a second. I have about 30 dependencies in my Puppetfile so the difference for my is about 30 seconds. That is huge because I run it on every provisioning of Puppet master with vagrant. That's annoying, that slow down when run 30 times a day.

Better to give options to all of this so users can adjust puppet-forge-server to its needs.

i11 commented 8 years ago

I fully understand yours and probably others need for caching. My point was that I'm not sure it should be part of this project. Varnish or squid for example could provide all possible bells and whistles for whatever the needs are in that area. I don't think we should be reinventing the wheel here.

Anyway we took the pill already, let it be. But let's not extend it anymore. Also I'll be setting the default cache ttl to 0 in the next release, so that it would be more explicit.

cardil commented 8 years ago

That seams completely reasonable.