pip cache support

[narkisr]

It will be awesome to support python caching as well

[fgrehm]

@narkisr yup :) u know where they get stored?

[narkisr]

It looks like its configurable:

http://stackoverflow.com/questions/10336308/how-to-cache-downloaded-pip-packages

Couldn't find the default value tough

Ronen

[fgrehm]

@narkisr I'm not a phyton dev so I'd love some help here :) would u mind trying out this approach to see if it works? you can manually share the cache folder with the guest as we'd do with this plugin and check if things work fine. if u need any help just LMK

[narkisr]

Im not a python dev either ;) (more of a ruby and Clojure guy), ill give it a shoot and let you know if it works

[thedrow]

You summoned me master? @fgrehm

[fgrehm]

@thedrow not sure if I got it right, but are you signing up to handle this one? if you are, be my guest :)

[thedrow]

@fgrehm I might if you tell me what exactly needs to be done. I'm no ruby dev so I can't write code but I can direct you exactly where you should be headed.

[fgrehm]

@thedrow we need to automatically detect pip places downloaded tarballs or, if not possible, how if there is a way to configure it to use a cache. based on this post on SO we should be able to set PIP_DOWNLOAD_CACHE but according to this comment it is "seriously flawed". after we know how to set things up, making it work from within vagrant-cachier should be dead easy :)

[narkisr]

In my setting PIP_DOWNLOAD_CACHE did not seem to work,

[thedrow]

@fgrehm Yup, it is seriously flawed. You should be using Basket instead which makes caching much easier and safer.

[thedrow]

@fgrehm Ok Basket doesn't fit here since it has no feature that enables you to cache a specific version of a dependency. This should be added or an alternative should be found before implementing this feature request,

[fgrehm]

@thedrow oh well, just to check, is PIP_DOWNLOAD_CACHE usable by any chance or it doesn't work at all?

[thedrow]

@fgrehm PIP_DOWNLOAD_CACHE is usable for pip, not you. You cannot control what happens there. You might want to wait until Wheel is stable. Wheel is the new packaging format for python and it will be available to all using the next version of pip. One of pip's features is to download and store Wheels in order to use them later. See here for usage. Wheel can certainly be controlled by you in order to cache packages for further use. This is the best solution I've got for you but you have to wait until pip 1.4 & setuptools 0.8 are released.

[fgrehm]

@thedrow tks for the information! I'll mark this as blocked while we don't have a way to make it work

[thedrow]

Quick update. Setuptools is now at version 0.9.x which is really good. Python packaging has been a mess in the last few months. The next pip release (1.4) should be out this week which means that we will be able to use wheel. That means that this issue won't be blocked.

[thedrow]

@fgrehm Ping? This issue shouldn't be blocked anymore.

[patcon]

Rockin. Removed label :)

[fgrehm]

Awesome!

@thedrow would you be able to send a PR for this? if not, how about a detailed step by step on how to set things up using plain old bash script? if you are up for doing the script please LMK and I'll try to write you a short high level overview on what we need to get this implemented. thanks in advance :)

[thedrow]

I have no ruby knowledge almost at all. I can provide a step by step guide to how to do this only that it will require some research as I haven't used wheel yet.

[fgrehm]

@thedrow here's some information I just wrote and thought would be nice to have available on the readme https://gist.github.com/fgrehm/fd3907dafde6aa7272b9

@patcon @cromulus @tmatilai @mindreframer @laggyluke @fh sorry for the spam but if any of you could review it and lemme know if that is a good explanation before I push it to master it would be awesome :)

[patcon]

No worries. Looks like a pretty sound explanation. The first paragraph is a little dense, but not sure if it can be made any clearer outside of providing a diagram :)

[gmiroshnykov]

That explanation looks good for me too! There's a small typo though: "if it is has". Also, "VMs rebuilds" doesn't sound right - maybe it should be "VM's rebuilds" or "VM rebuilds". Disclaimer: English is not my native language :)

[thedrow]

@fgrehm Yeh that works. I'll try to come up with something on Friday.

[fgrehm]

Thanks for the feedback everyone :) I've updated the gist accordingly: https://gist.github.com/fgrehm/fd3907dafde6aa7272b9/revisions

[fgrehm]

We now have docs for creating new buckets under http://fgrehm.viewdocs.io/vagrant-cachier/development

If someone is up for implementing this, please shoot us a PR :-)

[egasimus]

Has anyone gotten around to building a bucket for Pip? I sorely need one, but don't have the time to put it together with my rudimentary Ruby skills.

[fgrehm]

@egasimus if you know where pip keeps its cache we might be able to help, please see the "How to create a new bucket?" section of http://fgrehm.viewdocs.io/vagrant-cachier/development for the information we need.

[esmail]

It's configurable with the environment variable PIP_DOWNLOAD_CACHE since way back in pip 0.1.4 per the release notes.

[fgrehm]

@esmail there's some discussion around this at GH-80