search

PolicyStat / terrarium

Package and ship relocatable python virtualenvs, like a boss.
https://terrarium.readthedocs.io/en/latest
BSD 3-Clause "New" or "Revised" License
171 stars 14 forks source link

Support anonymous download from S3 when available #24

Open stefanw opened 11 years ago

stefanw commented 11 years ago

My use case for terrarium is to install my environment faster on Travis CI. Compiling e.g. lxml takes ages and the build would sometimes timeout. With terrarium, builds are four times faster!

I can't provide terrarium pre-built envs for Travis since they use a different architecture. That's why Travis has to build the env and upload it to S3 for later use. However, that doesn't work in pull requests because Travis' secure env var (S3_SECRET_KEY) is not available in pull requests. So a normal branch has to build the envs and upload them.

However, pull requests (if they don't change the requirements) should be able to use the existing pre-built envs on S3. The command crashes without a valid S3_SECRET_KEY and I have to manually download the file and trick terrarium into ignoring the S3_BUCKET env var: here is my .travis.yml config.

Terrarium and Travis CI are a good match and they should work better together. I could even see a terrarium travis command that looks at Travis' env vars and figures out the best strategy to get an env.

Points for improvement:

winhamwr commented 11 years ago

With terrarium, builds are four times faster!

We use terrarium for our CI builds on Jenkins, and we get the same kind of speedup. For the PolicyStat application with 100+ requirements, it's closer to 10x for us. I'm happy you were able to see some improvement!

I could even see a terrarium travis command that looks at Travis' env vars and figures out the best strategy to get an env.

That would be awesome! I haven't gotten the chance to use Travis much, myself, but it seems like a very natural fit.

do not crash if S3 auth connection doesn't work

That definitely makes sense in general. Or maybe at least an option to continue if there's a problem.

try getting the S3 env via normal HTTP download

This also makes a lot of sense. No reason to force authentication for S3 content that doesn't require authentication.

winhamwr commented 10 years ago

I worked with @michaelgzoller this morning and I think we got the bulk of this done. Remaining TODOs (which anyone else is highly-encouraged to continue):