initial stab at packaging using ship_it

EliAndrewC commented 10 years ago

So this is not yet ready to merge. The good news is that I'm able to make an RPM with this changeset which seems to install properly. The bad news is that now that we've changed Sideboard to use a single virtualenv, we are no longer able to bundle plugin Python dependencies with those plugins as easily :/

So basically, we need to be able to have packages somehow insert their dependencies into Sideboard's virtualenv on install. We do NOT want to pull down from the internet when we do this. Perhaps the solution is to include the packages and then do the equivalent of python setup.py install on each of them to make sure they're installed, but I'll need to give that some thought.

EliAndrewC commented 10 years ago

I'll probably reach out for guidance from @ftobia and @robdennis about how to handle dependencies for plugins, since I'm not sure the best approach, but I'll need to come up with something in the next few days.

robdennis commented 10 years ago

As a before install script you could collect all the plugin requirements and write out a new top level requirements.txt that has everything

On Mon, Aug 4, 2014 at 6:38 PM, Eli Courtwright notifications@github.com wrote:

I'll probably reach out for guidance from @ftobia and @robdennis about how to handle dependencies for plugins, since I'm not sure the best approach, but I'll need to come up with something in the next few days.

Reply to this email directly or view it on GitHub: https://github.com/appliedsec/sideboard/pull/78#issuecomment-51127707

EliAndrewC commented 10 years ago

So the problem is that we still want things to be modular. So like say we have three modules: foo and bar and baz. We want them to be able to be installable on either the same server or on three different servers. This was one of the original use cases for Sideboard in the first place.

So we could bundle up everything and put all of the dependencies into the code Sideboard RPM, but offhand I can think of the following downsides:

Everyone would need to roll their own package of Sideboard which shipped with the dependencies their plugins care about.
Sideboard would need to be repackaged every time we add a new plugin with new requirements OR add or change a dependency with an existing plugin

This all seem like dealbreakers. My current thinking is that both pip and easy_install actually have flags which will only download packages to a specified location but not actually install them, so we could use that to grab all of the dependencies, package those, and then at RPM install time we can install them into the Sideboard virtualenv. Downsides which immediately spring to mind include:

uninstalling a plugin's RPM will not clean up those dependencies
any compiled dependencies like py-bcrypt will need to actually have the Python development packages plus gcc and other things which might not always be on a production machine

On balance I much prefer the second set of downsides to the first, but I'm still not thrilled out this idea. I'm not sure that there's really a better option though. I must say I hadn't thought about this downside of getting rid of separate virtualenvs at the time :(

robdennis commented 10 years ago

the main reason I suggested ditching multiple virtualenvs is because the "1 rpm for server, arbitrary rpms for all the plugins" didn't (in my opinion) turn out to be a valid deployment user story.

I'd expect a single RPM that has whatever plugins constitutes a deployment and upgrading an rpm blows everything away (just like normal rpms do now), which is sort of in the first block of "downsides."

E.g. an "uber" RPM that has sideboard and whatever plugins makes up an uber

EliAndrewC commented 10 years ago

Hmm, I'll give that some thought. In my mind, separate RPMs is still kind of a feature that we care about, because otherwise if we have foo and bar plugins, and we want to support them being on either the same or different systems, we'd need to either have three separate packages (one with foo, one with bar, and one with both) or just only support one combination. I mean, we definitely had cases in our original Sideboard app where we'd deploy multiple things to one system in development and then deploy them to separate systems in production.

Still, the advantage of this is that we'd have RPMs that just work without running into needing gcc as a dependency, so that might be a worthwhile tradeoff. (Possibly the solution in the long run would be to support both approaches.)

robdennis commented 10 years ago

In my opinion the trade off here is:

production equivalent infrastructure "everywhere" including vagrant and development, meaning A and B should be on different machines in dev if they will be in productions
bring back multiple virtualenvs if you think a core user story is rpm-per plugin

possible third option, plugin rpms have to get really smart about only uninstalling dependencies if they aren't being used

edit: I recommend 1. Vagrant has only gotten easier and it's a much easier to understand mental model

EliAndrewC commented 10 years ago

So after looking at our requirements and infrastructure, I think requiring gcc on production systems would be a dealbreaker. So I was about to give up and go with Rob's option number one where production and development must be the same, but then I took one last look at the pip documentation and saw that there IS a solution! We can use wheel, which I believe @ftobia has been espousing recently. Wheel allows us to package binary distributions of Python dependencies and then just install them.

So basically, I think we can go with the third option @robdennis mentioned, which we'd implement as follows:

when building the RPM, we'd package the plugin itself, the requirements.txt file, and the wheelhouse directory which contains the dependencies
on install, we do the following:

/opt/sideboard/bin/pip install --use-wheel --find-links /opt/sideboard/plugins/myplugin/wheelhouse/ -r /opt/sideboard/plugins/myplugin/requirements.txt

on uninstall, we can be smart about what we remove by doing something like the following:

/opt/sideboard/bin/pip uninstall --yes -r /opt/sideboard/plugins/myplugin/requirements.txt

Now the main thing that the above doesn't accomplish is the situation where foo and bar have the same dependency so we don't want to uninstall it when foo is uninstalled. So what we ACTUALLY need to do is check all requirements.txt files and only a dependency if it doesn't show up anywhere else. This would be pretty simple, though I'll probably omit that in this pull request.

EliAndrewC commented 10 years ago

I'm going to go ahead and just merge this now, which sucks because Travis is failing because PyPI only has CherryPy versions going back to CherryPy 3.2.3 and we're specifying CherryPy 3.2.2 because of that bug I linked to (which already has a pull request). Sometime soon I'll try to do something to work around this because I don't want Sideboard to be a project with perpetually failing unit tests that no one cares enough to fix. (On the plus side, our local PyPI mirror does have CherryPy 3.2.2 so we can run tests on our local Jenkins.)

robdennis commented 10 years ago

we could add a specific dependency link for that version of cherrypy if it's up somewhere else

On Wed, Aug 6, 2014 at 3:10 PM, Eli Courtwright notifications@github.com wrote:

Merged #78 https://github.com/appliedsec/sideboard/pull/78.

— Reply to this email directly or view it on GitHub https://github.com/appliedsec/sideboard/pull/78#event-149913860.

EliAndrewC commented 10 years ago

Or include it ourselves like in a Wheel file :)

I'll make sure to get this sorted out one way or the other over the weekend, if not before.

EliAndrewC / sideboard

initial stab at packaging using ship_it #78

I'll probably reach out for guidance from @ftobia and @robdennis about how to handle dependencies for plugins, since I'm not sure the best approach, but I'll need to come up with something in the next few days.