As a plugin developer, I'd like to easily set up a dev environment in Vagrant/Heroku/Docker or whatever's easiest

[robdennis]

To support running a sideboard plugin in a Heroku-like container, packaging sideboard for rpm or deb would allow:

• sideboard to be installed via a package manager

• the plugin can configured the system installed sideboard to find the working directory

  sub-requirements:

• @EliAndrewC adds back in the RPM specfile to package-support/RPM
• I will use this as a base for developing an Ubuntu deb and put it to package-support/DEB
• "testing" (sorry for the hand waving here)
• Attempt deploying a plugin on heroku using this method

[EliAndrewC]

I've created a gist with a version of our specfile edited to remove references to our internal infrastructure, since that would be unhelpful to anyone trying to build this in the wild: https://gist.github.com/EliAndrewC/10328292

Here are some things I note about this file:

• openldap features in the dependencies, so we should probably try to resolve #29 before doing any packaging work
• since this is descended from old spec files that have been copy/pasted in our internal projects, there's a lot of "someone probably added this at some point to get a spec file to work and so this line has been in every spec file ever since", e.g.

# some systems dont have initrddir defined -- it seems that CentOS may be one of them (?)
%{?_initrddir:%define _initrddir /etc/rc.d/init.d}

• on a similar note, the very top of the file aliases Python to our own Python27 packages since I think we started building at a time when Red Hat Enterprise Linux 5 didn't have Python27 packages yet, so we pointed to that instead of the system Python - that worked internally for us, but is definitely an example of something that would need to be ripped out to make this work in general
• this is limited to i386, though at a glance we could presumably build 64-bit RPMs as well if we cared to
• I didn't bother to include the default config files for sideboard-core and sideboard-server, etc since those wouldn't be relevant to anyone
• Many of the descriptions appear to have been copy/pasted but not edited, e.g. we see "The Python libraries supporting Sideboard" in a bunch of places even when that doesn't look correct
• do we really need shadow-utils for Sideboard? or is that a vestigial remnant from an older specfile?
• Not a big deal, but --never-download is the default behavior for virualenv on all of versions that Sideboard requires/supports so we don't need it in our specfile
• the specfile creates a sideboard user and group, but this is not actually used; the sideboard python process just runs as root (this was a ticket which I didn't bother moving into the Github issue tracker because it wasn't relevant until now since we didn't include any packaging in the initial version uploaded to Github)
• we do a lot of totally-necessary stuff in the specfile related to packaging a virtualenv in our RPM; this is a design decision that made sense for us and probably(?) makes sense for a nontrivial number of other people, but it's entirely possible that it creates more problems than it solves - that's probably just an open question for now, and I don't have enough packaging experience to have a real sense of which way that will come down

[EliAndrewC]

Ensconce has now been open-sourced and I'm planning on converting it to be a Sideboard plugin. Having working packaging would be super-helpful for this. Even if we couldn't run on Heroku, just having working RPMs and Debian packages would be amazing.

[robdennis]

That's really cool :-)

On Tue, May 20, 2014 at 4:08 PM, Eli Courtwright notifications@github.com wrote:

Ensconce has now been open-sourced and I'm planning on converting it to be a Sideboard plugin. Having working packaging would be super-helpful for this. Even if we couldn't run on Heroku, just having working RPMs and Debian packages would be amazing.

Reply to this email directly or view it on GitHub: https://github.com/appliedsec/sideboard/issues/49#issuecomment-43676267

[hozn]

Does TravisCI provide a mechanism for hosting artifacts (i.e. RPM packages)? Or do you guys have a strategy for distributing RPMs you build for Sideboard -- or does that make sense as a standalone RPM? (I'm curious, because the same thing will apply to Ensconce, where distributing RPMs is quite a bit more helpful than the tar.gz)

[robdennis]

No it does not, the closest thing thing I know of is sourceforge

On Wed, May 21, 2014 at 7:41 AM, Hans Lellelid notifications@github.com wrote:

Does TravisCI provide a mechanism for hosting artifacts (i.e. RPM packages)? Or do you guys have a strategy for distributing RPMs you build for Sideboard -- or does that make sense as a standalone RPM? (I'm curious, because the same thing will apply to Ensconce, where distributing RPMs is quite a bit more helpful than the tar.gz)

Reply to this email directly or view it on GitHub: https://github.com/appliedsec/sideboard/issues/49#issuecomment-43743819

[EliAndrewC]

We definitely plan on having deb and rpm packages for Sideboard, as well as for Sideboard plugins. This will be especially helpful for the MAGFest code, since I'd like to be able to make a change and then push a button to deploy the new RPM to production.

I did a little Googling and it looks like Travis does support uploading artifacts to Amazon S3: http://blog.travis-ci.com/2012-12-18-travis-artifacts/

With that being said, needing to manually build packages would be a pain but isn't an unacceptable solution. So as long as we have some way to create deb and rpm packages, that would get us 90% of the way there.

[EliAndrewC]

After doing some more research, it looks like there's a new way to do this. Github added support for "releases" last year (https://github.com/blog/1547-release-your-software) and Travis closed their ticket (https://github.com/travis-ci/travis-ci/issues/1648) for supporting this 2 months ago. It looks like this uses the Travis dpl utility: https://github.com/travis-ci/dpl#github-releases

So the workflow seems to be something like this:

• add our Github private API keys to our .travis.yml file in the form of secure environment variables, which are encrypted so that we're not checking in sensitive stuff in plain text (http://docs.travis-ci.com/user/build-configuration/#Secure-environment-variables)
• Travis builds our packages and on every successful build, it builds the proper packages and creates a new release using the dpl tool and uploads the packages back to the Github repo
• Since we probably want to have both development and stable packages, we could probably make the build system check what branch Travis is running out of; if it's the releases branch then it builds the stable packages, otherwise it'll build development packages

That's just a thought, and it sounds like it might take awhile to actually set all of this up. I probably won't have time for any of that anytime soon, so like I said before, being able to manually create packages would get me 90% of the way to where I want to be.

[binary1230]

Hey sup Eli I am all commenting on your projects. Booya.

FYI, I might be taking a stab at this someday as it'd be useful for Magfest deploys for ubersystem which now runs as a sideboard plugin.

[EliAndrewC]

That's cool. For what it's worth, @robdennis has created a separate repo for codifying some Python packaging "best practices": https://github.com/robdennis/ship_it

Unfortunately, at the moment this works with Python 2.7 but fails on Python 3 due to the fact that it uses fabric, which is not yet Python 3 compatible. I'm going to give ship_it a try this week with a project at work that's a small Sideboard app which needs some packaging which also uses Python 2.7, and in the long run we can just wait for fabric to become Python 3 compatible or make a pull request to fix it ourselves :)

[binary1230]

Cool, also looks like Fabric might be just about ready to upgrade to py3: https://twitter.com/bitprophet/status/468082051842064384

[EliAndrewC]

Nice. I'd really love for MAGFest to be deploying packages instead of running git pull to fetch updates, but that might need to wait until next year. Or maybe we'll have some time in like September / October after I finish the department head checklist, revamping the Angular app for Shift Signups, and finish clearing out some of the open issues in our issue tracker.

[EliAndrewC]

We probably don't care about "Heroku" for this ticket, and instead replace that with "Vagrant and/or Docker and/or whatever makes it easiest for a new developer to get up and running with Sideboard".