Integrating with the front-end build process

[willbarton]

The front-ends have standardized on a single setup.sh file that invokes npm, bower, and grunt. In regulations-site, we've integrated that with the Python build, so that a pip install will run the frontendbuild.sh script.

I'd like @Ooblioob and @rosskarchner's thoughts on whether this is particularly useful for delivery, as well as the rest of the team's thoughts on how we can make this reusable. The build_ext and bdist_egg overrides are required for build_frontend to be invoked by pip, and those overrides make it a bit harder to just wrap this in a separate package that can be imported.

It's not impossible — that package just provides those overrides too. But this could make the process more complex for any projects that need to also override parts of setuptools. Of course, those cases might be doing something so completely different it doesn't really matter.

Thoughts?

[rosskarchner]

My first reaction, is that this is deeply weird-- but it's conceptually not that different from a Python module that includes C code, knowing how to compile itself.

This also relates to something @higs4281 is working through.

Just to make sure, even outside of the 'pip' context, a 'python setup.py install' will build the front end assets?

This might call for another discussion thread, but imagine we are distributing such an app on pypi: should the egg/wheel/whatever have this stuff already built?

[Ooblioob]

One strategy could be to include all the built stuff in some kind of distribution directory and commit that to source control. While I like the thought of only having to clone the code (and less setup), it pollutes our commit history in a way that makes reviewing PRs a nightmare. Plus there would be lots of files that "we just have" where it's unsure why we have those files.

My worry with the frontendbuild.sh strategy is dependencies. For example, if frontendbuild.sh calls a grunt task, then grunt needs to be installed. So does browserify, gulp and any other things we need to build, and it's not explicitly mentioned in our python setup.py file. If our goal is that all applications should be able to build themselves, this might be less than ideal.

Granted, the alternative is we have ansible do the build, which also requires the dependencies of grunt, gulp, browserify, etc. But at least there we can ensure those dependencies are met.

[willbarton]

@rosskarchner Yes, that's kind of what prompted this. @higs4281 and @marteki are pulling in what I did for regulations-site. I don't give any thought to using my own code, but when other people do it I tend to give it more thought ;)

The way this is set up right now, yes, python setup.py install will build the frontend assets. As to distributing... I'm not sure whether we should include those assets or not. Including generated files can be ugly (as @Ooblioob points out), and it seems a bit silly to ask the front-ends to commit their generated files, when if they came to us and asked if we could commit generated files so they could package things cleanly with npm, I'd be quite hesitant.

But yes, the front-end is generally assuming npm+brower+grunt. And this is including that in the process. I don't want to dictate possible dependencies to the front-end (IMHO they can build however they think it best to, it's their problem domain), but I don't want to introduce more back-end dependencies (like Ansible) because of this.

Perhaps the solution is something more to the effect of an npm, grunt, and bower build command that's not automatically executed. If we decided to package something for pypi we'd have to ensure that was done, but it wouldn't be automatic. That breaks the ability to just pip install one of our projects straight from GitHub, though (i.e. including the GitHub URL in a requirements.txt file).

[Scotchester]

Could be worth exploring trying to integrate the installation of Node/npm/Grunt/Gulp/Bower into the setup/frontendbuild.sh file.

@cfpb/front-end-team-admin have we considered this?

[willbarton]

@Scotchester @cfpb/front-end-team-admin I think I'd much rather include that node+npm are requirements in the README. Everything else should be able to be pulled in non-globally via npm if needed, right?

[anselmbradford]

@Scotchester @cfpb/front-end-team-admin I think I'd much rather include that node+npm are requirements in the README. Everything else should be able to be pulled in non-globally via npm, right?

This is how we have it in flapjack. Our global requirements are in our INSTALL.md and the rest is in setup.sh. To @Scotchester point, I'm not really clear why some tooling is installed globally. For flapjack this just seems to have evolved that way through the legacy of the build process. I guess the advantage is each project doesn't have to go about installing the base tooling themselves. Also, it's a relatively common workflow to switch branches and run ./setup.sh to clean everything out and get different branches into a clean state, which would be slightly faster with some tools moved to a global context. However, I think the simplification of the installation requirements lead me to +1 to @willbarton idea to make node+npm the only readme requirement and pull everything else in through a build script.

[Ooblioob]

This makes sense to me, having node+npm be our only dependencies. From an automation perspective it would simplify things, I'd just have to ensure those are in the playbook then pip install vs trying to figure out what each team uses and making sure those are met.

[Scotchester]

To use Grunt as one example (which may not apply to other programs), grunt-cli can be installed locally, but it's not even discussed as an option in their Getting Started guide, and Node's general guidance on global/local would seem to indicate that global is preferred.

I also found a good argument for not installing globally, particularly when dealing with services like Travis.

[anselmbradford]

nodejs on global vs local guidance

[willbarton]

So, the problem seems to be that in order to build front-end dependencies for our projects we need to have front-end build tools (npm, bower, grunt/gulp) installed somehow. But those build tools themselves do not need to travel with the packaged application.

In addition we may have front-end run-time dependencies that get installed via npm or bower. npm seems like it would be particular problematic in this regard, because we'd have no way to distinguish between build-dependency and run-dependency if they're all installed locally (in ./node_modules), correct?

@anselmbradford @Scotchester What makes the most amount of sense from a front-end perspective? To require npm/bower/grunt/gulp as required dependencies that we don't provide (like Python)? And then have those tools generate our CSS and JS, as well as install run-dependencies locally?

That puts more dependencies outside the Python installation process, but it might make more sense than pulling everything in. But this could be based entirely on a flawed understanding of all the above tools.

It would also mean that it's up to the front-end to ensure all of the above... the best we could do from the Python side is invoke setup.sh (or npm, bower, and grunt individually).

[ascott1]

My understanding is that many of our projects would need a global install of Bower and Grunt or Gulp in addition to Node. I just tested this on a clean version of Node and it seems to be the case, even with Grunt declared as a dependency, which unfortunately makes for a much less clean solution.

[contolini]

npm prepends ./node_modules/.bin to the PATH provided to any npm scripts. So if in your package.json file you have:

"scripts": {
  "test": "grunt test",
  "vendor": "bower install"
},
"devDependencies": {
  "grunt-cli": "latest",
  "bower: "latest"
},

It will run A-OK. You don't need to install grunt or bower globally. The user just needs to run npm install beforehand to pull down the deps, which will be stored in ./node_modules and any executables are dumped into ./node_modules/.bin. And then you should run npm test instead of grunt test.

I emphasize prepend because if you have bower installed both globally and locally, it'll use the local version. You can test this with:

"scripts": {
  "check-bower-version": "bower -v"
},
"devDependencies": {
  "bower: "0.1.0"
},

and then running npm run check-bower-version will return 0.1.0 instead of whatever version you have installed globally.

[willbarton]

@contolini But if we do that and need to carry ./npm-modules with us for run-time dependencies (would that be a situation we'd encounter?), that'll bring grunt and bower along for the ride, correct?

[wpears]

@willbarton the run-time dependencies will likely be built into something like bundle.min.js and shipped to the client. So the devDependencies would be excluded.

[contolini]

@willbarton Yeah what @wpears said. There should be no run-time deps stored in ./node_modules. They'll either be consumed by a bundler (like browserify) or they'll be strictly used to compile source files into production files.

But if you did copy ./node-modules to a different server, npm would pick them up. Just ensure both environments are running the same version of node.

[willbarton]

@contolini @wpears Ok, so really it wouldn't be a problem if we required node/npm, and if bower and grunt aren't available, installed them and anything else locally, and just made sure that ./node_modules doesn't get included with the Python package install, right?

[Scotchester]

One needs to add grunt-cli as a dependency to use it without the global install. grunt is already a dependency.

[contolini]

@Scotchester Ah, good call. Grunt has a separate cli: https://www.npmjs.com/package/grunt-cli#installing-grunt-cli-locally

[Scotchester]

Was this comment invisible or something? That's now two reposts of links I posted in that ;)

[willbarton]

So it sounds like from the Python setup.py we can call front-end setup.sh file (or frontendbuild.sh, whatever it happens to be) which calls npm, bower, grunt, and whatever else might be needed to build the front-end, and npm can install the appropriate tools locally if they don't already exist. The end result would be our generated front-end code with dependencies, etc.

At that point, a pip install would result in building the front-end and then discarding all of the locally-installed things. A python setup.py install would not clean up after itself, however, we would also need to provide some customization of the clean task that does that.

Are we okay with this? If so, we could provide our custom setuptools commands in a library that we can just import so projects aren't repeating themselves. I could package up what I've done on reg-site.

[contolini]

@Scotchester What comment? I don't see anything.

[willbarton]

@Scotchester

[rosskarchner]

What I think would be best, is if 'setup.py build' could run the front end stuff, and give us the option of having a wheel/egg/zip/sdist/whatever to move around and distribute-- so you could then 'pip install' that package on a machine without needed all of the front-end build dependencies. At that point it's just a plain old python package.

[willbarton]

That sounds good too — I took the approach I originally did because we were including the git+https URLs in requirements.txt files. But if we want to move to distributable archives of whatever form (I am a bad Pythonista and have only really glanced at wheels and gone "huh"), that would be fine too.

[rosskarchner]

It leaves the door open, at least: if we don't bother distributing the built packages, it'll still behave the way you describe (because 'install' implies 'build', I think)

[willbarton]

:+1:

So I'll package up what I've done here so that we have something reusable, and see if I can do something a little more generic than overriding bdist_egg.

[higs4281]

Retirement has a somewhat pressing need to iron out a path to production that unifies Python and front-end builds. We can't update our app because we no longer fit into the current build process.

Could we use Retirement as a test for the @willbarton setup.py scheme? The current blocker is that the built front-end assets don't have a path to cfgov.

This issue and #5 explore having a Python build process that includes front-end concerns; Front-end issue 49 and @anselmbradford suggest having a front-end build that includes Python concerns. The idea of a single script that would build any CFPB project is appealing, but legacy projects don't have front-end pipelines, and shouldn't we also allow for projects that don't include Django or Python? Would it be a nightmare to allow either setup.sh or setup.py to drive the bus? @rosskarchner @Ooblioob

[willbarton]

@higs4281 what's the problem exactly? I saw you imported the stuff I did in reg-site... did that solve the problem, or is there something else going on?

I made my changes to reg-site based on our current deployment process, which seems to prefer pip.

[higs4281]

@willbarton The Jenkins build routine for Django apps doesn't include a front-end build process. It assumes front-end assets are in place, and runs collectstatic to round them up. Before we added a front-end build, Retirement fit into this scheme. Now we need a step that would allow us to build resources and put them in place before collectstatic runs. Node is not currently installed on build, content or prod servers.

[anselmbradford]

Yeah, I'm still not understanding why use setup.py to call setup.sh and not the reverse? setup.sh is meant to be as close to the system and tech-agnostic as possible. I think perhaps since it came up in the front-end repo/evolved out of frontend.sh it's getting confused as being a purely front-end build script, but the idea was something more analogous to a makefile. Maybe having that written in python makes sense for a reason I'm not seeing, but it superficially seems like shell is more appropriate. If we created a nodejs project would it make sense to have a bootscript called setup.js that's the entry point? I don't think so. It would make more sense instead to call the project-specific bootscript from setup.sh.

[willbarton]

@anselmbradford Because right now we're installing our projects via pip. Doing so presumes that the CSS/JS/HTML/etc travel with the Python package as part of the application in that way, and are therefore part of the Django app that gets included in our main Django project.

Without modifying the existing build scripts, that was the easiest way to include the built front-end with the Python package.

I would argue that going forward that makes sense to me, because it's easier to transport the built-front-end as part of a built Python package than the reverse. And ultimately it's the Python package that gets served via Django. But perhaps I have a skewed view of that?

The only alternative is that front-ends and back-ends live in separate packages.

[anselmbradford]

@willbarton Ahh okay, so these are python packages that are distributed through a package manager? Is having two installation entry points an option? pip install calls ./setup.sh, but ./setup.sh on its own works comprehensively as well. One for python package distribution and another for manual installation. Perhaps this is the setup you've already written about.

[willbarton]

@anselmbradford So, pip allows you to install Python packages as egg files, wheel files, or straight from git. When you install from git directly (i.e. from github) it runs setup.py to build the package. Installing from GitHub is a relatively normal pattern (and allows you to specify requirements that way too) So, if you want to preserve that ability, it has to be setup.py. The best we could do is have a setup.sh that calls python setup.py [task].

I think it makes more conceptual sense that the front-end build stuff happens at the "build" stage of the Python packaging, because it's going to be generating the final CSS/JS that gets served via the Django app. So the Python package carries around with it the compiled/minified front-end code as well as the HTML and Python code.

[willbarton]

I will add that how the Python package builds the final front-end code is entirely up to the front-end team. You've settled on a setup.sh file for now for running npm, grunt/gulp, and bower. We could easily invoke them separately as part of the build process too, if needed.

[mehtadev17]

I am using https://github.com/nodesource/distributions as a way to install node 0.10. It already has .sh scripts. After running the script its a simple yum install nodejs on centos.

Note I did have to change this line https://github.com/nodesource/distributions/blob/master/rpm/setup#L133 to match one of these https://rpm.nodesource.com/pub_0.12/el/6/x86_64/

[willbarton]

Hi! We'll be discussing this issue in today's Dev COP!

I'm asking for two things:

1. Testable hypotheses: How can we make this work?
2. Experimentation: Does it work? Why? Can we use it?

Stay tuned for more Project BEFE: Mission: Impawsible

[cfarm]

@moka What do you think about trying this on CCDB 4.1?

[willbarton]

Here's what I have coming out of our Dev COP meeting last week, where I outlined the problem space. We have two hypotheses at the moment:

@marcesher, what are your thoughts?

Hypotheses:

1. Python Building

Hypothesis

setup.py can integrate the front-end setup.sh and it will be possible to build Python wheels that contain the built front-end assets and the Python code all in one package.

This would result in the most straight-forward deployment of Django app via Python packages, installed via pip into a regular Python location. There would be fewer surprises for those with Django and Python experience.

Testing it

Implement bdist_wheel and bdist_egg in a setup.py file and see if it can successfully invoke setup.sh. Then examine the subsequent wheel file for correct front-end assets.

Does the subsequent Python wheel (or egg) contain the correct and complete front-end assets? Can it be deployed via Jenkins and work?

Potential problems

Only usable with Python back-ends. It doesn't solve any problems we might have integrating the front-end process with other back-ends. This might not be such a bad thing — it might be best to encourage domain-specific solutions rather than a global one-size fits all approach.

Volunteer(s)

@willbarton

2. Tarball (or RPM?)

Hypothesis

We can create a language-agnostic system to build and install single units of software containing components regardless of their implementation languages. Front-end assets can be built and a single tar file (or RPM?) created that can be deployed that contains everything in place for deployment.

Testing it

Implement a simple wrapper (shell script?) around the front-end build process and the Python build process to create the archive.

Does the archive contain all the appropriate front-end and back-end files? Can it be deployed via Jenkins? Does it work? Can it be pluggable, so that we can use it for our Scala and Clojure projects as well?

Potential problems

It would require a re-think in how we deploy software and our dependence on language-based dependency discovery.

Volunteer(s)

???

[cfarm]

@willbarton if our standard is Django + Python apps I think an MVP solution should only care about that :+1:

[willbarton]

@cfarm I agree to an extent... the only caveat is that a Python-only solution is absolutely not portable to anything else. However, I don't know that we struggle with this problem in any context other than front-end + python.

[willbarton]

Just as a note, another alternative is to use Conda instead of pip/PyPi. Conda is meant for packaging Python plus other stuff.

[willbarton]

In my testing, this setup.py works.

I'll be presenting in Dev COP tomorrow, but the basic idea is that this setup.py file will create either egg or wheel files (or it can be pip installed directly if npm, gulp, and bower are already installed) which can then be deployed. Meaning that instead of building the front-end artifact, copying it to the deployment target, and then pip installing the Python package, we can generate a single egg or wheel file artifact in the build stage that has the Python packages and the front-end assets, and then pip install that file.

I'll have a more concrete example tomorrow.

[higs4281]

:+1: Conda is slick, but if we can adopt your method, I'd vote for avoiding another packaging dependency.

[rosskarchner]

I've been thinking about this (slowly, obviously...) since Will's presentation, and I'm starting to wonder if this is actually a good path to go down. Django does a pretty good job in allowing your "app", static assets, and even templates to be loosely coupled. They can be in separate locations on the filesystem, and hence can be distinct deployable artifacts.

In other words-- Is this added complexity really just helping us avoid generating two artifacts? (a python package, and a bundle of static assets). Is that a fair trade?

[willbarton]

I think that's a fair trade, as long as we're generating deployable artifacts.

I think the concern is the current process has front-end assets built, then the the Python is pip install -eed from a local checkout, the built front-end assets are copied into that checkout, and then collectstatic is called. That's convoluted.

I could see a process whereby Python wheels are installed, and NPM packages are installed to the Django static location, or something to that effect.

[willbarton]

While we await https://github.com/cfpb/drama-free-django with bated breath, why don't we put some effort into creating a sample Jenkins As Code build flow that will build front-end and Python packages and deploy them? Keep with our experimental framework and see how that works?

Does anyone want to take this on?

[rosskarchner]

I don't know much about the JaC part of this, but PR 390 in cfpb_django adds a new "static.in" directory-- we can deposit static assets there, and they will get sucked when collectstatic is run.

I'll make sure we maintain such a mechanism post-v1.

[imuchnik]

Mememememe! I would love to, but will probably need some guidance, so pairing on this with someone is much appreciated in advance.

[willbarton]

@rosskarchner :+1: I do like this approach.

@imuchnik I'll happily pair on this.

[higs4281]

Is this the idea: All front-end build deployment jobs would push their name-spaced assets to the central static.in directory for collection? Or is this a step toward having all Django apps in the same repo? or both?