Move from django-pipeline to gulp for static assets

[pmac]

This is a big change to how we build and develop things.

• There is a node image for building assets completely decoupled from Django
• The python image copys the resulting files from the assets container
• In dev the assets container runs browser-sync, which reloads your browser for each change you make and proxys the django so that you get a single URL (localhoost:3000) for viewing updates to the django app and static files.
• Simplified the docker-compose setup by having a single file now.
• Fixed up Makefile to make it a simple make build && make run to get going.

It's not complete yet. I still need some docs and I'd like more than just me to test. But this is a good direction IMHO and easier with which to work.

Caveats

• This moves the app static files to a /static directory in the project. Shouldn't matter much, but that happened.
• This vendors the Django admin static assets in the /static dir as well. This should be no problem until we upgrade Django: we'll have to remember to copy the updates to the app. I needed this to keep the static assets and app images as separate and easy to build as possible.

[pmac]

This is very much related to my similar work on bedrock, if you'd like to check that out as well.

[willkg]

Holy moly! I'll try to work through this tomorrow. Promise!

[willkg]

Out of curiosity, why vendor the JS files and not install them with npm or something?

[pmac]

You mean the jQuery and modernizer? I'm sure we could. They were just in the static files already and I'm already rocking the boat so much that I just left that alone. But we could easily just include them in package.json and add the directory to which they're installed to the gulpfile collection stuff.

[willkg]

Travis is dying with this:

ERROR: Couldn't find env file: /home/travis/build/mozilla/standup/.env

I had a similar issue with Socorro and did this:

https://github.com/mozilla-services/socorro/blob/master/Makefile#L22

Then had rules that ran docker-compose rely on that.

[willkg]

I'm going through this now. I'll stop trickling comments.

[pmac]

This has the same problem we had before where files get generated by one of the containers in the repo directory, but they're owned by root. That causes "make clean" (and thus "make rebuild") to fail. We can figure that out in another PR, though.

I'm confused by this. It should not be writing to your filesystem. What exactly is it writing? It should only be writing to the container for building the static files. I specifically attempted to fix this. I'm sure we can figure it out. Is it .pyc files?

[willkg]

This has the same problem we had before where files get generated by one of the containers in the repo directory, but they're owned by root. That causes "make clean" (and thus "make rebuild") to fail. We can figure that out in another PR, though.

I'm confused by this. It should not be writing to your filesystem. What exactly is it writing? It should only be writing to the container for building the static files. I specifically attempted to fix this. I'm sure we can figure it out. Is it .pyc files?

From a clean repository, I do make test and then I end up with this:

> ls -al
total 2960
drwxr-xr-x 11 willkg willkg    4096 May  7 09:05 .
drwxr-xr-x 74 willkg willkg    4096 Apr  9 11:48 ..
drwxrwxr-x  2 willkg willkg    4096 Apr  7 21:04 bin
-rw-rw-r--  1 willkg willkg     891 Sep 26  2016 contribute.json
-rw-rw-r--  1 willkg willkg     316 Sep 27  2016 CONTRIBUTORS
drwxrwxr-x  2 willkg willkg    4096 May  4 16:09 docker
-rw-rw-r--  1 willkg willkg       0 May  7 09:05 .docker-build
-rw-rw-r--  1 willkg willkg    1910 May  4 16:09 docker-compose.yml
-rw-rw-r--  1 willkg willkg     113 May 19  2017 .dockerignore
-rw-rw-r--  1 willkg willkg     316 Feb  8  2017 .editorconfig
-rw-rw-r--  1 willkg willkg      61 May  4 16:10 .env
-rw-rw-r--  1 willkg willkg      61 May  4 16:09 .env-dist
drwxr-xr-x  8 willkg willkg    4096 May  7 09:05 .git
-rw-rw-r--  1 willkg willkg     140 May  4 16:09 .gitignore
-rw-rw-r--  1 willkg willkg    3417 May  4 16:09 gulpfile.js
-rw-r--r--  1 willkg willkg    1474 Sep 11  2012 LICENSE
-rw-rw-r--  1 willkg willkg    2313 May  4 16:35 Makefile
-rwxrwxr-x  1 willkg willkg     250 May 19  2017 manage.py
-rw-rw-r--  1 willkg willkg    8829 Sep 30  2016 newrelic.ini
-rw-rw-r--  1 willkg willkg    1111 May  4 16:09 package.json
-rw-rw-r--  1 willkg willkg      23 Aug 10  2016 Procfile
drwxr-xr-x  3 root   root      4096 May  7 09:05 .pytest_cache
-rw-r--r--  1 willkg willkg     449 Oct 15  2017 .pyup.yml
-rw-rw-r--  1 willkg willkg    2343 Feb 13  2017 README.rst
-rw-rw-r--  1 willkg willkg    5756 May  4 16:09 requirements-dev.txt
-rw-rw-r--  1 willkg willkg   18379 May  4 16:09 requirements.txt
drwxrwxr-x  2 willkg willkg    4096 Sep 26  2016 root_files
-rw-rw-r--  1 willkg willkg     206 Mar  1  2017 setup.cfg
-rw-rw-r--  1 willkg willkg    2061 Jun 16  2016 spec.rst
drwxr-xr-x  7 willkg willkg    4096 May  7 09:05 standup
drwxrwxr-x  7 willkg willkg    4096 May  4 16:09 static
drwxr-xr-x  2 root   root      4096 May  7 09:05 staticfiles
drwxrwxr-x  2 willkg willkg    4096 Feb  9 12:53 tests
-rw-rw-r--  1 willkg willkg     921 Dec 22  2016 .travis.yml
-rw-rw-r--  1 willkg willkg  129460 May  4 16:09 yarn.lock

Does that help track it down?

[pmac]

Does that help track it down?

Definitely! Thanks. I'll let you know when to try again.

[pmac]

@willkg okay. I think I cracked it. Give it another go.

[willkg]

Same thing. I updated my local branch, verified I've got the most recent commits, cleaned everything out, and then ran make test and I get a staticfiles owned by root.

I can look into it more tomorrow and figure out what's making that directory, if that helps.

[pmac]

Damn... I tried it on my linux box and it worked. Docker for Mac works this out for you, but on Linux it lets to do other things. You sure you had deleted the root-owned one from the previous run? I'm sure it's something small we're now missing. Oh! Or perhaps your profile doesn't set the UID variable? My fix relies on that containing your numeric user ID.

[pmac]

If the $UID env var isn't reliable then we can just make a docker-compose.sh wrapper script that will add that to the environment in which docker-compose runs. I'm having to do this for bedrock to get all of the git info into the env. e.g.:

#!/bin/sh

set -eo pipefail
export UID=$(id -u)
docker-compose "$@"

Think that might be the issue?

[willkg]

I'm puzzled as to where the UID environment variable is coming from. It's not a standard environment variable as far as I know. I don't see it on my OSX machine, either.

For adjusting which user is used, I've been doing stuff like this:

• https://github.com/mozilla-services/antenna/blob/master/Makefile#L3
• https://github.com/mozilla-services/antenna/blob/master/Makefile#L76

We have a CURRENT_USER set in the Makefile already:

https://github.com/mozilla/standup/blob/master/Makefile#L4

Maybe we can use that with -u?

Having said that, I'm of the mind that the ownership of staticfiles is exceeding the scope of this PR. We should land the PR and push that off to a separate issue especially since it was a problem before, too.

[willkg]

Er, I meant we should land this as soon as Travis is ok with everything.

[pmac]

Cool. I'll make some final tweaks and clean up the commits and we can get it in. Thanks for the help!!

[pmac]

Okay. I've given the perms thing one last shot based on the antenna stuff and what I'd done earlier. I've also squashed the commits. Might be ready for merge.

[willkg]

I had one issue where the new value for USERID in the Makefile I think breaks what USER is expecting. Everything else looks fine to me and Travis is happy.

This is awesome work! I'm super excited about this! Thank you!

[pmac]

Okay. Working for me again. Might be ready :)

[willkg]

Looks great! Merge it!