search

gratipay / inside.gratipay.com

Here lieth a pioneer in open source sustainability. RIP
https://gratipay.news/the-end-cbfba8f50981
57 stars 38 forks source link

Tech Radar 3 #612

Closed gratipay-bot closed 8 years ago

gratipay-bot commented 8 years ago

← Tech Radar 2

Docs

http://inside.gratipay.com/howto/sweep-the-radar

Mission

The mission of the tech team is to support the product team in solving problems in the market.

Scope

chadwhitacre commented 8 years ago 
[gratipay] $ git branch --merged
* master
  symmetric-encryption
[gratipay] $

😳

Deploying gratipay/gratipay.com#3998!

chadwhitacre commented 8 years ago 
[gratipay] $ ./bin/keygen.py | pbcopy

I set CRYPTO_KEYS in the Heroku UI to avoid trails in my bash history.

chadwhitacre commented 8 years ago

Hmmm ... build error? And it didn't halt deployment? :-/

screen shot 2016-05-10 at 6 34 32 am

remote:        copying cffi/lock.py -> build/lib.linux-x86_64-2.7/cffi
remote:        
remote:        copying cffi/_cffi_include.h -> build/lib.linux-x86_64-2.7/cffi
remote:        
remote:        copying cffi/parse_c_type.h -> build/lib.linux-x86_64-2.7/cffi
remote:        
remote:        copying cffi/_embedding.h -> build/lib.linux-x86_64-2.7/cffi
remote:        
remote:        running build_ext
remote:        
remote:        building '_cffi_backend' extension
remote:        
remote:        creating build/temp.linux-x86_64-2.7
remote:        
remote:        creating build/temp.linux-x86_64-2.7/c
remote:        
remote:        gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/app/.heroku/python/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
remote:        
remote:        c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
remote:        
remote:         #include <ffi.h>
remote:        
remote:                         ^
remote:        
remote:        compilation terminated.
remote:        
remote:        error: command 'gcc' failed with exit status 1
remote:        
remote:        ----------------------------------------
remote:        Cleaning up...
remote:        Command /app/.heroku/python/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-FCRh4o-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-inx0Dz-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip-FCRh4o-build
remote:        Storing debug log for failure in /app/.pip/pip.log
remote: 
remote: 
remote: -----> Discovering process types
remote:        Procfile declares types -> web
remote: 
remote: -----> Compressing...
remote:        Done: 62.2M
remote: -----> Launching...
remote:        Released v1592
remote:        https://gratipay.herokuapp.com/ deployed to Heroku
remote: 
remote: Verifying deploy.... done.
To git@heroku.com:gratipay.git
   279b7ca..3da5ce0  master -> master
Counting objects: 4, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 350 bytes | 0 bytes/s, done.
Total 4 (delta 2), reused 0 (delta 0)
To git@github.com:gratipay/gratipay.com.git
   1bb41d3..3da5ce0  master -> master
Total 0 (delta 0), reused 0 (delta 0)
To git@github.com:gratipay/gratipay.com.git
 * [new tag]         1953 -> 1953
[gratipay] $
chadwhitacre commented 8 years ago

Blech, and we're 500 ... :-/

Rolling back ...

chadwhitacre commented 8 years ago

Rolled back. Hmm ...

chadwhitacre commented 8 years ago

Possibly relevant: https://github.com/heroku/heroku-buildpack-python/issues/231.

chadwhitacre commented 8 years ago

Looks like the Python buildpack makes libffi available when certain packages are named in requirements.txt. I believe we're pegged to an old buildpack, though ...

chadwhitacre commented 8 years ago

Here's the failure I captured in the heroku logs:

2016-05-10T10:40:29.538670+00:00 heroku[web.1]: Starting process with command `gunicorn gratipay.main:website --bind :14054 --workers=1`
2016-05-10T10:40:32.161043+00:00 app[web.1]: 2016-05-10 10:40:32 [3] [INFO] Starting gunicorn 18.0
2016-05-10T10:40:32.161718+00:00 app[web.1]: 2016-05-10 10:40:32 [3] [INFO] Listening at: http://0.0.0.0:14054 (3)
2016-05-10T10:40:32.161842+00:00 app[web.1]: 2016-05-10 10:40:32 [3] [INFO] Using worker: sync
2016-05-10T10:40:32.173027+00:00 app[web.1]: 2016-05-10 10:40:32 [8] [INFO] Booting worker with pid: 8
2016-05-10T10:40:33.288964+00:00 heroku[web.1]: State changed from starting to up
2016-05-10T10:40:33.239856+00:00 app[web.1]: 2016-05-10 10:40:33 [8] [ERROR] Exception in worker process:
2016-05-10T10:40:33.239865+00:00 app[web.1]: Traceback (most recent call last):
2016-05-10T10:40:33.239866+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/arbiter.py", line 495, in spawn_worker
2016-05-10T10:40:33.239867+00:00 app[web.1]:     worker.init_process()
2016-05-10T10:40:33.239867+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/base.py", line 106, in init_process
2016-05-10T10:40:33.239868+00:00 app[web.1]:     self.wsgi = self.app.wsgi()
2016-05-10T10:40:33.239869+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/base.py", line 114, in wsgi
2016-05-10T10:40:33.239869+00:00 app[web.1]:     self.callable = self.load()
2016-05-10T10:40:33.239870+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 62, in load
2016-05-10T10:40:33.239870+00:00 app[web.1]:     return self.load_wsgiapp()
2016-05-10T10:40:33.239871+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load_wsgiapp
2016-05-10T10:40:33.239872+00:00 app[web.1]:     return util.import_app(self.app_uri)
2016-05-10T10:40:33.239872+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/util.py", line 354, in import_app
2016-05-10T10:40:33.239873+00:00 app[web.1]:     __import__(module)
2016-05-10T10:40:33.239873+00:00 app[web.1]:   File "/app/gratipay/main.py", line 6, in <module>
2016-05-10T10:40:33.239874+00:00 app[web.1]:     import gratipay.wireup
2016-05-10T10:40:33.239875+00:00 app[web.1]:   File "/app/gratipay/wireup.py", line 32, in <module>
2016-05-10T10:40:33.239875+00:00 app[web.1]:     from gratipay.models.account_elsewhere import AccountElsewhere
2016-05-10T10:40:33.239876+00:00 app[web.1]:   File "/app/gratipay/models/account_elsewhere.py", line 17, in <module>
2016-05-10T10:40:33.239877+00:00 app[web.1]:     from gratipay.security.crypto import constant_time_compare
2016-05-10T10:40:33.239877+00:00 app[web.1]:   File "/app/gratipay/security/crypto.py", line 9, in <module>
2016-05-10T10:40:33.239878+00:00 app[web.1]:     from cryptography.fernet import Fernet, MultiFernet
2016-05-10T10:40:33.239878+00:00 app[web.1]: ImportError: No module named cryptography.fernet
2016-05-10T10:40:33.239879+00:00 app[web.1]: Traceback (most recent call last):
2016-05-10T10:40:33.239879+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/arbiter.py", line 495, in spawn_worker
2016-05-10T10:40:33.239880+00:00 app[web.1]:     worker.init_process()
2016-05-10T10:40:33.239881+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/base.py", line 106, in init_process
2016-05-10T10:40:33.239881+00:00 app[web.1]:     self.wsgi = self.app.wsgi()
2016-05-10T10:40:33.239882+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/base.py", line 114, in wsgi
2016-05-10T10:40:33.239882+00:00 app[web.1]:     self.callable = self.load()
2016-05-10T10:40:33.239883+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 62, in load
2016-05-10T10:40:33.239883+00:00 app[web.1]:     return self.load_wsgiapp()
2016-05-10T10:40:33.239884+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load_wsgiapp
2016-05-10T10:40:33.239884+00:00 app[web.1]:     return util.import_app(self.app_uri)
2016-05-10T10:40:33.239885+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/util.py", line 354, in import_app
2016-05-10T10:40:33.239885+00:00 app[web.1]:     __import__(module)
2016-05-10T10:40:33.239886+00:00 app[web.1]:   File "/app/gratipay/main.py", line 6, in <module>
2016-05-10T10:40:33.239886+00:00 app[web.1]:     import gratipay.wireup
2016-05-10T10:40:33.239887+00:00 app[web.1]:   File "/app/gratipay/wireup.py", line 32, in <module>
2016-05-10T10:40:33.239888+00:00 app[web.1]:     from gratipay.models.account_elsewhere import AccountElsewhere
2016-05-10T10:40:33.239888+00:00 app[web.1]:   File "/app/gratipay/models/account_elsewhere.py", line 17, in <module>
2016-05-10T10:40:33.239889+00:00 app[web.1]:     from gratipay.security.crypto import constant_time_compare
2016-05-10T10:40:33.239890+00:00 app[web.1]:   File "/app/gratipay/security/crypto.py", line 9, in <module>
2016-05-10T10:40:33.239891+00:00 app[web.1]:     from cryptography.fernet import Fernet, MultiFernet
2016-05-10T10:40:33.239891+00:00 app[web.1]: ImportError: No module named cryptography.fernet
2016-05-10T10:40:33.240346+00:00 app[web.1]: 2016-05-10 10:40:33 [8] [INFO] Worker exiting (pid: 8)
2016-05-10T10:40:33.383017+00:00 app[web.1]: 2016-05-10 10:40:33 [3] [INFO] Shutting down: Master
2016-05-10T10:40:33.383423+00:00 app[web.1]: 2016-05-10 10:40:33 [3] [INFO] Reason: Worker failed to boot.
2016-05-10T10:40:34.238289+00:00 heroku[web.1]: State changed from up to crashed
2016-05-10T10:40:34.220789+00:00 heroku[web.1]: Process exited with status 3
chadwhitacre commented 8 years ago

I believe we're pegged to an old buildpack, though ...

Yup:

[gratipay] $ heroku config -s | grep BUILDPACK
BUILDPACK_URL='https://github.com/heroku/heroku-buildpack-python.git#v54'
chadwhitacre commented 8 years ago

So I guess the thing to try is upgrading the buildpack. However, it'd be nice to know why the failure failed to abort the deployment.

chadwhitacre commented 8 years ago

Here's the archived build log.

chadwhitacre commented 8 years ago

it'd be nice to know why the failure failed to abort the deployment.

Seems likely to be related to using an old buildpack as well.

[gratipay] $ heroku config:set BUILDPACK_URL="https://github.com/heroku/heroku-buildpack-python.git#v80"
Setting BUILDPACK_URL and restarting ⬢ gratipay... done, v1594
BUILDPACK_URL: https://github.com/heroku/heroku-buildpack-python.git#v80
[gratipay] $
chadwhitacre commented 8 years ago

Alright, the build failed ... and succeeded to fail. :-)

remote:            copying cffi/model.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/ffiplatform.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/gc_weakref.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/recompiler.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/__init__.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/api.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/cparser.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/lock.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/vengine_gen.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/commontypes.py -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/_cffi_include.h -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/parse_c_type.h -> build/lib.linux-x86_64-2.7/cffi
remote:            copying cffi/_embedding.h -> build/lib.linux-x86_64-2.7/cffi
remote:            running build_ext
remote:            building '_cffi_backend' extension
remote:            creating build/temp.linux-x86_64-2.7
remote:            creating build/temp.linux-x86_64-2.7/c
remote:            gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/app/.heroku/python/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
remote:            c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
remote:             #include <ffi.h>
remote:                             ^
remote:            compilation terminated.
remote:            error: command 'gcc' failed with exit status 1
remote:            
remote:            ----------------------------------------
remote:        Command "/app/.heroku/python/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-9VEW36-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-xJda06-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-9VEW36-build/
remote: 
remote:  !     Push rejected, failed to compile Python app
remote: 
remote: Verifying deploy...
remote: 
remote: !       Push rejected to gratipay.
remote: 
To git@heroku.com:gratipay.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'git@heroku.com:gratipay.git'
[gratipay] $

What gives?

chadwhitacre commented 8 years ago

Hypothesis: the whitespace prefixing cffi is causing the buildpack not to see it.

chadwhitacre commented 8 years ago

Oooh, we have ./vendor/ in there, too. Here's pip-grep.

chadwhitacre commented 8 years ago

Hmm ... indeed, I don't see "Noticed cffi. Bootstrapping libffi." in the build log. I've cloned the buildpack locally to see about pip-grep. Correcting for https://github.com/whit537/heroku-buildpack-python/pull/2, pip-grep indeed does not find cffi in our requirements file.

$ ../../heroku/heroku-buildpack-python/vendor/pip-pop/pip-grep requirements.txt cffi
Not found.
$

I'm not sure it finds anything:

$ ../../heroku/heroku-buildpack-python/vendor/pip-pop/pip-grep requirements.txt cryptography
Not found.
$
chadwhitacre commented 8 years ago

Yeah, it's the local file paths that are confusing it. If I add cffi on a line by itself ...

$ ../../heroku/heroku-buildpack-python/vendor/pip-pop/pip-grep requirements.txt cffi
Package cffi found!
$
chadwhitacre commented 8 years ago

So the fix is to teach pip-grep about local file paths, and the workaround is to manually do the libffi bootstrapping.

chadwhitacre commented 8 years ago

Alright, fix reticketed as https://github.com/heroku/heroku-buildpack-python/issues/299. Now for the workaround ...

chadwhitacre commented 8 years ago

the workaround is to manually do the libffi boostrapping.

... or maybe to include cffi at the end of requirements.txt?

chadwhitacre commented 8 years ago

Harumph. Same result with just cffi. 😕 

Processing ./vendor/jmespath.py-0.9.0.tar.gz
Processing ./vendor/botocore-1.4.13.tar.gz
Processing ./vendor/boto3-1.3.0.tar.gz
Processing ./vendor/pycparser-2.14.tar.gz
Processing ./vendor/cffi-1.6.0.tar.gz
Processing ./vendor/enum34-1.1.4.tar.gz
Processing ./vendor/idna-2.1.tar.gz
Processing ./vendor/ipaddress-1.0.16.tar.gz
Processing ./vendor/cryptography-1.3.2.tar.gz
Collecting cffi==1.6.0 (from -r requirements.txt (line 67))
  Could not find a version that satisfies the requirement cffi==1.6.0 (from -r requirements.txt (line 67)) (from versions: )
No matching distribution found for cffi==1.6.0 (from -r requirements.txt (line 67))
make: *** [env] Error 1
$
chadwhitacre commented 8 years ago

Gah. Do we have to fork the buildpack? 😞

chadwhitacre commented 8 years ago

Forked to https://github.com/gratipay/buildpack-python, with a workaround on the gratipay-prod branch.

$ heroku config:set BUILDPACK_URL="https://github.com/gratipay/buildpack-python.git#gratipay-prod"
Setting BUILDPACK_URL and restarting ⬢ gratipay... done, v1595
BUILDPACK_URL: https://github.com/gratipay/buildpack-python.git#gratipay-prod
$
chadwhitacre commented 8 years ago 
remote: -----> Noticed cffi. Bootstrapping libffi.

👍

chadwhitacre commented 8 years ago

Blam. Okay!

screen shot 2016-05-10 at 8 27 44 am

chadwhitacre commented 8 years ago

Aaaaaand we have crypto! 💃 😳 🎏

chadwhitacre commented 8 years ago

@clone1018 Heads up that payday may blow up tomorrow because we're (planning on) introducing big changes today: https://github.com/gratipay/gratipay.com/issues/3994. I'll plan to be highly available during payday tomorrow, even though I'm not on this month, in order to help with any explosions that may occur. :)

cc: @rohitpaulk

chadwhitacre commented 8 years ago

Deploying gratipay/gratipay.com#4027 ...

clone1018 commented 8 years ago

@whit537 :+1:

chadwhitacre commented 8 years ago

Deploying gratipay/gratipay.com#4028 ...

chadwhitacre commented 8 years ago

Hmm ... deployment looks like it worked, but we're still seeing 1956 ...

screen shot 2016-05-11 at 5 01 41 pm

chadwhitacre commented 8 years ago

Heroku claims to have deployed https://github.com/gratipay/gratipay.com/commit/3910e3ad8fed9c3d01278bb8bce58023c1f04721, which is the 1957 version bump.

screen shot 2016-05-11 at 5 05 06 pm

chadwhitacre commented 8 years ago

heroku restart didn't fix it. 😳

chadwhitacre commented 8 years ago

I'm trying to inspect the git log on the heroku remote, to no avail, despite these SO answers:

http://stackoverflow.com/questions/13941976/commit-history-on-remote-repository http://stackoverflow.com/questions/26349191/fatal-ambiguous-argument-branch-name-both-revision-and-filename http://stackoverflow.com/questions/25559691/git-compare-local-branch-to-remote-staging-and-production

[gratipay] $ git log --oneline heroku/master..master
fatal: ambiguous argument 'heroku/master..master': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
[gratipay] $ git log heroku/master
fatal: ambiguous argument 'heroku/master': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
[gratipay] $ git log heroku/master --
fatal: bad revision 'heroku/master'
[gratipay] $ git log -- heroku/master
[gratipay] $
chadwhitacre commented 8 years ago 
[gratipay] $ git remote -v | grep heroku
heroku  git@heroku.com:gratipay.git (fetch)
heroku  git@heroku.com:gratipay.git (push)
[gratipay] $
chadwhitacre commented 8 years ago

Looking into the build logs at Heroku ...

chadwhitacre commented 8 years ago 
[gratipay] $ diff -rub working.log borked.log 
--- working.log 2016-05-11 17:14:46.000000000 -0400
+++ borked.log  2016-05-11 17:15:10.000000000 -0400
@@ -55,5 +55,5 @@
 -----> Compressing...
        Done: 66.2M
 -----> Launching...
-       Released v1597
+       Released v1598
        https://gratipay.herokuapp.com/ deployed to Heroku
[gratipay] $
chadwhitacre commented 8 years ago

Maybe another release will get us back on track? 😞 💠

chadwhitacre commented 8 years ago

😕

screen shot 2016-05-11 at 5 18 36 pm

chadwhitacre commented 8 years ago

Okay! Sunspots!

chadwhitacre commented 8 years ago

Deploying gratipay/gratipay.com#4031 ...

chadwhitacre commented 8 years ago

👍

screen shot 2016-05-11 at 5 40 11 pm

chadwhitacre commented 8 years ago

Deploying https://github.com/gratipay/gratipay.com/pull/4032 ...

chadwhitacre commented 8 years ago

👍

screen shot 2016-05-11 at 5 50 43 pm

chadwhitacre commented 8 years ago

Deploying https://github.com/gratipay/gratipay.com/pull/4033 ...

chadwhitacre commented 8 years ago

On my own now, @kaguillera headed out for the bus ... 🚌

chadwhitacre commented 8 years ago

👍

screen shot 2016-05-11 at 6 15 55 pm

chadwhitacre commented 8 years ago

Deploying https://github.com/gratipay/gratipay.com/pull/4006 ...

chadwhitacre commented 8 years ago

👍

screen shot 2016-05-11 at 7 10 22 pm

chadwhitacre commented 8 years ago

Empty! :-P

screen shot 2016-05-11 at 7 10 59 pm