Update deployment pipeline for new OHM-Labs org repos

[danrademacher]

Moving repos to this org has broken some things on staging:

1. This commit points to corrected GH Pages resources: https://github.com/Open-Historical-Map-Labs/ohm-website/commit/e301d5c6aae6b62db18f3f6789644b729182e935, need to get deployed on AWS
2. But that means updating the git submodule for ohm-website and then changingCode-Deploy to point at this repo.

this might be simple.

[danrademacher]

Also should be assigned to @batpad, but his invitation is still pending.

• [x] See this PR for proposed change to Git Submodule -- https://github.com/Open-Historical-Map-Labs -- can you test locally to make sure that's what we want before merging?

• [x] Sanjay and I tried editing the CodePipeline connection to Github here, https://console.aws.amazon.com/codesuite/codepipeline/pipelines/website/view?region=us-east-1, but the Github connection seems to be failing despite the payload seeming correct over at https://github.com/Open-Historical-Map-Labs/openhistoricalmap-docker/settings/hooks/88477917

[danrademacher]

Hmm, I did just have to press "Grant" here:

Pushed this empty commit again https://github.com/Open-Historical-Map-Labs/openhistoricalmap-docker/commit/f1288e7b37e49ab1aa4bc7e4c7d4f127f6f7f0a2, to test.

[danrademacher]

Hmm, that didn't seem to cause the Pipeline to try to run... though we do have a multiple payloads showing as run this AM:

[danrademacher]

Hmm, all my empty commits failed to cause the Pipeline to run BUT clicking this button:

made it happen:

We shouldn't have to click that button every time, but at least we CAN click that button!

Gotta run, but will check back shortly to see if its working after build and deploy go through

[danrademacher]

Hmm, well I dislodged the Code Pipeline and got that flowing again. Now we have CORS issues with resources from Github Pages... but a step closer to running!

I’m phone only and was checking by tempting into my Mac ... Which means no screenshot of errors. Will post in a bit

[danrademacher]

Ok, so we have the staging site working again, but the deploys still only happen when I press that big orange "Release Change" button, so not exactly CI/CD. I think it must still be the webhooks must still somehow be tied up with older repo URL or something

[geohacker]

@danrademacher thank you getting us so close! I think I managed to resolve this trigger issue -- now any commit to master will trigger a CodePipeline build.

This is how I approached the issue:

• Inspected the Github hooks. And at the time of inspection, saw that there were two hooks with different destinations. Both of them were firing events and getting a 200 success response. So that seems all ok.
• Pushed an empty commit to see the logs. The payload seemed correct, but the pipeline didn't kickoff.
• Triggered the job manually from the CodePipeline console, and that worked.
• So I was pretty certain this is a Github hook issue. Start looking up docs to inspect webhooks. Came across this page that said when we connect to Github it'll mention a webhook will be registered.
• Then I tried to auth with Github again to see if it would create a new hook. It did not.
• I thought perhaps it didn't because there are already webhooks, so I removed the existing ones from Github. Tried to auth again, it didn't create hooks in Github this time as well.
• Started looking for documentation on Github integration, and realised that my default Github access tokens may not have admin:web_hook. Stumbled on this page to create a new token and update the CodePipeline via the AWS CLI. Did that.
• Tried the Github integration again, still no sign of a hook on Github.
• Then I listed all the webhooks in our account in us-east-1 using this command: aws --profile openhistoricalmap codepipeline list-webhooks --endpoint-url "https://codepipeline.us-east-1.amazonaws.com" --region "us-east-1"
• This revealed that the webhook was indeed created, but it wasn't wired up with Github properly.
• Using the secret token and the URL, I manually added the webhook through the Github Webhook page.
• Tested with another commit and 🎉

I'm pretty sure this is some kind of an edge case were the hook gets created but it's not registered properly with Github. The token stuff shouldn't be required in case we run into this again.

@danrademacher I'll let you verify/test this with a commit and let us know next steps. cc @batpad

[danrademacher]

Confirmed working here as well!

(side note, I have notifications set up and feeding into a #github-ohm channel in our GreenInfo slack, which is the source of the above screenshot

[danrademacher]

Hmm, not so fast:

[danrademacher]

Link to the failure message, scroll way down for : https://console.aws.amazon.com/codesuite/codebuild/projects/website/build/website%3A0c953fd2-6813-4a33-9fb4-b86472de7ff9/log?region=us-east-1

Weird. I assume this is related, but the failure mode is quite different. Any ideas on what's going on?

Can you try and replicate?

[geohacker]

Ha, that's weird indeed. Looks like npm install fails with:

·[91mnpm ERR! code·[0m·[91m EEXIST 
npm ERR! syscall symlink 
·[0m·[91mnpm ERR! path ../lib/node_modules/jpegtran-bin/cli.js 
npm ERR! dest /usr/bin/jpegtran 
·[0m·[91mnpm ERR! ·[0m·[91merrno -17 
·[0m·[91mnpm ERR!·[0m·[91m EEXIST: file already exists, symlink '../lib/node_modules/jpegtran-bin/cli.js' -> '/usr/bin/jpegtran' 
npm ERR! File exists: /usr/bin/jpegtran 
·[0m·[91mnpm ERR!·[0m·[91m Remove the existing file and try again, or run npm 
npm ·[0m·[91mERR! with --force to overwrite files recklessly. 
·[0m·[91m 
·[0m·[91mnpm ERR! A complete log of this run can be found in: 
npm ERR!     /root/.npm/_logs/2019-12-12T01_50_57_517Z-debug.log 

I'm going to send the hook again to see if it was just a temporary hiccup.

[geohacker]

This is a bit mysterious. It didn't recover by itself. I tried building the image locally and that worked just fine.

A thing to check would if the AWS AMI or something has changed? cc @batpad if you have any clues.

[danrademacher]

Discussion today with Sajjad -- must be some issue with alignment of Docker and the AMI, since this error doesn't occur locally. Sajjad will find AMI id and investigate more closely tomorrow and post notes here.

[geohacker]

So few more thoughts as I look at this:

• It comes from this npm install line https://github.com/Open-Historical-Map-Labs/openhistoricalmap-docker/blob/master/website/Dockerfile#L70
• It looks like jpegtran is already there in /usr/bin
• It's very suspicious why npm is creating a symlink to /usr/bin than something like /usr/local/bin
• I'm starting to also feel unsure about this block https://github.com/Open-Historical-Map-Labs/openhistoricalmap-docker/blob/master/website/Dockerfile#L56-L60 of npm install code

I've confirmed that:

• Nothing has changed in the ruby 2.3 image we are using as the base docker image https://hub.docker.com/layers/ruby/library/ruby/2.3/images/sha256-0fd396e8d09d9b59bcb5d34159ed82ee48dae7b0668989e8940a7cca8df55401
• Nothing has changed with the jpegtran-bin npm module https://github.com/imagemin/jpegtran-bin
• The codebuild docker image used aws/codebuild/docker:18.09.0 -- nothing has changed here either.
• Good news: we can now reproduce this locally

[geohacker]

Build works now after merging #25. @danrademacher let me know if you get a chance to test and everything looks ok.

[danrademacher]

Just ran and confirmed the build pipeline is fixed for me too