Branch merging pipeline architecture (v1)

[dphiffer]

• [x] Basic user interface with a button that says "go", gather progress as a basic text log
• [x] Option to do a "dry run" of everything below
• [ ] Offline task to update admin area hierarchies
• [ ] Offline task to update venue hierarchies
• [x] Offline task to regenerate meta files
• [x] Do the git merge branch dance (should block other git tasks)
• [x] Attempt to push to GitHub (report failures properly)

[todrobbins]

I just wanted to stop by to give an encouraging word. You all are rock stars! Thank you for trying to solve the massive task that is WOF.

[dphiffer]

Thank you!! That is so good to hear :)

[thisisaaronland]

• [ ] create pending bucket in S3
• [ ] on upload, store file in pending S3 bucket
• [ ] add record to db queue
• [ ] determine what metadata goes in db queue in order to:
• [ ] what kind of task is being run: neighbourhoods, etc.
• [ ] prevent duplicate tasks
• [ ] schedule things so that two (or more) processes don't clobber each other's data
• [ ] what phase of the processing chain is happening: validation; scheduling; execution
• [ ] when to branch and from where (see above inre: scheduling)
• [ ] clean-up (files in S3) and send email

[thisisaaronland]

• [ ] words and sentences explaining why or why not to use the existing offline tasks toolchain

[thisisaaronland]

@dphiffer - Including the following as an FYI. Here I am manually merging two branches (pipeline should only do one branch at a time) in to staging-work and then forgetting to pull from master before pushing...

I am happy to be convinced that some of these steps are unnecessary but it's important that we ensure "break points" to be able to track down glitches and to roll back a branch/merge entirely.

cd /usr/local/data/whosonfirst-data
git checkout issue-784
git pull origin issue-784
git checkout staging-work
git pull origin issue-784
git push origin staging-work
git checkout issue-785
git pull origin issue-785
git checkout staging-work
git pull origin issue-785
git push origin staging-work
git checkout master
git pull origin staging-work
wof-build-metafiles
git commit -m "update metafiles" .
git push origin master
# MERGE CONFLICT BECAUSE METAFILES AND FORGETTING TO PULL FROM MASTER ABOVE...
git pull origin master
wof-build-metafiles
git commit -a
git push origin master