Move 23andMe uploads to own oauth2 project

[gedankenstuecke]

• [x] Started repo at OpenHumans/oh-23andme-source
• [x] Export data validation/conversion logic into stand-alone oauth2 project
• [x] Finalize integration to delete unprocessed file, etc.
• [x] Write copy for the project (is ftdna-specific right now)
• [x] Deploy external project
• [x] Wrote management task in oh-23andme-source repo to import users from old legacy activity
• [x] shutdown legacy integration
• [x] Generate oauth2 tokens for existing legacy members
• [x] Alter foreign-keys of existing data files to stand-alone project and create the additional direct/private/whatever sharing objects in the main site backend
• [x] Import newly created tokens into stand-alone project
• [x] Activate new project on main site

We want to move away from having some data sources integrated right on the main site as it's a PITA to maintain and leads to all sorts of problems. We've roadmapped out the general workflow and file-based uploads should be a good start to test it out.

[gedankenstuecke]

Memo for me: The processing code used so far can be found in https://github.com/OpenHumans/open-humans-data-processing/blob/master/sources/twenty_three_and_me

[gedankenstuecke]

I got the current processing code updated to py3 to be used in the stand-alone deployment. It still needs to be refactored and there are no tests so far (which is the same for the old integrated version)

[gedankenstuecke]

• The processing is now complete, it creates both the cleaned 23andMe format as well as the VCF.
• The unprocessed file is now being deleted no matter what happens
• If the task crashes the file is deleted too and the user is informed via email.
• I added some minimal tests, but it's not comprehensive yet

[gedankenstuecke]

• Thanks to @madprime the zip/bz2/gz parsing works now too
• BASE_URL can now be changed to more easily test against staging
• Uses the (by now finally fixed) ohapi AWS S3 method
• Extended the test coverage to make sure that the main processing job runs as expected with valid input

[gedankenstuecke]

• Wrote a management task to re-queue all files that haven't been processed so far
• Deployed the app on the OH heroku account at https://oh23andmesource.herokuapp.com and tested it

IMPORTANT NOTE: If you're an existing 23andMe integration user you don't have to upload your data again, we will merge this into a single application in the next steps!

[madprime]

Plan for wrapping up:

1. add code to make legacy activity invisible/inactive
2. merge all code to deactivate legacy activity & add management command
3. run management command to transfer users
4. activate new project so it's listed

[madprime]

Add management command to make CSV dump of all project member IDs + refresh tokens.

[gedankenstuecke]

The oh-23andme-source now has a simple import function that takes project_member_ids and refresh_tokens from a CSV to import the existing users in the standalone project.

[gedankenstuecke]

@madprime Do you agree that all things here are taken care of as checked off above and we can successfully close the first step in moving over our legacy projects? 🙏

[gedankenstuecke]

And done! 👍