Patents Pipeline

[india-kerle]

---

Description

This PR contains: 1) An initial analysis of the # of patents in google bigquery between 2016-2021 where at least one inventor is from the UK; 2) The full, initial patents pipeline of querying google bigquery for patents data, cleaning it and saving it to s3.

This PR closes:

19

3

4

Instructions for Reviewer

To pull data from google bigquery:

python dap_aria_mapping/pipeline/data_collection/patents.py run

BEWARE!!! If production=true, this will cost money! I accidentally charged myself £5.36 for the full dataset. I've since changed cards to be nesta's business card but lets not run it in production unless it feels like we've got to re-collect data.

To post process the data:

python dap_aria_mapping/pipeline/data_collection/processed_patents.py run

This reformats all the date columns, extracts english titles and abstracts and unnests the assignee and inventor to be in a list of strings format instead of a list of dict format.

It also creates a look up table sam will need to extract entities.

In order to test the code in this PR you need to ...

pytest dap_aria_mapping/pipeline/data_collection/tests/test_patents.py

To make sure that the clean patents dataset:

• Has a unique publication_number AND family_id per row;
• Each patent document's filing date is between 01-01-2016 and 31-12-2021;
• Has at least 1 inventor based in the UK associated to the invention

pytest dap_aria_mapping/pipeline/data_collection/tests/test_processed_patents.py To test the unnest_column and extract_english_text processing functions

Please pay special attention to ...

running python dap_aria_mapping/pipeline/data_collection/patents.py run in production - it will cost money!

Checklist:

• [ ] I have refactored my code out from notebooks/
• [x] I have checked the code runs
• [x] I have tested the code
• [x] I have run pre-commit and addressed any issues not automatically fixed
• [x] I have merged any new changes from dev
• [x] I have documented the code
  • [x] Major functions have docstrings
  • [ ] Appropriate information has been added to READMEs
• [x] I have explained this PR above
• [x] I have requested a code review

[Jack-Vines]

Side note: this is gonna cause merge issues with my branch, but once this is merged i'll fix them on my branch, then re-request a review