search

nasa-petal / PeTaL-labeller

The PeTaL labeler labels journal articles with biomimicry functions.
https://petal-labeller.readthedocs.io/en/latest/
The Unlicense
6 stars 4 forks source link

Mock up the labeller #9

Closed bruffridge closed 3 years ago

bruffridge commented 3 years ago

Write a python script to perform the following. Eventually the plan is to run this python code in this lambda function:

  1. Download the sample CORE data dump file from here. Eventually this will need to use the full data dump, but probably easiest to start with the sample since the full dump is ~400gb and 1.8tb once fully extracted. For more information about the format of these dump files see the CORE documentation.
  2. Extract contents
  3. Process all the individual .json files containing information about individual CORE articles, and assign a randomly selected label to them from the Biomimicry Functions list in GDrive. You can just store the functions list in a python data structure, you don't have to write code to extract it from the google sheet. This code will go away once the labeller is working.
  4. Save labelled data to PetalLabels dynamodb table using the fields described here
bruffridge commented 3 years ago

duplicate of #45