Update gold:Gs0110138 / nmdc:sty-11-33fbta56 SPRUCE (metagenomes, NOM, proteomics, metatranscritpomes)

[mbthornton-lbl]

Omics Type(s): metagenomes, metatranscriptomes, proteomics, NOM

Steps: Missing Data

• [x] load biosample_set.json
• [x] load omics_processing_set.json

Note: omics_processing was not loaded during testing, so we will need to re-run extract-records and process-records below QC data addition

• [x] 124 biosample_set records are returned with this find query {"part_of": "nmdc:sty-11-33fbta56"}
• [x] 16 omics_processing_set records returned with this find query {"omics_type.has_raw_value":"Metatranscriptome", "part_of": "nmdc:sty-11-33fbta56"}

Metagenomics - Note filesystem updates for SPRUCE are here: #175

• [x] update-study with updated_identifiers.tsv ~- [ ] extract-records~ ~- [ ] process-records no filesystem with updated_identifiers.tsv~
• [x] ingest-records nmdc:sty-11-33fbta56_re_ided_record_dump.json
• [x] delete-old-records nmdc:sty-11-33fbta56_associated_record_dump.json
• [x] delete-old-records nmdc:sty-11-33fbta56_failed_record_dump.json
• [x] re-add omics_processing_set.json

Proteomics:

• [x] add missing data_object_set records (proteomics raw data) nmdc.data_objects_set.spruce.metap.2024-04-30-01.json
• [x] changesheets to update omics_processing_set hasoutput records -depends on update-study [ changesheet tsv file](https://drive.google.com/file/d/167M6OKTUBblB4eGvB0cxwQaHdfYbHTz/view?usp=sharing)
• [x] metap_records_delete.py if not already run for gold:Gs0114663 - can be run before or after two steps above.

[mbthornton-lbl]

ingest-records failsL

ERROR:root:An error occurred - aborting transaction
ERROR:root:An error occurred while ingesting records: 500 Server Error: Internal Server Error for url: https://api-napa.microbiomedata.org/metadata/json:validate
Traceback (most recent call last):
  File "/Users/MBThornton/Documents/code/nmdc_automation/nmdc_automation/re_iding/scripts/./re_id_tool.py", line 731, in ingest_records
    _ingest_records(db_records, db_client, api_user_client)
  File "/Users/MBThornton/Documents/code/nmdc_automation/nmdc_automation/re_iding/scripts/./re_id_tool.py", line 757, in _ingest_records
    if api_user_client.validate_record(record):
  File "/Users/MBThornton/Documents/code/nmdc_automation/nmdc_automation/api/nmdcapi.py", line 410, in validate_record
    response.raise_for_status()
  File "/Users/MBThornton/Library/Caches/pypoetry/virtualenvs/nmdc-automation-VEpwcKpc-py3.9/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api-napa.microbiomedata.org/metadata/json:validate
INFO:root:Elapsed time: 3.8852598667144775

[ssarrafan]

Sprint over, removing from sprint.

[mbthornton-lbl]

Data Prerequisites for SPRUCE

nmdc:sty-11-33fbta56_biosample_set.json nmdc:sty-11-33fbta56_omics_processing_set.json nmdc:sty-11-33fbta56_updated_record_identifiers.txt

@aclum These are the data docs from the automation repository napa_reid_data_and_logs branch, the same as were used in the first, failed reid attempt. I am assuming we will be using these for a second run - could you please have a look at these and confirm?

[mbthornton-lbl]

nmdc:sty-11-33fbta56_prot_data_objects.json

[aclum]

Okay to close :)