search

microbiomedata / issues

public repo for issues related to NMDC work
1 stars 0 forks source link

Milestone - Support for submission of data and metadata from NMDC to international repositories (e.g. NCBI) (1.11) #445

Open ssarrafan opened 9 months ago

ssarrafan commented 9 months ago

Workflow Data Management The files and metadata generated by the NMDC workflows are available to the community in the Data Portal. In the Pilot we developed a manual metadata ingestion (Figure 4, C) process for completed workflow outputs, and this process must be automated in order for the NMDC to scale. The current tasks to be automated include post-processing workflow outputs to generate required entry for the schema and schema validation before the data finally becomes available to users. Additionally, production NMDC data must be automatically backed up at regular intervals (Milestone 1.10). Further, we plan to support submission of this validated data to different repositories to more fully support the research community and lower interoperability barriers (e.g., NCBI or other primary repositories) (Milestone 1.11). The two User Facilities routinely provide data submission to primary repositories as an important service to the community as part of their operations, and the NMDC will similarly provide this service as needed for data that is outside the User Facilities processes.

Page 27 Revised due date is FY24 Q4

aclum commented 6 months ago

We have info from NCBI on Nov 22 2023 on how to do ui-less submissions. We need to test this.

ssarrafan commented 6 months ago

From quarterly report: Had one in person and two virtual meetings with NCBI staff to discuss logistics for the NMDC to serve as a data broker for data submissions (e.g., a “trusted partner”). The NMDC team has instructions on API submission to NCBI and will evaluate and test in FY24 Q2 focusing on submitting NEON data.

moving to Q2 for more updates

aclum commented 5 months ago

This is expected to be delayed in favor of data ingest for GSP 2024.

ssarrafan commented 5 months ago

This is expected to be delayed in favor of data ingest for GSP 2024.

Should this be moved to Q3 or to a later quarter in 2025?

aclum commented 4 months ago

That will depend on what the post-GSP priorities are. Best to discuss at the weekly leadership meeting @ssarrafan

aclum commented 3 months ago

Squad work for this is planned to start April 22, squad board https://github.com/orgs/microbiomedata/projects/125

aclum commented 3 weeks ago

The squad that is working on this is actively in progress. Testing was performed in FY24 Q3 as expected. The MVP for this feature will be considered complete when https://github.com/microbiomedata/issues/issues/744 is complete.

ssarrafan commented 1 day ago

Is there a plan to have this and issue #744 done by September? @aclum @shreddd