Test DUOS in ingest UI #999

[arschat]

Description of the task:

List and test a number of different cases that we might encounter with the usage of Data use restriction & DUOS ID, in ingest UI described in #999. Feature is deployed in staging, where the test can be done.

First comment will include the list of cases, and in following comments a step by step reproduction of the case and test result should be included.

Acceptance criteria for the task:

• [x] List all available cases
• [x] Test all available cases
• [ ] Resolve issues that occurred during tests

[arschat]

List of available cases

• Create
• [x] create an NRES project with no DOI & no accession - test
• [x] create an GRU project with no DOI & no accession - test
• [x] create an GRU-NCU project with no DOI & no accession - test
• [x] create an NRES project from DOI - test
• [x] create an GRU project from DOI - test
• [x] create an GRU-NCU project from DOI - test
• [ ] create an NRES project from accession
• [ ] create an GRU project from accession
• [ ] create an GRU-NCU project from accession

  create an NRES project from accession fails but that fails in prod as well. GEO to HCA script is known to have problems and needs to be updated so we can skip these tests.

• Update
  • from UI
• [x] update project from NRES -> GRU
• [x] update project from NRES -> GRU-NCU
• [x] update project from GRU -> NRES
• [x] update project from GRU-NCU -> NRES
• [x] update project from GRU -> GRU-NCU
• [x] update project from GRU-NCU -> GRU
• [x] update DUOS-ID same data_use_restriction
• [x] update DUOS-ID and GRU -> GRU-NCU
  • through api
• [X] update project from NRES -> GRU
• [X] update project from GRU -> NRES
• [X] update project to GRU without DUOS-ID
• Bump & update
• [x] update project from project version 18.0.0 to NRES
• [x] update project from project version 17.1.1 to NRES
• [x] update project from project version 18.0.0 to GRU
• [x] update project from project version 17.1.1 to GRU
• [x] update project from GRU to project version 18.0.0
• [x] update project from GRU to project version 17.1.1

[arschat]

Create

create an NRES project with no DOI & no accession

• register new project
• added NRES as data_use_restriction
• save
• verify NRES in api
• Upload minimum_test.xlsx to create submission
• download spreadsheet and verify the field

Test: Pass

create an GRU project with no DOI & no accession

• register new project
• added GRU as data_use_restriction
• save
• verify GRU and DUOS-ID in api
• Upload minimum_test.xlsx to create submission
• download spreadsheet and verify the field

Test: Pass

create an GRU-NCU project with no DOI & no accession

• register new project
• added GRU-NCU as data_use_restriction
• save
• verify GRU-NCU and DUOS-ID in api
• Upload minimum_test.xlsx to create submission
• download spreadsheet and verify the field

Test: Pass

create an NRES project from DOI

• register new project from DOI 10.1016/j.cell.2024.03.009
• added NRES as data_use_restriction
• save
• verify NRES in api
• Upload minimum_test.xlsx to create submission
• download spreadsheet and verify the field

Test: Pass

create an GRU project from DOI

• register new project from DOI 10.1084/jem.20210815
• added GRU as data_use_restriction
• save
• verify GRU and DUOS-ID in api
• Upload minimum_test.xlsx to create submission
• download spreadsheet and verify the field

Test: Pass

create an GRU-NCU project from DOI

• register new project from DOI 10.1038/s41598-024-61485-6
• added GRU-NCU as data_use_restriction
• save
• verify GRU-NCU and DUOS-ID in api
• Upload minimum_test.xlsx to create submission
• download spreadsheet and verify the field

Test: Pass

[arschat]

create an NRES project from accession fails but that fails in prod as well. GEO to HCA script is known to have problems and needs to be updated so we can skip these tests.

[arschat]

Update

update project from NRES -> GRU in UI

• register new project
• added NRES as data_use_restriction
• save
• verify NRES in api
• Updated data_use_restriction & added duos-id from UI
• it return an error for no data in the publication info although no publication was added in the initial project creation. after removing publication from UI update completed. It probably has to do with editing projects with empty publications rather than the DUOS-ID in UI

Test: pass

update project from GRU-NCU -> NRES in UI

• register new project
• added GRU-NCU as data_use_restriction
• save
• verify GRU in api
• Updated data_use_restriction & added duos-id from UI
• same error with empty publication, after removing fields in UI, it updated completely

Test: pass

update project from GRU -> GRU-NCU

• starting from project TestGRU-UI-updateNRES
• Through ingest UI, updated the GRU to GRU-NCU
• same error with empty publication, after removing fields in UI, it updated completely

Test: Pass

update DUOS-ID same data_use_restriction

• continue from project TestGRU-UI-updateNRES
• Through ingest UI, updated the DUOS-Id from DUOS-123456 to DUOS-000000
• same error with empty publication, after removing fields in UI, it updated completely

update DUOS-ID and GRU-NCU -> GRU

• continue from project TestGRU-UI-updateNRES
• Through ingest UI, updated the DUOS-Id from DUOS-000000 to DUOS-123456 and GRU
• same error with empty publication, after removing fields in UI, it updated completely

update project from NRES -> GRU through api

• register new project
• added NRES as data_use_restriction
• save
• verify NRES in api
• Updated data_use_restriction & added duos-id from postman patching the following query that was copied from the previous json state.

Query

> { "content": { "describedBy": "https://schema.staging.data.humancellatlas.org/type/project/19.0.0/project", "schema_type": "project", "project_core": { "project_short_name": "TestNRES-api-updateGRU", "project_title": "Arsenios Test NRES update api GRU", "project_description": "Arsenios Test NRES update api GRU" }, "contributors": [ { "name": "Arsenios,,Chatzigeorgiou", "email": "arsenios@ebi.ac.uk", "institution": "Department of OMNI Bioinformatics, Genentech, South San Francisco, CA 94080, USA", "corresponding_contributor": true } ], "funders": [ { "grant_id": "unspecified", "organization": "NCI NIH HHS" } ], "data_use_restriction": "GRU", "duos_id": "DUOS-123456" } }

Test: pass

update project from GRU -> NRES through api

• continue from previous project api
• Updated data_use_restriction & added duos-id from postman patching the following query that was copied from the previous json state.

Query

> { "content": { "describedBy": "https://schema.staging.data.humancellatlas.org/type/project/19.0.0/project", "schema_type": "project", "project_core": { "project_short_name": "TestNRES-api-updateGRU", "project_title": "Arsenios Test NRES update api GRU", "project_description": "Arsenios Test NRES update api GRU" }, "contributors": [ { "name": "Arsenios,,Chatzigeorgiou", "email": "arsenios@ebi.ac.uk", "institution": "Department of OMNI Bioinformatics, Genentech, South San Francisco, CA 94080, USA", "corresponding_contributor": true } ], "funders": [ { "grant_id": "unspecified", "organization": "NCI NIH HHS" } ], "data_use_restriction": "NRES", } }

Test: pass

update project to GRU without DUOS-ID through api

• following the previous manner added

      "data_use_restriction": "GRU"

  without the DUOS-ID

• we get an error should have required property 'duos_id and in the UI the field is shown but is unspecified

Test: Pass

[arschat]

Bump

update project to have 17.1.1 project_schema version

• created NRES project with no DOI
• through api, removed the data_use_restriction field and downgraded the project to 17.1.1 with the following query
  query

{ "content": { "describedBy": "https://schema.staging.data.humancellatlas.org/type/project/17.1.1/project", "schema_type": "project", "project_core": { "project_short_name": "TestDowngradeProjectVersion", "project_title": "test downgrade project schema to 17.1.1", "project_description": "test downgrade project schema to 17.1.1" }, "contributors": [ { "name": "Arsenios,,Chatzigeorgiou", "email": "arsenios@ebi.ac.uk", "institution": "EMBL-EBI", "corresponding_contributor": true } ], "funders": [ { "grant_id": "82030097", "organization": "NCI NIH HHS" } ] } } Test: Pass

update project to NRES from project schema version 17.1.1

• created new project
• downgraded the project schema version to 17.1.1 like the previous test
• verified in api and in UI
• bumped the project schema version to 19 & assigned data_use_restriction with NRES
  query

{ "content": { "describedBy": "https://schema.staging.data.humancellatlas.org/type/project/19.0.0/project", "schema_type": "project", "project_core": { "project_short_name": "TestNRESfrom17-1-1", "project_title": "Arsenios Test DUOS Bump from 17.1.1 to NRES", "project_description": "Arsenios Test DUOS Bump from 17.1.1 to NRES" }, "contributors": [ { "name": "Arsenios,,Chatzigeorgiou", "email": "arsenios@ebi.ac.uk", "institution": "Department of OMNI Bioinformatics, Genentech, South San Francisco, CA 94080, USA", "corresponding_contributor": true } ], "funders": [ { "grant_id": "2019LJ002", "organization": "lefta" } ], "data_use_restriction": "NRES" } }

Test: Pass

update project to GRU from project schema version 18.0.0

• created project
• downgraded the project schema version to 18.0.0 like the previous test (using data_use_restriction as well)
• verified in the api
• bumped the project schema version to 19 & assigned data_use_restriction with GRU and DUOS-ID

query

> { "content": { "describedBy": "https://schema.staging.data.humancellatlas.org/type/project/18.0.0/project", "schema_type": "project", "project_core": { "project_short_name": "TestGRUfrom18-0-0", "project_title": "Arsenios Test DUOS Bump from 18.0.0 to GRU", "project_description": "Arsenios Test DUOS Bump from 18.0.0 to GRU" }, "contributors": [ { "name": "Arsenios,,Chatzigeorgiou", "email": "arsenios@ebi.ac.uk", "institution": "EMBL-EBI", "corresponding_contributor": true } ], "funders": [ { "grant_id": "unspecified", "organization": "lefta" } ], "data_use_restriction": "GRU", "duos_id": "DUOS-123456" } }

Test: Pass

[arschat]

Two very minor bugs found:

• if project does not have a publication, editing a project will return an error with empty publication fields and we need to go to publication tab, and select "Remove"
• when editing a project that already has a DUOS-ID, the DUOS-ID is not editable

  unless we pick another option from the data_use_restriction and then it is editable (and then we can select again the original option and edit the duos-id)

  

[arschat]

In the interest of finding the different cases that we might come up in backfilling, I investigated the project_schema versions in ingest.

from hca_ingest.api.ingestapi import IngestApi from collections import Counter

api = IngestApi(url="https://api.ingest.archive.data.humancellatlas.org/") api.set_token(f"bearer {token}")

all_url = 'https://api.ingest.archive.data.humancellatlas.org/projects/' all_projects = api.get_all(all_url, 'projects')

versions = [] for project in all_projects: versions.append(projects['content']['describedBy'][47:53])

Counter(versions)

The output was:

'19.0.0': 13 '18.0.0': 10 '17.1.1': 205 '17.1.0': 5 '17.0.0': 888 '14.1.0': 1 'type/p': 1

• type/p is from a project with the project version being from staging enviorment
• 14.1.0 is from the project 5116c081-8be7-49c5-8ce0-73b887328aa9

Therefore, in case of bump version from 17 and above, we only need to change the schema version and add the data_use_restriction and duos-id if needed. In case of bump from version 14.1.0 we need to fill official_hca_publication field as well.