incorporate pdb accession code

[hongsudt]

Support of multiple Accession_Code in deriva

PDB:Accession_Code

• Model change

  • change Entry to nullok=True
  • add PDBDEV_Accession_Code (type = text, nullok=False, comment="Accession code issued by PDB-Dev, if availabe") and copy the current Accession_Code column over
  • add PDB_Accession_Code (type = text, nullok=False, comment="Accession code issued by PDB, if available"). This column dictates what PDB Accession Code is supposed to be. For now based on PDB policy, it is the PDB_Code, but will be PDB_Extended_Code in the future.
  • add PDB_Code (type = text, nullok=False, comment="...4 digit code.")
  • add PDB_Extended_Code (type = text, nullok=False, comment="Extended the accession code issued by PDB, if available"). This will get filled in later by an ingest code.
  • add Notes (type = text, nullok=True, comment="Additional details about the accession codes"). This is for curator notes.
  • add unique constraints: key(Accession_Code), Key(PDBDEV_Accession_Code), key(PDB_Code), key(PDB_Extended_Code)
  • Note: Accession_Code records whatever final Accession_Code will be used in the pipeline. This allows us to easily adapts to PDBDEV and PDB policies.

• Ingest script

  • Update dev and staging with the ermrest database snapshot from production
  • ingest the newly assigned accession code (1000) into this table.
  • For simplicity, @brindakv please create a csv that has the mapping between PDB_Accession_Code and PDBDEV_Accession_Code. The mapping file can be found in the shared google drive.
    • Need 2 versions: one for dev and one for production? Or we should just do a dump from production to all systems for consistency and testing.
  • For existing PDBDEV_Accession_Code, assign the PDB_Accession_Code according to the mapping (.csv file)
  • Create new rows for non-existent PDBDEV_Accession_Code
  • Note: We decided NOT to create a fake mapping to be tested on dev. (decided not to)

• Triggers:

  • create a trigger to prevent reassignment
  • That old.Entry has to be NULL. If not, throw an error (RAISE stmt).. Something is wrong since the same number is reassigned.
  • Write a trigger on dev to prepend 'TEST-' to the PDB_Accession_Code column.

Annotations

• PDB:Entry: Add annotation to show PDB_Accession_Code
• PDB:Accession_Code: Make entry.Workflow_Status visible on Accession_Code table.

Backend Processing

• When issue an accession code, check whether the Entry already has one assigned or look for row with Entry = Null and smallest RID e.g. select * from PDB.Accession_Code where Entry is null order by RID

ACL

• Only PDB curator can see the Accession_Code content
• PDB:database_2 should be visible to depositor but not editable (at all stages of the workflow). It should be editable by curator.
• Vocab:database_2_database_id should have the same ACL as all other vocab tables

Support of PDB and PDB-Dev accession codes

Pipeline changes

Update mmCIF generation process after accession codes are issued (Submission_Complete --> HOLD and Release_Ready --> REL)

• Refer to #85 for background information
• When pdbx_database_status, pdbx_audit_revision_history, and pdbx_audit_revision_details are added to the system generated mmCIF file, add a fourth table, database_2.
  • _database_2.database_id = PDB-Dev
  • _database_2.database_code and _database_2.pdbx_database_accession = entry.Accession_Code
  • _database_2.pdbx_DOI = ?
  • add database_2 table during Submission_Complete --> HOLD
  • Retain database_2 table as is during Release_Ready --> REL (it does not require to be updated in this step; it needs to be populated with the same information as the previous step)
  • Example:

    loop_
    _database_2.database_id 
    _database_2.database_code 
    _database_2.pdbx_database_accession 
    _database_2.pdbx_DOI 
    PDB-Dev PDBDEV_00000001 PDBDEV_00000001 ?
    PDB   8zz1         pdb_00008zz1 10.2210/pdb8zz1/pdb

    Note: when we have access to the changes made to the Accession_Code table, apply these rules when generating the _database_2 rows above:

• Generate a function (e.g. get_primary_accession_code(mode=PDB|PDBDEV).
  • If PDB, return PDB_Accession_Code, else PDBDEV_Accession_Code
• Generate a function (e.g. get_database_2_string(mode)) that generate a database_2 string based on two modes (PDB and PDBDEV)
  • If the mode is PDB: "PDB <PDB_Accession_Code> <PDB_Extended_Code> 10.2210/pdb<PDB_Code>/pdb" <- lower in doi
  • If the mode is PDBDEV: "PDB-Dev <PDBDEV_Accession_Code> <PDBDEV_Accession_Code> ?"
  • If the mode is not recognized, throw an exception
  • If the corresponding or is NULL, throw a missing_accession_code exception
• Create 2 config parameters: primary_accession_code_mode (default PDBDEV) and alternative_accession_code_mode (default PDB):
• In the backend processing code:
  • call get_primary_accession_code(primary_accession_code_mode)
  • throws an error if get_primary_accession_code returns a value that is not the same as what's in "Accession_Code" column.
• In the database_2 loop, call the get_database_2_string with the primary_accession_code_mode, then alternative_accession_code_mode.

# here is the _database_2 strings
get_database_2_string(primary_accession_code_mode) -- if missing_accession_code exception is caught, throw an error
If alternate_accession-code_model: 
  get_database_2_string(alternate_accession_code_mode) -- if missing_accession_code exception is caught, don't write to mmCIF.

# deprecate the following method
if PDBDEV_Accession_Code is not NULL, then generate
   "PDB-Dev <PDBDEV_Accession_Cde> <PDBDEV_Accession_Code> ?"
If PDB_Accession_Code is NOT NULL, then generate
  "PDB <PDB_Code> <PDB_Extended_Code> 10.2210/pdb<PDB_Code>/pdb"

[brindakv]

Make entry.Workflow_Status visible on Accession_Code table.

[brindakv]

Based on the discussion on April-19-2024:

Add database_2 table to PDB with the following attributes (data will be populated manually during curation by looking up the information in Accession_Code)

• database_id
• database_code
• pdbx_database_accession
• pdbx_DOI
• Structure_Id
• Update annotations to show the table
• Update mmCIF generation code to include database_2 (config file?)

[brindakv]

In the database_2 table in the system generated mmCIF file, lowercase of the PDB_Code should be used in the DOI string.

[aozalevsky]

@brindakv System' object has no attribute 'databases'\n is an indication of a python-ihm version without database2 support. i was able to reproduce it locally. @svoinea can you please check that pythom-ihm on the host is 1.1 and it's mounted as we discussed here: https://github.com/informatics-isi-edu/protein-database/issues/163#issuecomment-2008132719

[svoinea]

@aozalevsky @brindakv Installed version 1.1:

root@docker-pdbdev-validation-o3558551:/etc/cron.daily# pip3 show ihm
Name: ihm
Version: 1.1
Summary: Package for handling IHM mmCIF and BinaryCIF files
Home-page: https://github.com/ihmwg/python-ihm
Author: Ben Webb
Author-email: ben@salilab.org
License: UNKNOWN
Location: /usr/local/lib/python3.8/dist-packages
Requires: msgpack
Required-by: 

Rerun for Accession_Code = TEST-9A8L. Got the error:

ERROR IN REPORT VALIDATION.
stdoutdata: b''
stderrdata: b'INFO:root:Current operational mode is: PRODUCTION\nINFO:root:Clean up and create output 
directories\nINFO:root:Directory /ihmv/output/TEST-9A8L created \nINFO:root:Directory /ihmv/output/TEST-9A8L/
TEST-9A8L created \nINFO:root:Directory /ihmv/output/TEST-9A8L/TEST-9A8L/htmls created \nINFO:root:Directory /ihmv/
output/TEST-9A8L/TEST-9A8L/images created \nINFO:root:Directory /ihmv/output/TEST-9A8L/TEST-9A8L/csv created 
\nINFO:root:Directory /ihmv/output/TEST-9A8L/TEST-9A8L/pdf created 
\nWARNING:selenium.webdriver.common.selenium_manager:The geckodriver version (0.33.0) detected in PATH at /opt/
conda/bin/geckodriver might not be compatible with the detected firefox version (121.0); currently, geckodriver 0.34.0 is 
recommended for firefox 121.*, so it is advised to delete the driver in PATH and retry\nINFO:root:Entry 
composition\nTraceback (most recent call last):\n  File "/opt/IHMValidation/ihm_validation/ihm_validator.py", line 282, in 
<module>\n    template_dict = report.run_entry_composition(Template_Dict)\n  File "/opt/IHMValidation/ihm_validation/
report.py", line 85, in run_entry_composition\n    Template_Dict[\'ranked_id_list\'] = self.input.get_ranked_id_list()\n  
File "/opt/IHMValidation/ihm_validation/mmcif_io.py", line 153, in get_ranked_id_list\n    if pdbdev_id is not None:
\nNameError: name \'pdbdev_id\' is not defined. Did you mean: \'pdb_dev_id\'?\n'

[aozalevsky]

@svoinea thanks. that's better. Does this instance (or rather test mode) use the same IHMValidation (/mnt/vdb1/dev_pdbihm/IHMValidation) version? If yes, can you try running it once more?

I pushed an update (https://github.com/salilab/IHMValidation/releases/tag/20240528) yesterday morning. The server picked it up overnight, so it should work now.

[svoinea]

@aozalevsky The dev instance is using origin/dev_2.0. I am not sure if that is the same with 20240528:

$ git log -1
commit 07d9af171014a86f982090a30597ecdc30f99315 (HEAD, tag: 20240528, origin/dev_2.0)
Author: Arthur Zalevsky <aozalevsky@gmail.com>
Date:   Tue May 28 10:02:52 2024 -0700

    eliminate redundant duplicating IDs

The result is the same.

[aozalevsky]

@svoinea got it.

It was a typo in the var name in a specific scenario (database_2 is present, but none of the IDs match the entry_id; I didn't have a test case for this if). Anyway, I've pushed the fix https://github.com/salilab/IHMValidation/commit/7d4c10a5fa1fb9dd76aed97f53352445283388e0. I've pulled it on dev instance and was able to generate the report manually. Can you please test it again?

[svoinea]

@aozalevsky I have tested and it works now. Thanks.

[aozalevsky]

@svoinea @brindakv Brinda asked me to put a note, that entry_id should match at least one of the ids in the database_2.

Here is the content of the test file:

data_TEST-9A8L

#
_entry.id  TEST-9A8L
<...>
#
_pdbx_database_status.status_code                     HOLD
_pdbx_database_status.entry_id                        TEST-9A8L
_pdbx_database_status.deposit_site                    ?
_pdbx_database_status.process_site                    RCSB
_pdbx_database_status.recvd_initial_deposition_date   2024-05-25 
#
loop_
_database_2.database_id 
_database_2.database_code 
_database_2.pdbx_database_accession 
_database_2.pdbx_DOI 
PDB 9A8L pdb_00009a8l 10.2210/pdb9a8l/pdb
PDB-Dev PDBDEV_00000385 PDBDEV_00000385 ?

[brindakv]

@svoinea In pdb mode, the value of _database_2.database_code when _database_2.database_id = PDB-Dev is incorrect in the system generated mmCIF file.

This is what it is currently:

loop_
_database_2.database_id
_database_2.database_code
_database_2.pdbx_database_accession
_database_2.pdbx_DOI
PDB TEST-XXXX test-pdb_0000xxxx 10.2210/pdbtest-xxxx/pdb
PDB-Dev TEST-XXXX TEST-PDBDEV_00000NNN ?

This is incorrect. It should be:

loop_
_database_2.database_id
_database_2.database_code
_database_2.pdbx_database_accession
_database_2.pdbx_DOI
PDB TEST-XXXX test-pdb_0000xxxx 10.2210/pdbtest-xxxx/pdb
PDB-Dev TEST-PDBDEV_00000NNN TEST-PDBDEV_00000NNN ?

[brindakv]

@svoinea can _pdbx_database_status.deposit_site also be set to RCSB instead of ?

[svoinea]

I have done the required updates and redeployed the backend.

[brindakv]

@svoinea The generated mmCIF file is still incorrect.

_pdbx_database_status.status_code                     HOLD
_pdbx_database_status.entry_id                        TEST-9A8N
_pdbx_database_status.deposit_site                    ?
_pdbx_database_status.process_site                    RCSB
_pdbx_database_status.recvd_initial_deposition_date   2024-06-05
#
loop_
_database_2.database_id
_database_2.database_code
_database_2.pdbx_database_accession
_database_2.pdbx_DOI
PDB TEST-9A8N test-pdb_00009a8n 10.2210/pdbtest-9a8n/pdb
PDB-Dev TEST-PDBDEV_00000387 TEST-PDBDEV_00000387 RCSB

_pdbx_database_status.deposit_site should be set to RCSB and _database_2.pdbx_DOI should be ? when _database_2.database_id == PDB-Dev.

The correct data should be:

_pdbx_database_status.status_code                     HOLD
_pdbx_database_status.entry_id                        TEST-9A8N
_pdbx_database_status.deposit_site                    RCSB
_pdbx_database_status.process_site                    RCSB
_pdbx_database_status.recvd_initial_deposition_date   2024-06-05
#
loop_
_database_2.database_id
_database_2.database_code
_database_2.pdbx_database_accession
_database_2.pdbx_DOI
PDB TEST-9A8N test-pdb_00009a8n 10.2210/pdbtest-9a8n/pdb
PDB-Dev TEST-PDBDEV_00000387 TEST-PDBDEV_00000387 ?

[svoinea]

I did the updates and redeployed.

[brindakv]

• Support alternative mode = None
• Handle cases where accession codes don't exist in ermrest

[hongsudt]

@svoinea I updated the description in the main issue to include the following:

• In the get_database_2_string(mode) function, check the corresponding or . If NULL, throw an missing_accession_code exception. Please note that I changed the logic for the PDB mode, to use <pdb accession code> instead of <pdb code>. This way, it will respond to whatever PDB policy for accession_code..
• In the main code that calls the get_database_2_string, handle the missing_accession_code exception. For the primary mode, throw an error. For the alternate mode, ignore.
• Check that alternate_accession_code_mode is not Null before calling the get_database_2_string function.

[svoinea]

The model for the Accession_Code table was defined as NOT NULL for the PDBDEV_Accession_Code, PDB_Extended_Code, PDB_Code and PDB_Accession_Code columns.

[brindakv]

pdbdev mode works as expected on dev.