Migration of existing PhysioBank projects

[tompollard]

Existing PhysioBank projects will need to be migrated to the new system. We will need to find a way of populating all of the mandatory metadata fields (or alternatively create a project model for historical content and accept that some information is missing).

List of PhysioBank projects at: https://physionet.org/physiobank/database/DBS

List of PhysioNetWorks projects at: https://physionet.org/works/

[tompollard]

I've drafted a form for collecting metadata for projects: https://goo.gl/forms/g2WHwAK03bbZEP2P2.

[tompollard]

The form now has the latest metadata fields for data and software (the fields merged in #185): https://goo.gl/forms/dgliUZbZuVh6UTh43

[tompollard]

To decide how to handle each of the PhysioNetWorks projects, it would be useful to collate the following details:

• Project name
• Date created
• Number of approved collaborators
• Date last accessed by a collaborator
• Date when a collaborator was last added.
• Email address of the person/people responsible for adding collaborators
• Number of users who have requested access and who have not received a response.

Is there a quick way of gathering these details?

[elfeto]

So, for the things mentioned above:

• The Project name is easy to get.
• If the Works project was created after January 15, 2017, then we have the creation and activation date.
• If the project has the waiting for access, then is easy to get the number of people waiting for access.
• Number of approved collaborators ->(only collaborators? or also reviewers ?)
• Not possible ... -> Date last accessed by a collaborator
• Not possible ... -> Date when a collaborator was last added.
• Not possible ... -> Email address of the person/people responsible for adding collaborators

We should ignore the Works for now, and ask the people that have the works projects, to create it as a new project in the new version of physionet. This is because we wont hold closed access to X projects, and everything WILL be published under a DUA or no DUA.

We should move the PhysioBank and PhysioTools just AS IS, and add a "Legacy" tag.

[tompollard]

Thanks, for "Number of approved collaborators", you are right, I mean collaborators and reviewers.

We should move the PhysioBank and PhysioTools just AS IS, and add a "Legacy" tag.

Think we have a plan for this now :) I'll add a few issues for points that came up in the conversation.

[cx1111]

Going to make a list of dbs to migrate. I'll comment the differences from the DBS file here.

• gait-maturation-db has 2 pages to be combined manually. Original data is in data subdir.
• meditation has 2 pages to be combined manually. Original data is in data subdir.
• synthetic/tns/ should just be tns (and added to DBS)
• merge mimicdb and mimicdb/numerics
• skip mimic2cdb for now
• skip mimicwdb for now
• skip mimic3wdb/matched for now
• mvtdb page has 2 pages to be combined manually
• exclude mimic2db and mimic2db/numerics
• exclude sleep-edf

For now I am skipping the challenges too. When we do move them, we will:

• merge challenge/2009 lines
• merge challenge/2010 lines
• merge challenge/2011 lines
• merge challenge/2013 lines
• merge challenge/2014 lines
• merge challenge/2015 lines
• merge challenge/2016 lines
• merge challenge/2017 lines
• merge challenge/2018 lines

[cx1111]

Outstanding issues of the initial metadata migration:

• Two DOIs missing. Being assigned.
• Titles in DBS file not always matching titles of html titles.
• Some databases in the pbank index not present in the DBS file.

[cx1111]

There were 114 pbank databases transferred over when I had 108

• testdb is not an actual database. Removed on new server.
• synthetic has been renamed to tns
• pbs is not a database. It's the physiobank search. Removed.
• udb is not a database. Removed.
• odb is not a database. Removed.
• Removing the eicu pages until we figure out how to present those.

Also there is this directory when html documentation is written: https://physionet.org/physiobank/database/html/

We should port this over in another way later.

This gives us a count of 108 databases excluding challenges, mimic2+, and eicu.

[cx1111]

Another issue we have to work out: what to do with additional html documentation for certain databases, such as MITDB, and WFDB.

[tompollard]

The current site has no explicit support for HTML documentation, as far as I'm aware, so approaches have been ad-hoc and inconsistent. In a small number of cases, HTML documentation has been manually created (e.g. the waveform database), but in general we have created independent sites for HTML documentation (e.g. https://mimic.physionet.org/ and https://eicu-crd.mit.edu/).

My preferred approach is to:

• Accept that there is currently no functionality for supporting HTML documentation on PhysioNet beyond the project description page.
• Add the PDF version of documentation to the relevant project.
• Where necessary, migrate existing HTML documentation to an independent website (in same way as the MIMIC website), perhaps at a PhysioNet subdomain like http://wfdb.physionet.org.
• Create a feature request for formal support of HTML documentation on PhysioNet.

[elfeto]

So, update ? this fixed? or what to do here ?