Reinstate, update + set up automation for schema visualization on the portal

[aclayton555]

Work request filed with FAIR on 2023.07.11: https://sagebionetworks.jira.com/browse/IBCSPRT-185

Following the 2023.08.18 FAIR sprint review, suggested approach should be to fork repo (https://github.com/Sage-Bionetworks/schema_visualization) and update config to point to our data model.

Adam notes We could have a viz that points to both the main and latest-release data modelsCould be part of the same action where we update mtDCA config to update the viz repo

[adamjtaylor]

Have forked in the nchitan repo (https://github.com/ncihtan/schema_visualization) and updated the config to point to the v23.8.2 HTAN data model.

Cloning locally and running python3 -m http.server, I can render the visualisation locally.

[adamjtaylor]

No progress made this sprint. Visualisation fixes per #288 were made. Suggest passing back to FAIR?

[aclayton555]

Adam will contact Ino and Milen about plan for this. This is more work than needed to update. Decision to either leave as is (not ideal) or take down entirely.

[aclayton555]

Ask FAIR to have visualization point to v23.9.1

[aclayton555]

Requested support from FAIR in December in https://sagebionetworks.jira.com/browse/IBCSPRT-185

[adamjtaylor]

I don't have access to the issue above :(

[aclayton555]

how about now: https://sagebionetworks.jira.com/browse/IBCSPRT-185

Unfortunately, most of the comments are in the 'internal comments' fields, so I doubt the portal view is very helpful: https://sagebionetworks.jira.com/servicedesk/customer/portal/5/IBCSPRT-185

[adamjtaylor]

Still no access to the issue directly :(

[adamjtaylor]

Maybe have for a HTAN coordination call agenda item to make sure Ino can drive this forward with Milen

[aclayton555]

Copying internal comments from the JIra ticket for visibility (sounds like D&T will need to consistently use the 'reply to customer' feature to ensure visibility to Sage staff:

2023.10.20 Hi @ Amy Heiser , I just chatted with Adam and Elvira about this and we were wondering if we can request support from FAIR on this in early December, ideally to allow us to update the schema viz to the data model release we are planning for end of November.

Longer term, we would like to understand if:

Updates to the schema viz are made, as needed, to align with a specified release and are therefore static until the next update is specified/made (manually)

Or, if this can somehow be automated as an action to update the schema viz with each versioned release of the data model (for example, we have already had 2 releases in October)

2023.11.1 @ Ashley Clayton Just wanted to provide a quick update - with a delay on the Schematic v23.10.1 release (now v23.11.1) this will be done in our next sprint (sprint 23) - mid/late Nov after Schematic is deployed to the REST API. It should still align with the ability to update your end of Nov data model release, but let me know if you have any questions.

[aclayton555]

I propose that we work with FAIR to at least have the viz updated with our end of Nov data model release. This can be followed with longer term coordination between Milen and Ino to understand how we can automate maintaining this product on the portal.

[aclayton555]

Some unexpected behavior being observed. Ticket to be filed on FAIR

[adamjtaylor]

[aclayton555]

To be discussed during mid-sprint 2024.01.04 - I propose that we take the schema vis down from the HTAN Data Portal due to limited utility of this tool, and unclear prioritization for support of this product.

As of today, the tool is linked from this page of the HTAN Data Portal: https://humantumoratlas.org/standards

This currently links out to this version of this tool, which is not helpful to users: https://sage-bionetworks.github.io/schema_visualization/

So it seems the portal needs to be linked out to https://ncihtan.github.io/schema_visualization/ . If so, are we still seeing the unexpected behaviour with this visualization (seen screencast from Adam in earler comment)? Is this tool helpful for users to navigate the HTAN Data Model? How can we automate updating this tool in the future?

[adamjtaylor]

I'm happy with the proposal to remove the data model visualization. Suggest we have this on Monday meeting agenda for approval by PIs + Ino.

[aclayton555]

Discussed on 23-12 sprint close out on 2024.01.30:

• update the portal to point to the https://ncihtan.github.io/schema_visualization/ - Adam can file a ticket for this.
• but do this after we fix the DAG-related issues following the schematic refactor (#332)

[aclayton555]

Raised on HTAN Ops call on 2024.01.30:

• Alex and Ino agreed that we should take the schema vis down until we complete fixes and make this more useful
• Ino removed (PR: https://github.com/ncihtan/htan-portal/pull/579), but this can be returned when ready
• Still planning to prioritize DAG-related issues in 24-2 sprint

[milen-sage]

@aclayton555 I'm assuming Amy is in the loop? Otherwise let me know and we can triage + prioritize.

[aclayton555]

@milen-sage Adam will be prioritizing efforts to address some of the issues discovered in the data model following the schematic refactor testing. Once this work has been completed, we think we will have a more accurate and helpful visualization. I don't think any additional effort from FAIR Data is needed at this time to re-instate the data visualization once we have completed these data model updates (but we will reach out if additional support is needed here).

Related, in the initial ticket linked above (https://sagebionetworks.jira.com/servicedesk/customer/portal/5/IBCSPRT-185) I indicated an opportunity to engage with Ino to support utility and automation of the visualization with the portal. I think Amy has had some engagement with Ino on this, but I am not sure if that is tracked separately.

Going forward, I think there is an opportunity to establish clarity around ownership of this visualization within the HTAN team, with understanding of the user stories driving this, and the key stakeholders to continue support of this product (this is not something Adam and I have had much involvement with to-date...my impression is that it was added to the portal a while ago and otherwise forgotten).

[milen-sage]

Got it - thanks @aclayton555

[AmyHeiser]

@aclayton555 I didn't receive a reply from Ino after my initial email in July 2022, I'll make another attempt today. If this is included on an HTAN agenda I would be happy to attend and discuss further.

[aclayton555]

@AmyHeiser Thanks again for following up with Ino. Through end of HTAN 1.0 (through August)I don't think the visualization itself is considered a high priority from our side. Adam is currently working through some of the data model fixes following the schematic refactor that will ultimately improve visualization once we re-instate it. Longer term, it would be helpful for us to understand how FAIR is planning to support this visualization tool going forward, and what expectations are for HTAN DCC members as well to support it (i.e. who owns this). I think not having a clear owner and understanding of the development roadmap led to this falling very out of sync over the last year. I hope a conversation with Ino will be helpful to understand how we can keep this updated on the portal with minimal effort, and perhaps inform how users are expected to engage with this tool.

Please feel free to bring this to an HTAN meeting once you have had a chance to discuss with Ino.