search

ncihtan / HTAN-data-curator

HTAN Data Ingest Shiny App
https://sagebio.shinyapps.io/HTAN-data-curator/
Apache License 2.0
1 stars 0 forks source link

Turn off Data Dashboard in production #47

Closed adamjtaylor closed 1 year ago

adamjtaylor commented 2 years ago

our update to https://github.com/Sage-Bionetworks/data_curator/releases/tag/v22.11.3 turns on the data dashboard.

I imagine this will result in quite a few questions from centers about data completeness. I have some already.

I would like to explore the possibility of hiding the data daashboard in production until there is appropriate documentation for centers and we can properly launch the feature at a steering committee.

I would like to remove ASAP and then target adding back in during the 23/01 sprint.

adamjtaylor commented 2 years ago

Related issues:

adamjtaylor commented 2 years ago

FYI @milen-sage - we hare having some issues with the Data Dashboard in production. Is it possible to selectively remove the Dashboard module to make sure we have the other fixes in data_curator v22.11.3 deployed, or are we best to revert to be level with data_curator v22.11.2 as per the draft PR linked.

When working the dashboard looks great but I think we should hold it to launch at a steering committee or similar once fixed, and update our FAQ docs on how to use/interpret at the same time

adamjtaylor commented 2 years ago

Confirming that we are going to revert per the PR above

adamjtaylor commented 2 years ago
milen-sage commented 2 years ago

@adamjtaylor I can confirm that reverting to codebase in the main branch of the DCA upstream (i.e. v22.11.2) would get the bug fixes noted in v22.11.3; you'd only be missing the dashboard features and suppression of warning messages in the log. The steps above are good to go!

elv-sb commented 2 years ago

@milen-sage we just completed the actions above. I can confirm that production looks ok again.

milen-sage commented 2 years ago

@adamjtaylor and @elv-sb could you start gh issues for the bugs/features you'd like to see with the dashboard (aside from docs - we are planning on that work)? Thanks!

elv-sb commented 2 years ago

@milen-sage we can add those during our next sprint. Thank you!

milen-sage commented 2 years ago

Thanks @elv-sb! Meanwhile feel free to explore more of the dashboard in the testing HTAN instance. Once we finish the docs on how schema validation rules are reflected in the dashboard and how the dashboard results can be interpreted we can have a dashboard exploration/play meeting. But certainly agreed with you and @adamjtaylor that we don't want to scare centers, if they see a bunch of their metadata, clinical and biospecimen data are marked as invalid out of the blue.

adamjtaylor commented 1 year ago

Bring to next data with FAIR on 12/12

What is the plan at the end of this sprint?

aclayton555 commented 1 year ago

Per discussion with Milen on 1/9 HTAN/FAIR stakeholder call:

Adam: Ability to toggle dashboard - is this being considered? Milen: Approach will be to remove by default and make optional; only enable for projects that have gone through process of ramping up users/rolling this out - this will then be turned on. Next DCA release is expected to have this removed (targeted for 1/11; v23.1.1), so we can pull in latest release with other fixes

afwillia commented 1 year ago

@adamjtaylor the dashboard has been disabled by default in the main branch of DCA and release v23.1.1.

One note: you'll need to incorporate the changes in shinyapps_deploy.yml . The step Save service account credentials for Schematic will properly convert the schematic service account credentials stored as a github secret to a json file. Moving forward, schematic will authenticate the google API using the service account credentials instead of the token.pickle file.

adamjtaylor commented 1 year ago

I'd like to be able to update our DCA to v23.1.1, but when I try and open a PR for this it won't allow me https://github.com/ncihtan/HTAN-data-curator/compare/update-dca...Sage-Bionetworks:data_curator:v23.1.1

adamjtaylor commented 1 year ago

Following @milen-sage's comment in FAIR Data sprint review we should actually update to v23.1.2 (ie level with the head repo main branch) as the Google auth changes are included in the newest version.

adamjtaylor commented 1 year ago

Moving to ready for review as the Data Dashboard (Not DCA!) is off. Suggest we have a new issue for the cleanup and actually getting to the latest DCA version.

milen-sage commented 1 year ago

@adamjtaylor why is the DCA off? Based on discussion with @afwillia and the jira issue, I had assumed the DCA was running w/ updated schematic code (v23.1.1) and DCA v22.11.x, which would have switched to updated Google auth mechanism, even if not using the latest DCA version. I was assuming @afwillia was syncing up with @adamjtaylor to update the DCA to v23.1.3 to move on to the latest version but that was not a blocker for DCA running, based on conversation with @afwillia.

Was my understanding incorrect? Or is there a different issue?

adamjtaylor commented 1 year ago

@milen-sage Typo on my behalf. Data dashboard is currently off - not the DCA. Have updated comment. Otherwise your understanding correct and we will need to sync up with @afwillia to get us fully up to v23.1.x Sorry for the confusion!