Update production - Githubissues

anngvu commented 1 year ago

Mainly:

Workflow with new deploy strategy that maps branches to app instances somewhat differently; also schematic has been pinned to latest real pypi release, which seems stable enough (though I've only tested generation and submission with one template on staging). We disable the default shiny-deploy.yml as seen in https://github.com/nf-osi/NF_data_curator/actions/workflows/shinyapps_deploy.yml, and custom-deploy-shiny.yml has been running fine as seen with develop and edge in latest Action runs. Note that tagged releases are no longer required (ignored) for deployment to production -- if this is desired, can update wf again.
Workflow handling releases by data model repo explicitly pinned to 20.04 (because GitHub updated ubuntu-latest runner to mean 22.04, which broke stuff as seen here).
Surface new templates and terms in production.

allaway commented 1 year ago

@anngvu, thanks for doing this!

I am noticing one bug; the annotations are submitted with capitalized keys (see, for example, CellType vs cellType, which I just added to this file using the staging app: https://www.synapse.org/#!Synapse:syn30595914).

I think this is a schematic config. It's quite possible I messed this up when I last updated the app...

allaway commented 1 year ago

The issues I mentioned earlier seem to have been fixed with those parameters!

allaway commented 1 year ago

However, I am loath to merge stuff that will cause a Friday afternoon deployment, so let's wait till monday. :)

allaway commented 1 year ago

With this latest update to address https://github.com/nf-osi/nf-metadata-dictionary/issues/230, should we deprecate PdxGenomicsAssay, since the xenograft info is now available in the metadata manifest for the other genomics assay types? Should we also pull in the other xenograft-relevant terms transplantationType transplantationRecipientSpecies 1transplantationRecipientTissue`? https://github.com/nf-osi/nf-metadata-dictionary/blob/main/modules/Experiment/annotationProperty.csv

allaway commented 1 year ago

Also, selecting pdxGenomicsAssay breaks the app - here's the error message:

2023-01-10T22:58:13.555917+00:00 shinyapps[5711947]: [WARNING] /srv/connect/apps/NF_data_curator-staging/.venv/lib/python3.8/site-packages/synapseclient/table.py:596: DtypeWarning: Columns (10,11,22,23,53,56) have mixed types. Specify dtype option on import or set low_memory=False.
2023-01-10T22:58:13.555971+00:00 shinyapps[5711947]:   df = pd.read_csv(filepath,
2023-01-10T22:58:13.555983+00:00 shinyapps[5711947]: WARNING: [2023-01-10 22:58:13] py.warnings - /srv/connect/apps/NF_data_curator-staging/.venv/lib/python3.8/site-packages/synapseclient/table.py:596: DtypeWarning: Columns (10,11,22,23,53,56) have mixed types. Specify dtype option on import or set low_memory=False.
2023-01-10T22:58:13.555987+00:00 shinyapps[5711947]:   df = pd.read_csv(filepath,
2023-01-10T23:03:28.991359+00:00 shinyapps[5711947]: warning: The `use_annotations` option is currently only supported when there is no manifest file for the dataset in question.
2023-01-10T23:03:29.018219+00:00 shinyapps[5711947]: Warning: Error in py_call_impl: KeyError: 'PdxGenomicsAssayTemplate'
2023-01-10T23:03:29.018265+00:00 shinyapps[5711947]: Detailed traceback:
2023-01-10T23:03:29.018273+00:00 shinyapps[5711947]:   File "/srv/connect/apps/NF_data_curator-staging/.venv/lib/python3.8/site-packages/schematic/models/metadata.py", line 141, in getModelManifest
2023-01-10T23:03:29.018278+00:00 shinyapps[5711947]:     mg = ManifestGenerator(
2023-01-10T23:03:29.018283+00:00 shinyapps[5711947]:   File "/srv/connect/apps/NF_data_curator-staging/.venv/lib/python3.8/site-packages/schematic/manifest/generator.py", line 92, in __init__
2023-01-10T23:03:29.018288+00:00 shinyapps[5711947]:     is_file_based = "Filename" in self.sg.get_node_dependencies(self.root)
2023-01-10T23:03:29.018299+00:00 shinyapps[5711947]:   File "/srv/connect/apps/NF_data_curator-staging/.venv/lib/python3.8/site-packages/schematic/schemas/generator.py", line 211, in get_node_dependencies
2023-01-10T23:03:29.018304+00:00 shinyapps[5711947]:     required_dependencies = self.se.explore_class(source_node)["dependencies"]
2023-01-10T23:03:29.018311+00:00 shinyapps[5711947]:   File "/srv/connect/apps/NF_data_curator-staging/.venv/lib/python3.8/site-packages/schematic/schemas/explorer.py", line 332, in explore_class
2023-01-10T23:03:29.018316+00:00 shinyapps[5711947]:     if "subClassOf" in self.schema_nx.nodes[schema_class]:
2023-01-10T23:03:29.018321+00:00 shinyapps[5711947]:   File "/srv/connect/apps/NF_data_curator-staging/.venv/lib/python3.8/site-packages/networkx/classe [... truncated]
2023-01-10T23:03:29.029619+00:00 shinyapps[5711947]:   15: <Anonymous>
2023-01-10T23:03:29.029655+00:00 shinyapps[5711947]:   13: fn
2023-01-10T23:03:29.029676+00:00 shinyapps[5711947]:    8: retry
2023-01-10T23:03:29.029682+00:00 shinyapps[5711947]:    7: connect$retryingStartServer
2023-01-10T23:03:29.029687+00:00 shinyapps[5711947]:    6: eval
2023-01-10T23:03:29.029692+00:00 shinyapps[5711947]:    5: eval
2023-01-10T23:03:29.029696+00:00 shinyapps[5711947]:    4: eval
2023-01-10T23:03:29.029705+00:00 shinyapps[5711947]:    3: eval
2023-01-10T23:03:29.029710+00:00 shinyapps[5711947]:    2: eval.parent
2023-01-10T23:03:29.029720+00:00 shinyapps[5711947]:    1: local

so perhaps another reason to deprecate it.

anngvu commented 1 year ago

Yeah, I think we can remove it from the selection currently -- will make proposed change.

nf-osi / NF_data_curator

Update production #152