Certain project details fields are empty intermittently

[hannes-ucsc]

Actual: description and other fields are empty

Expected: fields should be populated. The service returns them, as the screenshot proves.

After switching to the details page for a different project, and then switching back, the description is actually displayed so I'm inclined to think that there is a race at play here:

[MillenniumFalconMechanic]

Hi @hannes-ucsc , it looks like the root cause of this defect is in the area where we format the project's organ value/s for display. We are currently assuming there is always a value for organ but this is not the case. We get a JS error when we attempt to manipulate an undefined organ value, which in turn prevents subsequent formatting of the project values to occur (for example, project description).

This defect will be fixed once I update this area of the code to (re)use the mappers I created for #660.

I am unsure however, about the intermittent behavior you mentioned.

Looking at your screenshot of the project detail page above, it has an organ value of decidua, blood, placenta.

Looking at the project in production right now, it appears to have no value for organ.

Here is a screenshot of the project response in the data table:

Here is a screenshot of the project response on the detail page:

The write to and read from our local store is atomic. That is, the selected project is updated completely or not at all: there is no partial update of the project. Is it possible that the server is intermittently returning a different value for organ?

I can also see how the data was "mixed" on our end, creating the "intermittent" effect. I will fix this in addition to the parse-for-display defect above.

[hannes-ucsc]

Not sure what you are asking, @MillenniumFalconMechanic. Looks like you initially thought Azul was intermittently returning different results for the same project detail URL but later retracted that observation? To reproduce the intermittent nature of this bug, follow these steps:

On the Projects tab, click on a project that has a value other than Unspecified in the Organ column. Hit Back button. Click on a project with Unspecified in the Organ column. The project description will be displayed but the organ will be displayed as the value from the previously displayed project. Hit Refresh. The project details page reverts to not displaying the project description.

[malloryfreeberg]

We are currently assuming there is always a value for organ but this is not the case.

I just want to clarify that according to the metadata standard, there is always an organ for all projects. The complexity comes from displaying organ for specimens and organ_model for cell lines and organoids. Cell lines and organoids do in fact have their own organ metadata. There is a ticket in Azul about this topic, maybe @zperova can find/post it here?

[zperova]

@MillenniumFalconMechanic I wonder whether this issue is related with the recent changes in Azul to capture both organ for specimens and organ_model for cell lines and organoids: https://github.com/DataBiosphere/azul/issues/1022 https://github.com/DataBiosphere/azul/issues/977

[MillenniumFalconMechanic]

Hi @hannes-ucsc, that is correct, I retracted my question regarding the Azul response after I was able to consistently reproduce the problem. The root cause remains unchanged: there is a JavaScript error on this page when we encounter a project with no key for organ.

The steps to reproduce are:

1. Navigate to projects tab
2. Select a project with a specified value for organ
3. Navigate back to the projects table
4. Select a project with a value of "Undefined" for organ

At this point, you can see the project page is displaying a mix of the two projects you have selected.

[hannes-ucsc]

@malloryfreeberg

Cell lines and organoids do in fact have their own organ metadata.

Not all projects in production, do. Pe'er doesn't have model_organ on cell_lines. But that's special because you're still planning to reingest it, correct?

There are other projects in production that also display Unspecified in the Organ column and that must have a different reason. Azul returns the model organ correctly under hits[*].samples.modelOrgan for those projects.

@danielsotirhos will be exposing the effective organ as part of hits[*].samples.effectiveOrgan in DataBiosphere/azul#1022 which @zperova linked to above. IF Data Browser only wants to display a single value in a single organ column (and I am not sure that's advisable and have argued against ad nauseam before), Data Browser should use hits[*].samples.effectiveOrgan.

[zperova]

@hannes-ucsc I can see 4 dataseta that have Unspecified in the Organ model in the Browser. I looked at the metadata and we have the corresponding fields filled in (including in the Pe'er dataset, which indeed needs to be re-ingested but that should not matter for displaying the Organ model).

The Organ column should be populated from cell_line.model_organ.text (or cell_line.model_organ.ontology_label, I forgot which one you use at the moment) for the following datasets:

• Single-cell RNA-seq analysis throughout a 125-day differentiation protocol that converted H1 human embryonic stem cells to a variety of ventrally-derived cell types.
• Comparison, calibration, and benchmarking of high-throughput single cell RNA-Seq techniques for unbiased cell-type classification
• Profiling of CD34+ cells from human bone marrow to understand hematopoiesis (Pe'er)

The Organ column should be populated from organoid.model_organ.text (or organoid.model_organ.ontology_label) for Assessing the relevance of organoids to model inter-individual variation.

Could you please confirm that this is indeed the case? Otherwise, I do not understand why it is displayed as Unspecified.

[hannes-ucsc]

@zperova: Looking at Pe'er. I took one example bundle (11be0dbd-ad77-49b9-8c46-ebb455e88eea.2018-12-05T172232.264458Z) and its cell_line_0.json does not have an model_organ:

{
        "describedBy": "https://schema.humancellatlas.org/type/biomaterial/9.0.0/cell_line",
        "schema_type": "biomaterial",
        "biomaterial_core": {
            "biomaterial_id": "HS_BM_1_cell_line",
            "biomaterial_name": "Bone Marrow CD34+ stem/progenitor cells",
            "ncbi_taxon_id": [
                9606
            ]
        },
        "supplier": "AllCells, LLC",
        "catalog_number": "ABM022F",
        "catalog_url": "https://www.allcells.com/products/bone-marrow-cd34-stem-progenitor-cells",
        "cell_line_type": "stem cell",
        "cell_type": {
            "text": "CD34-positive, CD38-negative hematopoietic stem cell",
            "ontology": "CL:0001024"
        },
        "date_established": "2016-10-12T00:00:00Z",
        "genus_species": [
            {
                "text": "Homo sapiens",
                "ontology": "NCBITaxon:9606"
            }
        ],
        "provenance": {
            "document_id": "b95bd00c-1c00-4f57-820f-f3f431e63377",
            "submission_date": "2018-12-04T19:02:13.661Z",
            "update_date": "2018-12-04T19:02:25.327Z"
        }
    }

So for Pe'er the ball is in Metadata/Ingest's court, as far as I can tell.

[zperova]

@hannes-ucsc you are right - I checked the spreadsheet that is updated for re-ingestion, not the one that corresponds to the dataset currently in the Data Browser. Once re-ingested this will be fixed.

[hannes-ucsc]

HPSI human cerebral organoids: Azul returns hits[*].samples[0].modelOrgan as brain but Data Browser does not display it.

Drop-seq, DroNc-seq, Fluidigm C1 Comparison: Azul returns hits[*].samples[0].modelOrgan as stem cell but Data Browser does not display it.

Single cell RNAseq characterization of cell types produced over time in an in vitro model of human inhibitory interneuron differentiation: Azul returns hits[*].samples[0].modelOrgan as brain but Data Browser does not display it.

[zperova]

Hmm, it is interesting that the Organ is not displayed at all in the Project view, however in the Project Browser it is displayed as Unspecified - should not these be pulled from the same place?

[MillenniumFalconMechanic]

Hi @zperova, I believe this is due to the defect @hannes-ucsc mentioned originally. I should have the fix, and corresponding tests, ready to push today. Going forward, project data displayed in the data table should always match the project data listed on the corresponding project detail pages.

[NoopDog]

This update for this is deployed to dev and integration. Will move this to staging on the 6/12 staging deploy.

[NoopDog]

The update for this is deployed to staging.

[NoopDog]

@zperova @malloryfreeberg, @hannes-ucsc an update for this ticket has been deployed to production.

1. The project detail page no longer errors if the organ is unspecified. We are now more resilient to variations on missing data "null", [], [null] etc. or just no key for the property in the json.

2. In all cases the values displayed on the project detail page should match the values displayed on the projects' row in the projects tab. There was some inconsistency in displaying blank or Unspecified or "--". Now the only indicator used is "Unspecified".

3. We are still not displaying model organ on the project tab or the project detail page. We will add this first to the detail page and then to the project table once we are able to horizontal scroll the table as the table is quite full and we don't have room for new columns.

[zperova]

@NoopDog thank you for the update. I do not think Model organ should be a separate column - instead renaming the Organ column to Organ/Model organ should be sufficient as it will be either an organ in case of primary tissue or a model organ in case of organoids and cell lines. Since there is a Sample type column right next to Organ it will be obvious to the consumer whether the value refers to an organ or a model organ. Should I make a separate ticket for that @NoopDog ?

[hannes-ucsc]

@NoopDog Consider displaying both fields in a single column and annotating model organ values:

Note how the two values are wrapped allowing the column to remain narrow. Horizontal scrolling is a usability nightmare IMO.

[zperova]

I agree with @hannes-ucsc about the horizontal scrolling. More against having a Model organ as a separate column is that in most of the cases (primary tissue samples) it will be empty. We have to keep in mind that once we get imaging datasets in we will have to add imaging related columns or repurpose existing ones to serve both sequencing and imaging metadata. I am very happy to discuss this in more details.

[NoopDog]

Ok sure, I agree that horizontal scroll can be a pain and we have avoided it so far. We do have many more columns to add so we need some approach to handle. We can investigate alternatives to the horizontal scrolling. Keep in mind that we need to fit on a 13inch mac screen which i believe is only 1280 pixels or so. In any case, we can explore a more card like "row" that can get taller.

We can try the combined column for the Organ/Model organ in this case. We have already a ticket to add model organ. I can just add to that to combine it with the organ.

[hannes-ucsc]

Keep in mind that we need to fit on a 13inch mac screen which i believe is only 1280 pixels or so.

Where does that requirement originate from?