General: Allow different mechanisms of hiding a variable (downloads vs browser)

[danicahelb]

Some variables are not useful for exploring on the site but are useful for analyses offline. For example, in WASH-b Kenya "Cluster ID" is not useful to display on the website.

on the WDK site, the presenter allows us to hide these variables from the site but keep them in the downloads

on the EDA site, we can annotate a variable as "hidden" in the .owl but this will hide it from both the site and the downloads

[SheenaTomko]

Adding Danica's original post, in phase 1, we have an annotation property called Hidden in the owl file that can be left blank or set to yes. During preprocessing and loading, that annotation property is used to set DisplayType to Hidden. If a variable is Hidden, the variable is excluded from the download files and not displayed on the website.

For phase 2, we want to introduce different types of hidden variables. Below are some potential options. These could be the different values of the owl file property Hidden that are then propagated to DisplayType during loading. The only potential roadblock is that DisplayType can also have the value multifilter, so we'd want to make sure first that we'd never want something to both be part of a multifilter and have one of the options below downloadOnly - variable is not available in the browse and subset tab or the visualization tab, but is present in the download files noDownload - variable is available in the browse and subset tab and visualization tab, but is not present in the download files (either bulk or from the subset table) hidden - variable is not available anywhere (browse and subset, visualization, downloads)

[steve-fischer-200]

we can go "cheap" or we can go "correct"

cheap:

• use the existing displayType property, and hope that noDownload never collides with Hidden
• patch the relevant variable's metadata in the db (and update the owl file)

correct:

• introduce a new scope property (similar to genomics sites), with values like downloadOnly, noDownload, hidden
• retain the DisplayType property with, so far, only *hidden
• this, i believe, will require a reload of all studies, or, anyway, an update to the variables metadata for each study

maybe for build 55 we should do cheap,

we could also file a ticket for the correct solution, to be done as part of one big phase 2 reload, based on issues we discover over some months

[SheenaTomko]

Discussed on 10/20/2021 with Steve and Danica. Decided that

• For b55 we will continue to have the UI use the displayType property
• Outreach team will use the "hidden" annotation property and mark "yes" for all variables that should be hidden in any way in new studies being loaded in b55
• New "scope" annotation property needs to be created in the owl files and in the UI. This will be a positive term, indicating where the variable should appear. Studies loaded in b55 will have this annotation
  • variableTree - should appear in the variable tree in the browse and subset tab and visualization tab
  • download - should appear in the download files
  • everywhere - should appear in the variable tree and download files (may want to have Jay load it as everywhere if no value is entered in the owl)
  • hidden - should not appear anywhere
• In b56, we will switch from using displayType to using scope. All phase 1 studies with hidden terms will need to be reloaded to use scope

[danicahelb]

This is how we've annotated "hidden" in the .owl files:

Annotations will indicate where the variable should NOT appear.

Set to “everywhere” when you want to hide the term everywhere (hidden everywhere in the UI and also hidden in download files). For example:

• Entity names: Household (PCO_0000024); Participant (EUPATH_0000096); Sample (EUPATH_0000609),etc.
• Subsets of variables that a provider would like to make available in the future. For example: MAL-ED data is available on ClinEpiDB.org, but the study team is actively working on cog dev publication & would like to keep cog dev variables private until the paper is published

Set to “variableTree” when you do NOT want the term to appear in the variable tree (subsetting & visualize tabs), but you DO want the term to appear in download files.

• For example: Cluster ID (EUPATH_0035024) in WASH-b Kenya

Set to “downloads” when you do NOT want the term to appear in the download files, but you DO want the term to appear in the variable tree.

• For example: Birth date (EFO_0004950) in public studies like PRISM (This will be rare)

Leave blank for terms that should appear everywhere. Most terms in a study should be left blank

Leave blank for all categories, unless ALL terms under that parent have been hidden.

• For example: MAL-ED data is available on ClinEpiDB.org, but the study team is actively working on cog dev publication & would like to keep cog dev variables private until the paper is published. “Cognitive development” is the parent category for all of these cog dev variables and gets annotated accordingly

Note: additional annotations for “hidden” can be added as needed going forward. For example, if we want to hide a term from being used in viz apps, annotate “hidden” with “visualizations”.

Multiple values can be allowed, allowing for increases in purview going forward. For example, set to “visualizations, download” for terms that should NOT appear in the visualizations or download files, but should appear elsewhere in the UI As of b55, there is no use case for this.

[jaycolin]

For the near future this can be assigned to me. I have to handle the cases where variables must be excluded from downloads; the other case of excluding from variable tree is already handled. For future, speculative cases where variables must be hidden from viz, etc. I'll populate scope in the db so that UI can check it. Steve can tell me what value should mean "hide from viz" and I'll load it that way.

[danicahelb]

Jay has updated all studies so that they have columns for both HIDDEN and display type = hidden

Here is every instance of HIDDEN in the studies currently loaded (note that the studies will need to be reloaded or patched to correct HIDDEN):

BRAZIL0001_1    EFO_0004950 Birth date  ["download"]
BRAZIL0001_1    EUPATH_0022001  Settlement  ["download"]
BRAZIL0001_1    EUPATH_0022071  Hospitalization year    ["download"]
BRAZIL0001_1    EUPATH_0022072  Hospitalization year, 2 ["download"]
BRAZIL0001_1    EUPATH_0022073  Hospitalization year, 3 ["download"]
BRAZIL0001_1    EUPATH_0022074  Hospitalization city    ["download"]
BRAZIL0001_1    EUPATH_0022075  Hospitalization city, 2 ["download"]
BRAZIL0001_1    EUPATH_0022076  Hospitalization city, 3 ["download"]
BRAZIL0001_1    EUPATH_0022088  Hospitalization reason  ["download"]
BRAZIL0001_1    EUPATH_0022089  Hospitalization reason, 2   ["download"]
BRAZIL0001_1    EUPATH_0022090  Hospitalization reason, 3   ["download"]
BRAZIL0001_1    EUPATH_0022174  Second household ID ["download"]
BRAZIL0001_1    EUPATH_0022176  Third household ID  ["download"]
BRAZIL0001_1    EUPATH_0022178  State lived in previously, 2    ["download"]
BRAZIL0001_1    EUPATH_0022179  State lived in previously, 3    ["download"]
BRAZIL0001_1    EUPATH_0022180  State lived in previously, 4    ["download"]
BRAZIL0001_1    EUPATH_0022181  State lived in previously, 5    ["download"]
BRAZIL0001_1    EUPATH_0022182  First dwelling settlement   ["download"]
BRAZIL0001_1    EUPATH_0022183  Second dwelling settlement  ["download"]
BRAZIL0001_1    EUPATH_0022184  Third dwelling settlement   ["download"]
INDIABCS1_1 EUPATH_0021308  Previous participant ID ["download"]
INDIABCS1_1 EUPATH_0021310  Previous household ID   ["download"]
LLINEUP01_1 EUPATH_0035024  Cluster ID  ["download"]
PROVIDE01_1 EFO_0004950 Birth date  ["download"]
SCOREMZ01_1 OBI_0001620 Latitude    ["everywhere"]
SCOREMZ01_1 OBI_0001621 Longitude   ["everywhere"]
SCORESL01_1 EUPATH_0030004  Village code    ["download"]
SCORESL01_1 OBI_0001620 Latitude    ["download"]
SCORESL01_1 OBI_0001621 Longitude   ["download"]
SCORESM01_1 ENVO_00000009   Country ["download","variableTree"]
SCORESM01_1 EUPATH_0000047  Hemoglobin  ["download","variableTree"]
SCORESM01_1 EUPATH_0000090  Malaria diagnosis   ["download","variableTree"]
SCORESM01_1 EUPATH_0010075  Height  ["download","variableTree"]
SCORESM01_1 EUPATH_0010607  Ascaris lumbricoides, by microscopy ["download","variableTree"]
SCORESM01_1 EUPATH_0010615  Trichuris trichiura, by microscopy  ["download","variableTree"]
SCORESM01_1 EUPATH_0010734  Hookworm, by microscopy ["download","variableTree"]
SCORESM01_1 EUPATH_0030024  Schistosoma mansoni infection intensity, by microscopy ["download","variableTree"]
SCORESM01_1 EUPATH_0030121  Schistosoma mansoni, by microscopy  ["download","variableTree"]
SCORESM01_1 EUPATH_0044175  Village study arm   ["download","variableTree"]
SCORESM01_1 IAO_0000414 Weight  ["download","variableTree"]
SCORESM01_1 OBI_0001169 Age ["download","variableTree"]
SCORESM01_1 OBI_0001508 Study timepoint ["download","variableTree"]
SCORESM01_1 PATO_0000047    Sex ["download","variableTree"]
UMSP00001_1 OBI_0001620 Latitude    ["download"]
UMSP00001_1 OBI_0001621 Longitude   ["download"]
WASHBBANG_1 EUPATH_0035024  Cluster ID  ["download"]
WASHBBANG_1 EUPATH_0035026  Randomization block ["download"]
WASHBKEN0_1 EUPATH_0035024  Cluster ID  ["download"]
WASHBKEN0_1 EUPATH_0035026  Randomization block ["download"]

@steve-fischer-200 we are ready for you to:

1. update the java code to include the new HIDDEN field
2. update the client code to use HIDDEN
3. remove the previous display type == hidden from the db whenever we get a chance

Once Steve has updated the UI to handle HIDDEN, @jaycolin will patch / reload to correct the annotations

[danicahelb]

@jaycolin has patched all studies to the latest values from .owl(s). NOTE displayType is still in use, so don't get confused. In the future the only value in DISPLAY_TYPE will be multifilter or null.