Subject counts next to ontology concepts and modfiers

[WiseIndian]

This feature brings the display of subject counts per concept present in the ontology.

User benefit: a privileged user can see which concepts have no observations and which are densely populated and optimize inclusion/exclusion criteria.

The related back-end PR is here.

[jimmydjabali]

@WiseIndian I pulled the front&back branchs but I only see the loading count , then nothing more :

Did I missed something ?

[WiseIndian]

Thanks for the review. Did you build the PR for the backend too? Did you run the python script to update the subject counts of the ontology?

[jimmydjabali]

I didn't ran the python script, as I simply didn't know there were one to run. The make build or make up command (for the back-end) should not make it automatically (in Docker) ? If it really can't, does it is actually documented ? I didn't found any information about it

[WiseIndian]

Yeah it is not automated. Executing this script automatically in the build phase is an interesting idea. But it could pose some problems in a development environment. Indeed it takes some time when the ontology gets large to execute this script. And as the build phase is already time consuming I avoided doing that.

But it would be a good idea to include this kind of automation you suggest for a production environment like the network deployment.

Yeah I did not attach some documentation except for the comments present in the script itself. Here is an excerpt from the description of the script from the backend PR:

Creation of a script updating the totalnum column. This script is actually generated via a python script which can generate multiple version of a psql script updating the totalnum column. This python script takes as input: • the schema and table containing the ontology to update • the schema where the observation_fact table is present • Whether the totalnum column will contain the number of patients associated to a concept or the number of observations related to that concept. Once the python script is done, the PSQL script is generated and can be run by the person responsible for updating the database. Since the totalnum column must be updated on each medco node. The database of each medco node has to be updated using this script. This script functions in a recursive manner. The set of patients related to a concept is first gathered for concepts (or modifiers) with location being the deepest in the ontology (leaf nodes). Then those patients sets are propagated to parent concepts (or modifiers).

Basically you need to generate the postgresql script to update the totalnum column by executing that python script. The python script will generate a script personalized for the database of your choice (table, schema, ...). Once the python script is executed, you can execute the resulting PSQL script on the different i2b2 databases from the medco nodes (the 3 nodes in the development profile).

You can execute this bash script which will execute the python script and then execute the resulting PSQL script for updating the totalnum columns. Replace the schema and table by the ones of your choice (for example, set the _tablecopy variable to e2etest for the e2etest ontology):

export PSQL_PARAMS="-h localhost -p 5432 -U postgres"
export I2B2_DB_USER="postgres";
export PGPASSWORD=postgres

table_copy=sphn
ontology_schema=medco_ont
option=p #updating the counts to the number of patients per 

python3 generateUpdateTotalnumScript.py "${ontology_schema}" "${table_copy}" i2b2 i2b2demodata_i2b2 "${option}"

#To be executed after generateUpdateTotalnumScript.py has been executed to generate the updateTotalnum script.
for i in 0 1 2; do
    export I2B2_DB_NAME="i2b2medcosrv${i}"
    echo "============================"
    echo "Starting totalnum update in server $i"
    echo "============================"
    seconds1=$(date --date "$(echo "$datetime1" | sed -nr 's/(....)(..)(..)(..)(..)(..)/\1-\2-\3 \4:\5:\6/p')" +%s)

    time psql $PSQL_PARAMS -d "${I2B2_DB_NAME}" -f updateTotalnum.psql # --echo-all

    seconds2=$(date --date "$(echo "$datetime2" | sed -nr 's/(....)(..)(..)(..)(..)(..)/\1-\2-\3 \4:\5:\6/p')" +%s)
    delta=$((seconds1 - seconds2))
    echo "time ellapsed: ${delta}"
done

I will try to write some more explanation that I will include for the medco documentation in the following days.

[WiseIndian]

I have a question for you @jimmydjabali. Do the loading count disappear at some point or are they displayed forever? Thanks for taking the time for this review process.

[jimmydjabali]

@WiseIndian It appear for around 3-4 seconds and then disappear. If I close and open again it does not show it again.

[WiseIndian]

Thanks for the feedback. I was scared there was a hidden bug.