search

jhu-bids / TermHub

Web app and CLI tools for working with biomedical terminologies. https://github.com/orgs/jhu-bids/projects/9/views/7
https://bit.ly/termhub
GNU General Public License v3.0
8 stars 10 forks source link

Bug: `fetch_failures_0_members` fails on: actual 0 members csets #761

Open joeflack4 opened 2 months ago

joeflack4 commented 2 months ago

Overview

There are actually cases where there are >1 expression items, but 0 members in the expansion. This creates a (minor) issue where, at the end of fetch_failures_0_members, after it successfully runs & handles all other outstanding cases, it will throw its own exception because it detected a failure that it could not address. This makes shows up as a "failed run" in the GitHub action log and makes it appear on the surface that something is wrong.

Possible solutions

a. Permanent, programmatic solution

In such cases, we can do a query to see if all of the conditions in the "possible cause" section are met. For includeDescendants, I think this involves simply seeing if none of the concept IDs are listed as ancestors in concept_ancestor, and for any includeMapped, I think it entails making sure that there is no entry where the concept IDs appear in concept_relationship.

b. Manually address as issue arises

For now, I've marked the 1 problematic cset, 337213998 as having been successfully handled. So it will no longer appear in fetch_failures_0_members runs and throw an error. I did this by first checking its expressions in concept_set_version_item, and seeing if I felt that they matched this condition of actually having 0 members. Of course, I can't be fully sure without running such a query mentioned in the programmatic solution.

In any case, I manually set a success_datetime timestamp, and appended this to the comment:

Manually setting to success because it appears that 100% of the expressions are isExcluded=true and includeDescendants=true and includeMapped=true, but it seems that there are 0 mapped or descendant concepts.

So, the solution here is that when future situations arise, we apply this same procedure.

Additional information

Possible cause

I think that this can only happen when all of the following conditions are met, and I suspect that this cset meets these conditions:

  1. 100% of isExcluded = true
  2. If any includeDescendants=true, 100% of those expression concepts have 0 descendants
  3. If any includeMapped=true, 100% of those expression concepts have 0 mapped concepts

Real world case

I only know of 1 cset where this is the case: 337213998 (Mental Health).

Has 0 members in both the Enclave and TermHub.

All 852 expressions set to:

joeflack4 commented 2 months ago

If this happens again before November, maybe worry about doing.