Ability to highlight "variants of interest"

NShaforostov commented 3 years ago

Background

VCF files may include a large number of variations and it would be helpful to highlight variants of interest based on INFO field values. NGB shall provide an option to configure set of filters (highlight profiles) and highlight variants matching filter in Variation table and VCF track.

Approach

API:

Add the ability to read the conditions that define "variants of interest" from the JSON-file (e.g. interest_profiles.json). Such file will be placed on the server side. Content from that file should be read every time on the data update (to provide a hot swap of the conditions if needs). This file should contain a list of profiles. Each of these profiles shall include an own set of conditions by which "variants of interest" can be defined.

The format of that file should be the following:

{
  "<profile_name1>" : {
    "is_default" : "<is_default_value>",
    "conditions" : [
      {
        "condition" : "<condition_set1>",
        "highlight_color" : "<highlight_color1>"
      },
      {
        "condition" : "<condition_set2>",
        "highlight_color" : "<highlight_color2>"
      },
      ...
    ]
  },
  "<profile_name2>" : {
    "conditions" : [ ... ]
  },
  ...
}

Where:

"" - a profile name
"is_default" (boolean, not-required) - key for specifying whether the current profile is default one. If several profiles are marked as "default" profile (have "is_default" : "true") - only first of them becomes "default" profile, other are ignored
"conditions" - an array of condition sets. For each set should be specified conditions for variants' matching check and the color in which these variants will be highlighted:
- "condition" - a condition set. May include one or several conditions for VCF info fields which values may be of interest (for which the conditions are being specified and comparisons will be performed). If the condition set is not specified - no variants will be highlighted in the corresponding color
- "highlight_color" - key for specifying a color. The variants matched the conditions of the current set will be highlighted in the GUI in this color (color should be specified in HEX)

Each <condition_set> should have a structure:

(<id1> <comparison_operator1> <value1>) <logic_operator1> (<id2> <comparison_operator2> <value2>) ...

Where:

\<id> (string) - VCF info field ID. Should be specified in quotes (examples: 'ac', "mp", 'excess het')
\<comparison_operator> - operator that will be used for the comparison of the VCF info field value with the specified in the current condition. Possible values:
- == - equals - for string, numeric and boolean values
- != - not equals - for string, numeric and boolean values
- > - greater than - for numeric values
- >= - greater than or equal to - for numeric values
- < - less than - for numeric values
- <= - less than or equal to - for numeric values
- in - in - for arrays
- notin - not in - for arrays
\<value> (string) - value with which the comparison is being performed. Should be specified in quotes, arrays should be specified in square brackets (examples: '2', "true", '[1, 10, 22]')
\<logic_operator> - defines how specified comparisons will combinate:
- variants of interest should match both comparisons. In this case, \<logic_operator> should be and
- variants of interest should match any comparison (at least one). In this case, \<logic_operator> should be or

There should be the ability to make up more complex sets from several comparisons by rules of Boolean algebra, using additional brackets, e.g.: ((<comparison1>) or (<comparison2>)) and ((<comparison3>) or (<comparison4>)).

Example of the profile:

{
  "two_alleles_high_mapping_quality" : {
    "is_default" : true,
    "conditions" : [
      {
        "highlight_color" : "ffbdbd",
        "condition" : "('ac' == '2') and ('mq' >= '80')"
      }
    ]
  }
}

By the example above, the default profile was described that has the following properties - variants of interest will be highlighted in color #FFBDBD if they match both conditions: allele count equals 2 and mapping quality greater than or equals to 80.

GUI:

Users shall have the ability to enable/disable highlighting of the "variants of interests". For that, the corresponding block shall be added into the VCF tab of the main settings, e.g.:

When the corresponding checkbox is enabled, the dropdown list with all profiles shall appear, e.g.:

In this dropdown list:

elements should be sorted alphabetically (by profile name)
the default profile should be selected
if no default profile is defined, there is no selected profile should be in the list
without profile selecting, the changes can't be saved
multi-select is not allowed

When user selects the profile and saves changes:

at the VARIANTS panel, rows background of those variants that matches "interest" conditions, should be highlighted in the profile color, e.g.:
at the BROWSER form, for the vcf tracks, background of those variants that matches "interest" conditions, should be highlighted in the profile color, e.g.: The highlighted zone should contain all positions taken by the certain variant (one or several nucleotide widths according to the variant type), e.g.:
- additionally, for BND type of variants:
- label with coordinates (when the "start"/"end" position is "out" of the screen or point to other chromosome) should be highlighted in a condition color too
- in case, if the certain BND variant doesn't match the profile conditions and shall not be highlighted - described label should not be highlighted at all (currently, it is being always highlighted in yellow)
if the same variant matches different condition sets from the profile, it should be highlighted only in a color of the first condition set in the list
also at the BROWSER form, for the collapsed vcf tracks, when several variants are merged into a "bubble" and among them there are some variants of interest from different condition sets - such "bubble" should be highlighted in all corresponding colors (like pie chart diagram). Details of "bubble" highlighting:
- if in the "bubble" several variations were merged and these variations were highlighted in different colors, a "bubble" pie chart should be drawn proportionally to count of different highlighting colors (not to highlighting counts of each color). In this case, lack of highlighting should be count as separate color. For example, if 7 variants were merged into 1 "bubble" and before the merging 4 of these variants were highlighted in blue, 2 - in red, 1 - without highlighting - after merging, the "bubble" should be "divided" to 3 fully size-equal parts - blue, red and transparent (without highlighting):
- "bubble" color highlighting should be being performed only at a small zoom out. At a large zoom out, when in a single "bubble" lots of variations are being merged, "bubble" should be fully transparent (current behavior) - highlighting of separate variations in this "bubble" should not be taken into account

rodichenko commented 3 years ago

@NShaforostov @mzueva

API: Add the ability to read the conditions that define "variants of interest" from the JSON-file (e.g. interest_conditions.json). Such file will be placed on the server side

When the functionality is enabled:

at the VARIANTS panel, rows background of those variants that matches "interest" conditions, should be highlighted in the configured color, e.g.:

how these conditions will be calculated - on the server-side (by providing a calculated result as a property of variant, e.g. highlighted=true) or on the client?

If we choose the server-side calculation, we need such highlighted property for each variant;
If we choose the client-side calculation, we need an API method for retrieving such conditions.

TBD for client-side calculation: how often should we fetch those conditions: on every VCF data request / once per session / once per datasets etc.?

mzueva commented 3 years ago

@rodichenko I think highlight value shall be calculated on the client, I'd suggest to use an existing endpoint "/defaultTrackSettings" to get these settings.

rodichenko commented 3 years ago

@mzueva sure, it's definitely the only option after the last edition of the issue 😃

NShaforostov commented 3 years ago

Possible enhancements

1. Short profiles info in the Settings

As only admins can configure profiles, it may be useful to show info about the certain profile - for "general" users. For that, add an "Info" icon near the profile selected in the Settings, e.g.: When user clicks that icon - the profile short description is shown in a tooltip, e.g.:

Here, a list of all configured profile's conditions and the corresponding highlighting colors should be displayed.

2. Highlighting on big scales

Think about highlighting variants of interest, when the big scale is set for the browser. Via the implementation described above (in the issue), variants of interest can be not very noticeable if for the browser the big scale is set. Perhaps, another approach is required for highlighting in such case.

NShaforostov commented 3 years ago

Example of the testing profile:

{
    "testing_profile" : {
        "is_default" : true,
        "conditions" : [
            {
                "highlight_color" : "0083FF",
                "condition" : "('ReadPosRankSum' > '1') and ('ReadPosRankSum' < '3') and ('Dp' <= '200')"
            },
            {
                "highlight_color" : "FF00B2",
                "condition" : "('ReadPosRankSum' < '-1') and ('ReadPosRankSum' > '-3') and ('Dp' <= '200')"
            },
            {
                "highlight_color" : "D8FF00",
                "condition" : "(('BaseQRankSum' > '1') or ('BaseQRankSum' < '-1'))  and ('Mq' > '83') and ('Dp' > '260')"
            },
            {
                "highlight_color" : "149150",
                "condition" : "('Dp' in '[386, 435]') and ('Sor' == '1.461')"
            },
            {
                "highlight_color" : "CE632D",
                "condition" : "('Dp' in '[386, 435]') and (('Sor' == '1.461') or ('Sor' == '1.227'))"
            }
        ]
    }
}

Should be tested on "dm6_data" dataset. Check the following position:

12592259 - here should be 2 variants highlighted by different colors (0083FF and FF00B2)
12591884-12591885 - here should be 2 neighboring variants highlighted by the same color (D8FF00)
12591627-12591629 - here should be 2 neighboring variants highlighted by different colors (149150 and CE632D)

epam / NGB