SV gene variantS search

dnil commented 3 months ago

This PR adds a functionality or fixes a bug.

Fix #1179 - With mongodb 5.0 and higher it is possible to hold multiple partial indices on the same collection and keys, given that the partial criteria do not overlap. Use that to be able to set up a gene variantS search for the SV category as well. An initial test seemed decently quick, so maybe we will not need to filter on size, which is a bit complicated with inf and small sizes in the bag.
The hits may be a bit messy with large variants, so we will at least add a filter on category to the search page. A "length" filter could be considered as well; could be part of this PR or a later issue if it is needed.
Backgrounding is not needed with the new mongodb 7.0 indexing logic. They do a blocking lock during the start and finish of the indexing, but otherwise runs a very fast, but still read-write indexing. Neato!

We prepared for this during the db migration, and have a category-less version of this index in place on prod.

The stage collection built quickly, opting for the same style as on prod, ie one index for all variant categories. rankscore >= 5 effectively excludes most of cancer, Mei variants, etc that are not yet ranked.

Testing on cg-vm1 server (Clinical Genomics Stockholm)

**Prepare for testing** 1. Make sure the PR is pushed and available on [Docker Hub](https://hub.docker.com/repository/docker/clinicalgenomics/scout-server-stage) 1. Fist book your testing time using the Pax software available at [https://pax.scilifelab.se/](https://pax.scilifelab.se). The resource you are going to call dibs on is `scout-stage` and the server is `cg-vm1`. 1. `ssh @cg-vm1.scilifelab.se` 1. `sudo -iu hiseq.clinical` 1. `ssh localhost` 1. (optional) Find out which scout branch is currently deployed on cg-vm1: `podman ps` 1. Stop the service with current deployed branch: `systemctl --user stop scout.target` 1. Start the scout service with the branch to test: `systemctl --user start scout@` 1. Make sure the branch is deployed: `systemctl --user status scout.target` 1. After testing is done, repeat procedure at [https://pax.scilifelab.se/](https://pax.scilifelab.se), which will release the allocated resource (`scout-stage`) to be used for testing by other users.

Testing on hasta server (Clinical Genomics Stockholm)

**Prepare for testing** 1. `ssh @hasta.scilifelab.se` 1. Book your testing time using the Pax software. `us; paxa -u -s hasta -r scout-stage`. You can also use the WSGI Pax app available at [https://pax.scilifelab.se/](https://pax.scilifelab.se). 1. (optional) Find out which scout branch is currently deployed on cg-vm1: `conda activate S_scout; pip freeze | grep scout-browser` 1. Deploy the branch to test: `bash /home/proj/production/servers/resources/hasta.scilifelab.se/update-tool-stage.sh -e S_scout -t scout -b ` 1. Make sure the branch is deployed: `us; scout --version` 1. After testing is done, repeat the `paxa` procedure, which will release the allocated resource (`scout-stage`) to be used for testing by other users.

How to test:

search for a few genes on stage, note both SVs and SNVs in results
toggle search to only SNV or only SV and see that the filter behaves

Expected outcome: The functionality should be working Take a screenshot and attach or copy/paste the output.

Review:

[x] code approved by CR
[x] tests executed by DN

codecov[bot] commented 3 months ago

Codecov Report

Attention: Patch coverage is 87.50000% with 1 line in your changes missing coverage. Please review.

Project coverage is 84.44%. Comparing base (4dd74e1) to head (47b912d). Report is 1 commits behind head on main.

Files	Patch %	Lines
scout/adapter/mongo/query.py	75.00%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #4692 +/- ## ========================================== - Coverage 84.44% 84.44% -0.01% ========================================== Files 311 311 Lines 18787 18793 +6 ========================================== + Hits 15865 15870 +5 - Misses 2922 2923 +1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

dnil commented 3 months ago

I'll just add the search-page filter part here as well I guess; it won't make sense without the index anyway. Draft again for now then.