Failure when overwriting small+svs in a case that had small vars only

[holtgrewe]

Describe the bug Importing variants fails when overwitring a case with small+svs that had small variants only.

To Reproduce Steps to reproduce the behavior:

1. Import case with small variants.
2. Override with small and structural variants.
3. See error

Expected behavior Import should work.

Additional context

• The logs say Import case file failed: Error: insert or update on table "variants_case" violates foreign key constraint "variants_case_latest_variant_set_i_65a85e18_fk_variants_" DETAIL: Key (latest_variant_set_id)=(5653) is not present in table "variants_smallvariantset".
• Internal Link: https://varfish.bihealth.org/importer/17328747-5a36-43bc-900d-5941008c872a/import/20cc16b2-0f39-45c4-8fd6-e14af8023efb/
• Maybe a duplicate as more than one variant set is there #497
• Follow-Up: #499

[stolpeo]

I can't replicate the issue.

Importing small variants with

varfish-cli case create-import-info <project_uuid> bwa.gatk_hc.varfish_annotated.case1.* case1.ped

Then force overwriting small variants and importing structural variants with

varfish-cli case create-import-info <project_uuid> --force bwa.*.varfish_annotated.case1.* case1.ped

Variants are imported as expected.

Using varfish-cli 0.3.2 and varfish main branch using GRCh37.

[stolpeo]

According to @eudesbarbosa , the import command that has been used was part of cubi-tk:

cubi-tk snappy varfish-upload --min-batch 207 <project_uuid> --yes

https://github.com/bihealth/cubi-tk/blob/master/cubi_tk/snappy/varfish_upload.py#L236

[stolpeo]

Most likely the varfish CLI version that has been used in cubi tk was varfish-cli 0.2.8 pypi_0 pypi

[stolpeo]

Running the import with varfish-cli 0.2.8 works as well in my setup.

[holtgrewe]

@stolpeo could you try to re-import svs and small vars indepently now?

[stolpeo]

Cubi TK seems to not use the --force option. However, omitting it didn't change anything. Also, SVs and smallvars seem to be imported in one step. One difference I noticed is that in the failing example, gcnv and xhmm were imported for SVs while I only imported xhmm results. Importing both didn't provoke the bug.

[stolpeo]

I picked a different case, but it's still working fine.

[holtgrewe]

I tried to reproduce on the original problematic data set but it now works.

[stolpeo]

One thing I noticed is that when you trigger a smallvars import and a SVs import rapidly after each other (while the first import is still running), it causes a different type of crash:

Traceback (most recent call last):
  File "/home/oliver/miniconda3/envs/varfish/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/oliver/miniconda3/envs/varfish/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/oliver/dev/varfish-server/importer/tasks.py", line 18, in run_import_case_bg_job
    return models.run_import_case_bg_job(pk=pk)
  File "/home/oliver/dev/varfish-server/importer/models.py", line 322, in run_import_case_bg_job
    CaseImporter(job).run()
  File "/home/oliver/dev/varfish-server/importer/models.py", line 403, in run
    raise RuntimeError("Problem during variant import ") from e
RuntimeError: Problem during variant import 

[stolpeo]

@holtgrewe should we backlog the ticket then if the error occurs again?

[holtgrewe]

We fixed related issues but could not reproduce this one. Maybe this is already resolved with https://github.com/bihealth/varfish-cli/pull/14.

We close this now as "not planned" and re-open when the problem reoccurs.