Sometimes VV returns random output that makes no sense - possible race condition

[ifokkema]

Describe the bug I believe @beboche mentioned this before, but perhaps that was over email and not as an issue because I can't find it.

I've been breaking my head for hours on why I get random fails of my tests and new code. After digging and digging and digging, I had to conclude VV just randomly gave me a different output, and I debugged the connection.

To Reproduce You need to be patient to reproduce this. I don't know why or when this happens - it seems random. I was calling the following URL repeatedly until I got JSON back which made no sense. https://rest.variantvalidator.org/VariantValidator/variantvalidator/GRCh37/NM_004006.3%3Ac.100del/all?content-type=application%2Fjson

Expected behavior I was expecting, and usually getting, the following output:

{
  "NM_004006.3:c.101del": {
    "alt_genomic_loci": [],
    "annotations": {
      "chromosome": "X",
      "db_xref": {
        "CCDS": "CCDS14233.1",
        "ensemblgene": null,
        "hgnc": "HGNC:2928",
        "ncbigene": "1756",
        "select": "MANE"
      },
      "ensembl_select": false,
      "mane_plus_clinical": false,
      "mane_select": true,
      "map": "Xp21.2-p21.1",
      "note": "dystrophin",
      "refseq_select": true,
      "variant": "DP427M"
    },
    "gene_ids": {
      "ccds_ids": [
        "CCDS14231",
        "CCDS14234",
        "CCDS48091",
        "CCDS55395",
        "CCDS14232",
        "CCDS75965",
        "CCDS14233",
        "CCDS55394"
      ],
      "ensembl_gene_id": "ENSG00000198947",
      "entrez_gene_id": "1756",
      "hgnc_id": "HGNC:2928",
      "omim_id": [
        "300377"
      ],
      "ucsc_id": "uc004dda.2"
    },
    "gene_symbol": "DMD",
    "genome_context_intronic_sequence": "",
    "hgvs_lrg_transcript_variant": "",
    "hgvs_lrg_variant": "",
    "hgvs_predicted_protein_consequence": {
      "lrg_slr": "",
      "lrg_tlr": "",
      "slr": "NP_003997.2:p.(K34Sfs*18)",
      "tlr": "NP_003997.2:p.(Lys34SerfsTer18)"
    },
    "hgvs_refseqgene_variant": "",
    "hgvs_transcript_variant": "NM_004006.3:c.101del",
    "primary_assembly_loci": {
      "grch37": {
        "hgvs_genomic_description": "NC_000023.10:g.32867931del",
        "vcf": {
          "alt": "C",
          "chr": "X",
          "pos": "32867929",
          "ref": "CT"
        }
      },
      "grch38": {
        "hgvs_genomic_description": "NC_000023.11:g.32849814del",
        "vcf": {
          "alt": "C",
          "chr": "X",
          "pos": "32849812",
          "ref": "CT"
        }
      },
      "hg19": {
        "hgvs_genomic_description": "NC_000023.10:g.32867931del",
        "vcf": {
          "alt": "C",
          "chr": "chrX",
          "pos": "32867929",
          "ref": "CT"
        }
      },
      "hg38": {
        "hgvs_genomic_description": "NC_000023.11:g.32849814del",
        "vcf": {
          "alt": "C",
          "chr": "chrX",
          "pos": "32849812",
          "ref": "CT"
        }
      }
    },
    "reference_sequence_records": {
      "protein": "https://www.ncbi.nlm.nih.gov/nuccore/NP_003997.2",
      "transcript": "https://www.ncbi.nlm.nih.gov/nuccore/NM_004006.3"
    },
    "refseqgene_context_intronic_sequence": "",
    "selected_assembly": "GRCh37",
    "submitted_variant": "NM_004006.3:c.100del",
    "transcript_description": "Homo sapiens dystrophin (DMD), transcript variant Dp427m, mRNA",
    "validation_warnings": [
      "NM_004006.3:c.100del automapped to NM_004006.3:c.101delA"
    ],
    "variant_exonic_positions": {
      "NC_000023.10": {
        "end_exon": "3",
        "start_exon": "3"
      },
      "NC_000023.11": {
        "end_exon": "3",
        "start_exon": "3"
      }
    }
  },
  "flag": "gene_variant",
  "metadata": {
    "variantvalidator_hgvs_version": "2.0.1",
    "variantvalidator_version": "2.1.1.dev34+gc25d2c9",
    "vvdb_version": "vvdb_2022_04",
    "vvseqrepo_db": "VV_SR_2022_02/master",
    "vvta_version": "vvta_2022_02"
  }
}

However, during debugging I suddenly got this:

{
  "flag": "warning",
  "metadata": {
    "variantvalidator_hgvs_version": "2.0.1",
    "variantvalidator_version": "2.1.1.dev34+gc25d2c9",
    "vvdb_version": "vvdb_2022_04",
    "vvseqrepo_db": "VV_SR_2022_02/master",
    "vvta_version": "vvta_2022_02"
  },
  "validation_warning_1": {
    "alt_genomic_loci": [],
    "annotations": {},
    "gene_ids": {},
    "gene_symbol": "",
    "genome_context_intronic_sequence": "",
    "hgvs_lrg_transcript_variant": "",
    "hgvs_lrg_variant": "",
    "hgvs_predicted_protein_consequence": {
      "lrg_slr": "",
      "lrg_tlr": "",
      "slr": "",
      "tlr": ""
    },
    "hgvs_refseqgene_variant": "",
    "hgvs_transcript_variant": "",
    "primary_assembly_loci": {},
    "reference_sequence_records": "",
    "refseqgene_context_intronic_sequence": "",
    "selected_assembly": "hg19",
    "submitted_variant": "NM_007294.4:c.5092G>T",
    "transcript_description": "",
    "validation_warnings": [],
    "variant_exonic_positions": null
  }
}

The data is missing, the warning seems to point to nowhere, and the submitted_variant makes no sense. When I submit that variant, it actually seems to work fine. So this may be:

• A race condition that not only selects output for a different variant but also breaks something inside VV that makes it return this almost-empty JSON.
• Some other condition that perhaps returns some hard-coded output or some cached file from disk or so.

Could you please check your Apache logs if somebody submitted NM_007294.4:c.5092G>T just now? (note; you'd likely need to apply URL encoding while grepping, or grep parts of the variant) I submitted it once to confirm that the variant works. You'll be able to find my IP address by just grepping the logs for NM_004006.3%3Ac.100del that I submitted a LOT today. If a different IP submitted the c.5092G>T variant while I was busy with my c.100del variant, then we have concluded that a race condition exists in VV. It may not solve the error yet, but at least it gives an idea of where to start.

[vidboda]

Hi both,

I can confirm that I already mentionned this issue, and discussed it with @Peter-J-Freeman, but mainly outside github. The issue seemd to be less occuring since the last VV code upgrade a few weeks ago, however I can confirm that for about 1 week it is occuring again, in my case on a daily basis via MobiDetails. The "submitted variant" returned in the json containing the error is totally different from the real submitted variant. I'm also thinking of a race condition, but I was also suspecting it to come from my app. Apparently this is not the case (and I found nothing from my side).

[Peter-J-Freeman]

Thanks for the report

I have restarted all the services. See if that helps. If it does, I will add a cron job to restart all services weekly

[ifokkema]

I can confirm that I already mentioned this issue (...) I can confirm that for about one week, it has been occurring again, in my case, on a daily basis via MobiDetails.

Thanks, @beboche!

I have restarted all the services. See if that helps. If it does, I will add a cron job to restart all services weekly.

I can imagine that if you don't have time to look into it right now, it might be worth patching it up this way. I would recommend, however, keeping this issue open until the cause has been found and fixed.

For now, I'll start a script looping the same request every 20 seconds. That's not a lot, but I assume the issue is a race condition with another user, and 4320 timepoints per day should be OK. Whenever an error occurs, I will log the time and output.

It would still be worth checking your Apache log to verify we're indeed talking about other users' input leaking into our output.

[Peter-J-Freeman]

@ifokkema . Might well be a race issue, but I know nothing about them. If you can find a way to trigger the issue, I'd appreciate the help in resolving this. I have time to look, I have just never understood how!!! :)

[ifokkema]

@ifokkema . Might well be a race issue, but I know nothing about them. If you can find a way to trigger the issue, I'd appreciate the help in resolving this. I have time to look, I have just never understood how!!! :)

Hehe, it greatly depends on your architecture. But it's really best to start simply grepping your Apache logs for the variant that I encountered; NM_007294.4:c.5092G>T. So:

1. Try something like grep "/NM_007294.4%3Ac.5092G%3ET/" /var/log/apache2/access.log Change the path to your apache logs for rest.variantvalidator.org. I just made a request, so you should find it. If you find only one request with this variant (today's), then your logs have been rotated since yesterday. Then check access.log.1 instead. We just need the timestamp(s) of any matching requests.
2. Then, use the timestamps to search for any other requests around the same time in the same file. You will hopefully see my requests for NM_004006.3%3Ac.100del there as well. If the time overlaps closely enough (max 1 second, I assume), then we have evidence that I got somebody else's variant in my output.

For now, I'm running this for a few minutes, and I found no issues yet:

while (true);
do
  sJSON=$(GET "https://rest.variantvalidator.org/VariantValidator/variantvalidator/GRCh37/NM_004006.3%3Ac.100del/all?content-type=application%2Fjson");
  sFlag=$(echo "${sJSON}" | jq -r .flag);
  echo -ne "${sFlag}\t";

  if [[ "$(date +%M | cut -b 2)" == "0" ]];
    then echo $(date "+%t%Y-%m-%d %H:%M") | tr '\n' '\t';
  fi;

  if [[ "${sFlag}" != "gene_variant" ]];
  then
    echo -n "writing data file";
    echo "${sJSON}" | jq . > "$(date +%Y-%m-%d.%H.%M.%S).VV-warning.json";
  fi;
  echo "";
  sleep 20s;
done

[ifokkema]

Several errors have already occurred. I'm seeing a range of variants that I didn't submit, and also once the variant that I did submit, but somehow it returned the same kind of warning without real data.

[vidboda]

I've also had the latter case once today

[ifokkema]

In https://github.com/LOVDnl/LOVD3/commit/face85423e0d6aa58a24a5272087b57348e93d0f, I've built in that LOVD will try VV once more when we get output like this. It'll help reduce issues on our side while VV is still affected by this issue. I am not sure if the LOVD endpoint shows the same issue, haven't updated that code yet. Just the VV endpoint code for now. If the issue is bothering you a lot, @beboche, you might want to do something similar.

[leicray]

I suspect that this issue has perhaps been around for nearly a month.

Starting on 16th August, I began to receive automated email error messages from VV relating to several different types of error, not just validation issues. They included:

• /accounts/confirm-email
• /service/validate
• /bed/
• /service/vcf2hgvs
• /service/gene2trans/

This suggests that it's not simply a problem with validation requests, but is more general. Was any major change made to VV on, or just before, 16th August?

[ifokkema]

Starting on 16th August, I began to receive automated email error messages from VV relating to several different types of error, not just validation issues.

Do you mean internal server errors? I got a few of those, too.

So far, I got:

for file in *.json;
do
  echo -ne "${file}\t";
  cat "${file}" | jq ".validation_warning_1.submitted_variant";
done
2022-09-08.14.36.21.VV-warning.json "NC_000017.11:g.7674258del"
2022-09-08.14.37.03.VV-warning.json "NM_004006.3:c.100del"
2022-09-08.14.37.44.VV-warning.json "NM_000314.6:c.802-3_802delTAGGinsAGGT"
2022-09-08.14.38.26.VV-warning.json null
2022-09-08.14.40.30.VV-warning.json "NM_181872.6:c.647C>T"
2022-09-08.14.43.36.VV-warning.json "NM_004006.3:c.100del"
2022-09-08.14.47.23.VV-warning.json "NM_000051.3:c.5006-40C>T"
2022-09-08.14.51.10.VV-warning.json "NM_000059.4:c.275A>G"
2022-09-08.14.54.36.VV-warning.json "NM_000059.4:c.6319C>T"
2022-09-08.15.08.58.VV-warning.json "NM_007294.3:c.548-149C>T"
2022-09-08.15.15.12.VV-warning.json "NM_004006.3:c.100del"
2022-09-08.15.49.24.VV-warning.json null
2022-09-08.17.19.27.VV-warning.json "OMIM*609132:c.1323-1delG - ENST00000400181"
2022-09-08.17.43.50.VV-warning.json "NM_006306.4:c.2974-2939A>G"
2022-09-08.22.03.25.VV-warning.json "NM_206933.4:c.4933G>T"
2022-09-09.09.16.43.VV-warning.json "NM_000545.8:c.256C>T"
2022-09-09.09.28.48.VV-warning.json "NM_006859.4:c.80C>G"
2022-09-09.11.21.51.VV-warning.json "NM_001318496.1:c.507+5G>A"
2022-09-09.12.39.07.VV-warning.json "NM_004006.3:c.100del"
2022-09-09.16.13.20.VV-warning.json "NM_000267.3:c.5873T>C"
2022-09-11.19.55.22.VV-warning.json "NM_000546.6:c.28G>C"
2022-09-11.19.56.44.VV-warning.json "NM_000546.6:c.175G>T"
2022-09-11.19.57.05.VV-warning.json "NM_000546.6:c.212C>A"
2022-09-11.19.57.46.VV-warning.json "NM_000546.6:c.292C>G"
2022-09-11.19.58.27.VV-warning.json "NM_000546.6:c.364G>C"
2022-09-11.19.58.47.VV-warning.json "NM_000546.6:c.400T>G"
2022-09-11.19.59.29.VV-warning.json "NM_000546.6:c.475G>C"
2022-09-11.19.59.49.VV-warning.json "NM_000546.6:c.513G>T"
2022-09-11.20.00.51.VV-warning.json "NM_000546.6:c.620A>C"
2022-09-11.20.01.32.VV-warning.json "NM_000546.6:c.692C>T"
2022-09-11.20.02.34.VV-warning.json "NM_000546.6:c.797G>A"
2022-09-11.20.03.15.VV-warning.json "NM_000546.6:c.871A>G"
2022-09-11.20.04.38.VV-warning.json "NM_000546.6:c.1020G>C"
2022-09-11.22.02.40.VV-warning.json "NM_000546.6:c.164C>A"
2022-09-11.22.03.00.VV-warning.json "NM_000546.6:c.199C>G"
2022-09-11.22.04.23.VV-warning.json "NM_000546.6:c.353C>A"
2022-09-11.22.05.25.VV-warning.json "NM_000546.6:c.464C>A"
2022-09-11.22.06.07.VV-warning.json "NM_000546.6:c.537T>A"
2022-09-11.22.06.27.VV-warning.json "NM_000546.6:c.571C>T"
2022-09-11.22.06.48.VV-warning.json "NM_000546.6:c.611A>G"
2022-09-11.22.07.30.VV-warning.json "NM_000546.6:c.684C>G"
2022-09-11.22.07.50.VV-warning.json "NM_000546.6:c.717C>G"
2022-09-11.22.08.11.VV-warning.json "NM_000546.6:c.751A>C"
2022-09-11.22.09.34.VV-warning.json "NM_000546.6:c.896T>C"
2022-09-11.22.09.54.VV-warning.json "NM_000546.6:c.931A>T"
2022-09-11.22.10.15.VV-warning.json "NM_000546.6:c.972T>A"
2022-09-11.22.10.35.VV-warning.json "NM_000546.6:c.1008G>T"
2022-09-11.22.10.56.VV-warning.json "NM_000546.6:c.1039G>A"
2022-09-11.22.11.16.VV-warning.json "NM_000546.6:c.1075C>A"
2022-09-11.22.11.37.VV-warning.json "NM_000546.6:c.1111T>A"
2022-09-12.03.43.44.VV-warning.json null
2022-09-12.09.03.03.VV-warning.json "NC_000023.11:g.32386310_32386465del"

(the null entries are internal server errors)

The time stamps and variants show that, indeed we're talking about input crossing over between requests. Also, it's likely that this occurs more frequently when multiple requests come in at the same time, like yesterday evening. Sometimes I do see my own variant there; in that case, the error is Failed to fetch NM_004006.3 from SeqRepo (/local/seqrepo/VV_SR_2022_02/master) ([Errno 2] bNo such file or directory).

[leicray]

@Peter-J-Freeman and I are admins for VV and we receive sometimes lengthy server error messages by email. AFAIK, only the admins receive these emails.

[vidboda]

Hi all,

@ifokkema is your script still running? I've just had about 10 of these errors during the last 15-20 minutes - if so maybe you could stop it as I think we have enough data to investigate. Btw, @Peter-J-Freeman could you investigate the apache logs and confirm the race condition?

[leicray]

Interestingly, I have received NO such error messages in that same time period.

[Peter-J-Freeman]

@leicray , this is the rest server. It operates differently.

@beboche I don't know what to make of the error log

Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block: error reading file
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block: error reading file
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] [E::bgzf_uncompress] Inflate operation failed: 1
CRC32 checksum mismatch[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)

[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::fai_retrieve] Failed to retrieve block: error reading file
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block: error reading file
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block: error reading file
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 1
[E::bgzf_uncompress] Inflate operation failed: 1
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] CRC32 checksum mismatch
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[E::bgzf_uncompress] Inflate operation failed: 3
[E::fai_retrieve] Failed to retrieve block. (Seeking in a compressed, .gzi unindexed, file?)
[Wed Sep 14 13:30:36.143210 2022] [wsgi:error] [pid 12180:tid 140680614708992] [remote 192.168.17.243:60180] CRITICAL: <class 'vvhgvs.exceptions.HGVSDataNotAvailableError'> Failed to fetch NC_000019.10 from SeqRepo (/local/seqrepo/VV_SR_2022_02/master) ([Errno 2] b'No such file or directory')

It looks like the SqLite3 database is failing somehow. @John-F-Wagstaff have you seen this before

@beboche @ifokkema I may need to go back further. What exactly am I looking for?

[vidboda]

well do you trace in apache log files the input variant for a given request? Our queries are GET queries, then in the apache access_log you should be able to look for the variants @ifokkema mentionned in URLs - then you should check if the second variant was asked by another user at ~the same time and confirm it on 2-3 others - @ifokkema would probably explain this better than me. The key is the GET URL in apache access_log

[vidboda]

ok then we can confirm the race condition:

those 2 lines:

XXX.XXX.XXX.XXX - - [14/Sep/2022:14:00:25 +0100] "GET /VariantValidator/variantvalidator/hg38/NM_000546.6:c.239C%3EG/NM_000546.6?content-type=application%2Fjson HTTP/1.1" 200 2240 "-" "python-requests/2.22.0"
YYY.YYY.YYY.YYY - - [14/Sep/2022:14:00:25 +0100] "GET /VariantValidator/variantvalidator/GRCh38/NM_001291303.3:c.569C%3ET/all?content-type=application/json HTTP/1.1" 200 2240 "-" "-"

the 2nd IP is MobiDetails - I've received an error email when the user submitted this variant:

VV check failed for variant NM_001291303.3:c.569C>T with args: {'NM_000546.6:c.239C>G': {'alt_genomic_loci': [], 'annotations': {'chromosome': '17', 'db_xref': {'CCDS': 'CCDS11118.1', 'ensemblgene': None, 'hgnc': 'HGNC:11998', 'ncbigene': '7157', 'select': 'MANE'}, 'ensembl_select': False, 'mane_plus_clinical': False, 'mane_select': True, 'map': '17p13.1', 'note': 'tumor protein p53', 'refseq_select': True, 'variant': '1'}, 'gene_ids': {'ccds_ids': ['CCDS11118', 'CCDS73963', 'CCDS73967', 'CCDS73966', 'CCDS73970', 'CCDS73969', 'CCDS73971', 'CCDS73968', 'CCDS45606', 'CCDS45605', 'CCDS73964', 'CCDS73965'], 'ensembl_gene_id': 'ENSG00000141510', 'entrez_gene_id': '7157', 'hgnc_id': 'HGNC:11998', 'omim_id': ['191170'], 'ucsc_id': 'uc060aur.1'}, 'gene_symbol': 'TP53', 'genome_context_intronic_sequence': '', 'hgvs_lrg_transcript_variant': '', 'hgvs_lrg_variant': '', 'hgvs_predicted_protein_consequence': {'lrg_slr': 'LRG_321p1:p.(P80R)', 'lrg_tlr': 'LRG_321p1:p.(Pro80Arg)', 'slr': 'NP_000537.3:p.(P80R)', 'tlr': 'NP_000537.3:p.(Pro80Arg)'}, 'hgvs_refseqgene_variant': '', 'hgvs_transcript_variant': 'NM_000546.6:c.239C>G', 'primary_assembly_loci': {'grch37': {'hgvs_genomic_description': 'NC_000017.10:g.7579448G>C', 'vcf': {'alt': 'C', 'chr': '17', 'pos': '7579448', 'ref': 'G'}}, 'grch38': {'hgvs_genomic_description': 'NC_000017.11:g.7676130G>C', 'vcf': {'alt': 'C', 'chr': '17', 'pos': '7676130', 'ref': 'G'}}, 'hg19': {'hgvs_genomic_description': 'NC_000017.10:g.7579448G>C', 'vcf': {'alt': 'C', 'chr': 'chr17', 'pos': '7579448', 'ref': 'G'}}, 'hg38': {'hgvs_genomic_description': 'NC_000017.11:g.7676130G>C', 'vcf': {'alt': 'C', 'chr': 'chr17', 'pos': '7676130', 'ref': 'G'}}}, 'reference_sequence_records': {'protein': '[https://www.ncbi.nlm.nih.gov/nuccore/NP_000537.3](https://urldefense.com/v3/__https://www.ncbi.nlm.nih.gov/nuccore/NP_000537.3__;!!DV4KuIgKKrh48VMFxQ!F-zIvz38DkmGCWD46UZn9SBQtADl5RW16MAzcBKvdt5L8Ug8gCYecp7jCy6umpwrU3vRnBuk_Co4CoYVqZgDGkIi378RrgCpPqM$)', 'transcript': '[https://www.ncbi.nlm.nih.gov/nuccore/NM_000546.6](https://urldefense.com/v3/__https://www.ncbi.nlm.nih.gov/nuccore/NM_000546.6__;!!DV4KuIgKKrh48VMFxQ!F-zIvz38DkmGCWD46UZn9SBQtADl5RW16MAzcBKvdt5L8Ug8gCYecp7jCy6umpwrU3vRnBuk_Co4CoYVqZgDGkIi378RnedYMyY$)'}, 'refseqgene_context_intronic_sequence': '', 'selected_assembly': 'hg38', 'submitted_variant': 'NM_000546.6:c.239C>G', 'transcript_description': 'Homo sapiens tumor protein p53 (TP53), transcript variant 1, mRNA', 'validation_warnings': [], 'variant_exonic_positions': {'NC_000017.10': {'end_exon': '4', 'start_exon': '4'}, 'NC_000017.11': {'end_exon': '4', 'start_exon': '4'}}}, 'flag': 'gene_variant', 'metadata': {'variantvalidator_hgvs_version': '2.0.1', 'variantvalidator_version': '2.1.1.dev34+gc25d2c9', 'vvdb_version': 'vvdb_2022_04', 'vvseqrepo_db': 'VV_SR_2022_02/master', 'vvta_version': 'vvta_2022_02'}}

You can see that the variant MD submitted was NM_001291303.3:c.569C>T and it received an answer for NM_000546.6:c.239C>G

Then I believe @ifokkema you can stop your script which worsens the situation as it is clear that it is a race condition.

[Peter-J-Freeman]

Nice work. I will speak to IT for advice.

[vidboda]

however I agree that what I described is slightly different from what @ifokkema reported (and that I see myself from time to time), i.e. here the VV report is consistent (both submitted variant and returned variants fields are the same (NM_000546.6:c.239C>G), but this result has been sent to the wrong client (myself) while it was queried by Ivo. But for sure the issues are related.

[vidboda]

I might be related to the MPM config of apache?

[Peter-J-Freeman]

It is likely beyond me alone, so I think I will need to look round for folks who might have an idea how to trace what causes it. Will keep you all posted

[ifokkema]

Does this help? https://stackoverflow.com/questions/23685630/python-wsgi-on-a-multi-threaded-web-server-is-this-a-race-condition

(also check links provided on that page, like this resource)

[Peter-J-Freeman]

Thanks. I will take a look tomorrow

[ifokkema]

@ifokkema is your script still running?

Nah, we had a power test, and I had to turn off my PC. Didn't turn it back on after that, as I got so many errors that I realized I was just making things worse :grimacing:

Oh, access files. My mistake (...)

You probably want to delete that again since the IP address is perhaps considered private information, and also the requests themselves may contain sensitive information. But it's good, we confirmed the issue!

the 2nd IP is MobiDetails - I've received an error email when the user submitted this variant: (...) however I agree that what I described is slightly different from what @ifokkema reported

Actually, I think that can make sense. If variables are shared, what happens depends on what variable is overwritten.

1. The output object at the last moment, or the input right at the beginning? You'll get somebody else's output.
2. Something crucial along the way? Internal Server Error or strange errors showing your input.
3. Multiple variables? Strange errors that show somebody else's input.

I've had only options 2 (both) and 3, but your earlier mentions of option 1 make just as much sense.

Hopefully, this can be fixed easily by changing the mod-wsgi settings; this may slow down VV, but this race condition seems like a bigger problem to me. After that, by editing the code to make it thread-safe, the threading can be re-enabled. Perhaps by temporarily adding sleep() calls in your code in a test environment, you can reproduce this quite easily as you'll be able to lengthen the duration in which multiple calls can share data.

[vidboda]

Nah, we had a power test, and I had to turn off my PC. Didn't turn it back on after that, as I got so many errors that I realized I was just making things worse 😬

Then who the hell is running all those NM_000546.6 queries?

[Peter-J-Freeman]

This is why we are creating a new API server :) We want to know who is on there and start to track usage. Think its more popular than we had realised.

OK, this looks interesting

https://stackoverflow.com/questions/71139245/python-threading-in-flask-webapp and this https://www.pythontutorial.net/python-concurrency/python-threading-lock/

Hope I can figire out which global to protect and how to do it easily

[ifokkema]

Then who the hell is running all those NM_000546.6 queries?

Well, judging by their IP, somebody in Cambridge 😂 I saw the same transcript a lot the 11th. So it's also somebody who's just repeating the calls over and over? Perhaps there should be some terms and conditions. I tend to block this kind of API usage myself, there don't seem to be any waits built in, and they don't identify themselves using the User Agent. My API replies quicker because it just looks up data, so sometimes I get hundreds of requests a second from the same IP. That's just plain offensive.

[Peter-J-Freeman]

This is why I need a software engineer who can deal with the server side stuff. Really struggle to have the time to do this complex stuff. Always seems to come up when teaching is about to kick off. Gah!

[ifokkema]

This is why we are creating a new API server :) We want to know who is on there and start to track usage. Think its more popular than we had realised.

Yeah, it also gives you the chance to explain them how to behave :laughing:

Hope I can figire out which global to protect and how to do it easily

I hope so! You may need to protect a lot of variables, but I don't know enough about the structure of the code to help, sorry :disappointed:

[vidboda]

while this is not directly related, I once solved some postgreSQL kind of race conditions using connection pools instead of direct connections https://pynative.com/psycopg2-python-postgresql-connection-pooling/ As you also have a pg DB somewhere in VV, you might want to have a look - only if you manage your connections yourself, and not through e.g. SQLAlchemy

[vidboda]

The thing is: is it coming from the Flask app or Apache? That we don't know - we expect from a flask call to a specific endpoint that the returned json goes to the right 'client' (apache process) - not sure protecting variables will change anything. I would more easily understand that a wrong apache config (not well fitted with the server hardware) could mix processes, but this is just my guess. How could we imagine a simple test?

[vidboda]

print on error_log the output json and calling IP of the client then match it with queries on access_log? If everythings fit => the issue is on the apache side (Flask works well) else => Flask is guilty

[Peter-J-Freeman]

I think you might be on the right track @beboche but I think it might well be a SQLite3 issue. Look again at the error log I posted. It is all linked to SeqRepo and the seqrepo database. So I think we might need to manage connections where this data is being accessed.

We actually interface with it through vv_hgvs fork though https://github.com/openvar/vv_hgvs/blob/master/vvhgvs/dataproviders/seqfetcher.py#L32

The first thing I think I should check is the config on the server https://github.com/openvar/vv_hgvs/blob/master/vvhgvs/dataproviders/seqfetcher.py#L38 and check whether check_same_thread is True or False. I need to remember what it should be set to by visiting VVweb

My guess is that if we need to apply a thread lock, this is where it will go

[vidboda]

right could be as well - btw what is the purpose of using SQLite? Is this for read-only data?

[Peter-J-Freeman]

Ask me in the meeting lol ;P

[John-F-Wagstaff]

@Peter-J-Freeman I have never seen errors like this before. I assume you are looking at errors from within seqrepo's query(by name)->query id lookup->bgziped fasta file location->within file bgzip block location->bgzip sequence fetch(and decompress), chain here? (where the query name to within seqrepo bgziped fasta file location steps are SQLite dependant).

Unfortunately I can not find a complete list out of the meanings of the error return types for bgzip but it seems like serepo uses bgzf_uncompress from pysam which then uses libdeflate_deflate_decompress from libdeflate so this should apply

enum libdeflate_result {
    /* Decompression was successful.  */
    LIBDEFLATE_SUCCESS = 0,

    /* Decompressed failed because the compressed data was invalid, corrupt,
     * or otherwise unsupported.  */
    LIBDEFLATE_BAD_DATA = 1,

    /* A NULL 'actual_out_nbytes_ret' was provided, but the data would have
     * decompressed to fewer than 'out_nbytes_avail' bytes.  */
    LIBDEFLATE_SHORT_OUTPUT = 2,

    /* The data would have decompressed to more than 'out_nbytes_avail'
     * bytes.  */
    LIBDEFLATE_INSUFFICIENT_SPACE = 3,

So bgzip is getting bad data (which could be caused by incorrect/invalid file locations from SQLite errors) or insufficient space(which would suggest a differnt error cause)?

On the other hand "CRC32 checksum mismatch" means a failure at the last step where the block it is trying to decompress is not matching it's own checksum. With this in mind unless the ID output is mangled such that it is pointing at a real file that is valid enough to look like bgzip block, but is incorrect/not a bgzip block, then this error log looks like at least partially the result of damaged seqrepo bgzip files?

SQLite has a bit of a bad reputation for concurrency issues but it usually just fails to give a result due to file lock if it is going to go wrong. Worse we are triggering multiple different error types here, it all looks less clear the more I look. if we still think it might be the SQLite then some kind of extra longing in your local copy of FabgzReader (in seqrepo/fastadir/fabgz.py) which is where the ouput of the SQLite query gets piped into the bgzip part of the code might give a bit more info if you have the time and can reliably reproduce.

EDIT : we would want the filename from init and possibly the seq_id from fetch I think

[vidboda]

This is why we are creating a new API server :) We want to know who is on there and start to track usage. Think its more popular than we had realised.

Yeah, it also gives you the chance to explain them how to behave 😆

Maybe you should block that cambridge IP - the guy would shortly contact you - not fair, but efficient

[ifokkema]

This is why we are creating a new API server :) We want to know who is on there and start to track usage. Think its more popular than we had realised.

Yeah, it also gives you the chance to explain them how to behave laughing

Maybe you should block that cambridge IP - the guy would shortly contact you - not fair, but efficient

To clarify the reason of blocking them, in reality I'm actually sending them a 403 (redirect) rather than a block, but effectively it breaks their script. The page they then get shows:

Your IP address has been blacklisted after abuse of our terms of service

You have been forwarded here because your IP address is blacklisted following abuse of our terms of service originating from this IP address. If you believe you are receiving this page in error, please notify us and mention your IP address: and include the URL you were trying to visit. The terms of service and fair use policy of our APIs are included in the LOVD 3.0 manual, chapter "The LOVD API".

[vidboda]

This is why we are creating a new API server :) We want to know who is on there and start to track usage. Think its more popular than we had realised.

To clarify the reason of blocking them, in reality I'm actually sending them a 403 (redirect) rather than a block, but effectively it breaks their script. The page they then get shows:

Your IP address has been blacklisted after abuse of our terms of service

You have been forwarded here because your IP address is blacklisted following abuse of our terms of service originating from this IP address. If you believe you are receiving this page in error, please notify us and mention your IP address: and include the URL you were trying to visit. The terms of service and fair use policy of our APIs are included in the LOVD 3.0 manual, chapter "The LOVD API".

Clever. I use API tokens. I think users are usually more cautious if they know they can be tracked down. But most of them make a fair usage of the API.

[Peter-J-Freeman]

Hi guys.

@John-F-Wagstaff. I think that what your are telling me agrees with my thoughts. This looks like a set of SeqRepo sQLite 3 errors that we cannot figure out why they ought to be happening. Race conditions is a good theory.

I spotted this https://github.com/openvar/variantValidator/blob/update_to_vvta/VariantValidator/modules/vvMixinInit.py#L127

self.sf = vvhgvs.dataproviders.seqfetcher.SeqFetcher()  # Seqfetcher

We want to update this to pass to vv_hgvs check_same_thread=False

https://github.com/openvar/vv_hgvs/blob/master/vvhgvs/dataproviders/seqfetcher.py#L19

This is my first job. Hang fire all, let me go through this methodically

[Peter-J-Freeman]

OK, these commits have added the check_same_thread to the VV methodology. I will now check whats happening on the server and update

[Peter-J-Freeman]

OK @ifokkema and @beboche , I have made the changes. Can you please hammer the server and see if the race conditions persist?

[ifokkema]

I'm getting a lot of Internal Server Errors... You'd better check your apache logs; likely a lot of HTTP 500s in there now.

[Peter-J-Freeman]

Bet you wish you hadn't mosned about race conditions now! lol. on it!

[ifokkema]

Yeah, you need to revert... everything is down now.

[Peter-J-Freeman]

OK I'm getting

[Thu Sep 15 15:08:22.622190 2022] [wsgi:error] [pid 23654:tid 140356410799872] [remote 192.168.17.243:49362] CRITICAL: <class 'vvhgvs.exceptions.HGVSDataNotAvailableError'> Failed to fetch NM_000088.3 from SeqRepo (/local/seqrepo/VV_SR_2022_02/master) (SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 140356419192576 and this is thread id 140356410799872)

I think I miss-set the check_same_thread. Think it needs to be False and not True

[Peter-J-Freeman]

Yeah, you need to revert... everything is down now.

Nah, think I got it. One sec. Although this will not have solved the race issue but has mad thee requirement for threading selectable in VV

[Peter-J-Freeman]

OK, should be good ow, here is the code change

https://github.com/openvar/variantValidator/commit/7952d11df9c153974ac9d542d57fd06ffcbb90f6

[Peter-J-Freeman]

So, now that I understand the issue a little better now, I think we need to protect the following line with thread locks

https://github.com/openvar/vv_hgvs/blob/master/vvhgvs/dataproviders/seqfetcher.py#L39

This is where VV connects to the SQLite database. I'll see if I can figure it out and whether it will work

[Peter-J-Freeman]

This looks potentially useful and I will see if I can adapt it

https://stackoverflow.com/questions/41206800/how-should-i-handle-multiple-threads-accessing-a-sqlite-database-in-python

import threading
import sqlite3
dburi = "file:TESTING_MEMORY_DB?mode=memory&cache=shared"
connection = sqlite3.connect(dburi, uri=True, check_same_thread=False)
# NOTE: You'll need to share this same lock object with every other thread using the database
lock = threading.Lock()
with lock:
    with connection:
        cursor = connection.cursor()
        cursor.execute("SQL")
        connection.commit()

[ifokkema]

No change; I'm still getting internal server errors for most requests.