Are insertions broken?

[corneliusroemer]

I can't see any insertions, but there should be insertions in JN.1

https://cov-spectrum.org/explore/World/AllSamples/Past6M/variants?nextcladePangoLineage=JN.1*&

[Taepper]

~It seems to be only for this lineage in particular?~ Ah, other pages were still cached for me.

[corneliusroemer]

Alex and I investigated and it looks like all data ingested (submitted) since 2023-01-21 does not have ingestions. There's a sharp cutoff between 2023-01-20 and 2023-01-21.

That makes it very likely that it's ingest related?

See: https://lapis.cov-spectrum.org/gisaid/v2/sample/nucleotideInsertions?dateSubmittedFrom=2023-01-21&accessKey=9Cb3CqmrFnVjO3XCxQLO6gUnKPd -> no insertions since 2023-01-21

But before: https://lapis.cov-spectrum.org/gisaid/v2/sample/nucleotideInsertions?dateSubmittedFrom=2023-01-20&accessKey=9Cb3CqmrFnVjO3XCxQLO6gUnKPd

[chaoran-chen]

Hmm. Did you check the data that SILO gets? Are the insertions missing there?

[chaoran-chen]

I am now extracting the nucleotide insertions from the file that we give to SILO via

zstdcat provision.1716905734.ndjson.zst | \
  jq -c '{gisaidEpiIsl: .metadata.gisaidEpiIsl, nucleotideInsertions: .nucleotideInsertions.main, dateSubmitted: .metadata.dateSubmitted}' | \
  zstd > nucleotideInsertions.ndjson.zstd

This will take a while. I'll let you know tomorrow.

[chaoran-chen]

@Taepper, I used zstdcat nucleotideInsertions.ndjson.zstd | jq -c 'select((.nucleotideInsertions | length > 0) and .dateSubmitted > "2023-02-23")' | less to see the sequences with insertions after 2023-02-23. I found 253678 entries with insertions.

[corneliusroemer]

Interestingly, open does show some insertions as well, even for recently submitted sequences, but not for recently colllected ones https://open.cov-spectrum.org/explore/World/AllSamples/AllTimes/variants?&

https://open.cov-spectrum.org/explore/World/AllSamples/Past6M/variants?&