search

Clinical-Genomics / loqusdbapi

A simple REST api for loqusdb
2 stars 0 forks source link

Missing SV frequencies? #28

Closed dnil closed 2 years ago

dnil commented 2 years ago

I just had a chat with two users who had noticed an abundance of seemingly rare SVs on their cases, or with other words a paucity of variants reporting as local. Is loqusdb/loqusdbapi misbehaving or have filter settings changed in MIP? On a quick look, it does seem we have some rather common variants "missing" from loqusdb, e.g.

Screenshot 2022-03-01 at 08 54 30

dnil commented 2 years ago

Yes, I think I can confirm this. It appears to be a pydantic problem with loqusdbapi, presumably some dependency changing a type. I'll move this over.

$ curl -X GET "http://cg-prod-services.scilifelab.se:7079/svs/?chrom=X&end_chrom=X&pos=65549568&end=88119178&sv_type=DEL"
Internal Server Error

Log:

Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:   File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 376, in run_asgi
Mar 01 09:13:50 cg-prod-services.scilifelab.se podman[1648665]:   str type expected (type=type_error.str)
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:     result = await app(self.scope, self.receive, self.send)
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:   File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:     return await self.app(scope, receive, send)
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:   File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 179, in __call__
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:     await super().__call__(scope, receive, send)
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:   File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 111, in __call__
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:     await self.middleware_stack(scope, receive, send)
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:     raise exc from None
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:     await self.app(scope, receive, _send)
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:   File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:     raise exc from None
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:   File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:     await self.app(scope, receive, sender)
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 566, in __call__
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:     await route.handle(scope, receive, send)
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 227, in handle
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:     await self.app(scope, receive, send)
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 41, in app
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:     response = await func(request)
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 190, in app
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:     response_data = await serialize_response(
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 111, in serialize_response
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:     raise ValidationError(errors, field.type_)
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]: pydantic.error_wrappers.ValidationError: 1 validation error for StructuralVariant
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]: response -> _id
Mar 01 09:13:50 cg-prod-services.scilifelab.se conmon[1648715]:   str type expected (type=type_error.str)
dnil commented 2 years ago

Reproduces on stage:

$ curl -X GET "http://cg-vm1.scilifelab.se:7079/svs/?chrom=X&end_chrom=X&pos=65549568&end=88119178&sv_type=DEL"
Internal Server Error
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]: 10.0.2.100:38494 - "GET /svs/?chrom=X&end_chrom=X&pos=65549568&end=88119178&sv_type=DEL HTTP/1.1" 500
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]: 10.0.2.100:38494 - "GET /svs/?chrom=X&end_chrom=X&pos=65549568&end=88119178&sv_type=DEL HTTP/1.1" 500
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]: [2022-03-01 08:27:41 +0000] [4] [ERROR] Exception in ASGI application
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]: Traceback (most recent call last):
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:   File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 376, in run_asgi
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:     result = await app(self.scope, self.receive, self.send)
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]: [2022-03-01 08:27:41 +0000] [4] [ERROR] Exception in ASGI application
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]: Traceback (most recent call last):
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:   File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 376, in run_asgi
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:     result = await app(self.scope, self.receive, self.send)
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:   File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:     return await self.app(scope, receive, send)
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:   File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 179, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:     await super().__call__(scope, receive, send)
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:   File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 111, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:     await self.middleware_stack(scope, receive, send)
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:     raise exc from None
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:     await self.app(scope, receive, _send)
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:   File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:     raise exc from None
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:   File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:     await self.app(scope, receive, sender)
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 566, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:     await route.handle(scope, receive, send)
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 227, in handle
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:     await self.app(scope, receive, send)
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 41, in app
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:     response = await func(request)
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 190, in app
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:     response_data = await serialize_response(
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 111, in serialize_response
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:     raise ValidationError(errors, field.type_)
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]: pydantic.error_wrappers.ValidationError: 1 validation error for StructuralVariant
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]: response -> _id
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:   File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se podman[340135]:   str type expected (type=type_error.str)
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:     return await self.app(scope, receive, send)
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:   File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 179, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:     await super().__call__(scope, receive, send)
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:   File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 111, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:     await self.middleware_stack(scope, receive, send)
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:     raise exc from None
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:     await self.app(scope, receive, _send)
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:   File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:     raise exc from None
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:   File "/usr/local/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:     await self.app(scope, receive, sender)
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 566, in __call__
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:     await route.handle(scope, receive, send)
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 227, in handle
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:     await self.app(scope, receive, send)
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 41, in app
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:     response = await func(request)
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 190, in app
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:     response_data = await serialize_response(
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 111, in serialize_response
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:     raise ValidationError(errors, field.type_)
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]: pydantic.error_wrappers.ValidationError: 1 validation error for StructuralVariant
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]: response -> _id
Mar 01 09:27:41 cg-vm1.scilifelab.se conmon[340192]:   str type expected (type=type_error.str)

But if we switch back to an old container:

[hiseq.clinical@cg-vm1 ~]$ systemctl --user start loqusdbapi-rd@vm-ready-package
[hiseq.clinical@cg-vm1 ~]$  curl -X GET "http://cg-vm1.scilifelab.se:7079/svs/?chrom=X&end_chrom=X&pos=65549568&end=88119178&sv_type=DEL"
{"chrom":"X","observations":1228,"families":["choiceracer","tightdoe","briefglider","primeasp","livingpup","topghost","tidygull","caringhound","neatkodiak","fluentsnake","trueroughy","movingbengal","superbhyena","growncrane","novelfilly","awareliger","frankswan","greatosprey","activehyena","mintlocust","suitedakita","subtlestag","poeticraptor","validmoose","intentboar","normallion","vocalrodent","gamegoblin","crackkodiak","moralmidge","actualfeline","squarequail","betterturtle","loyaloryx","onechimp","robustsnipe","happyrobin","busyloon","viableparrot","socialbream","truedeer","frankloon","neatmole","honestparrot","livingmarmot","freeelk","gladurchin","caringskunk","pumpedquail"],"total":5323,"end_chrom":"X","end_left":88116180,"end_right":88122180,"sv_type":"DEL","length":22569593,"pos_left":65546587,"pos_right":65552587}[hiseq.clinical@cg-vm1 ~]$
dnil commented 2 years ago

This appears to be the last working (stage) container:

$ systemctl --user start loqusdbapi-rd@update_mongo_adapter_dep
[hiseq.clinical@cg-vm1 ~]$  curl -X GET "http://cg-vm1.scilifelab.se:7079/svs/?chrom=X&end_chrom=X&pos=65549568&end=88119178&sv_type=DEL"
{"chrom":"X","observations":1228,"families":["choiceracer","tightdoe","briefglider","primeasp","livingpup","topghost","tidygull","caringhound","neatkodiak","fluentsnake","trueroughy","movingbengal","superbhyena","growncrane","novelfilly","awareliger","frankswan","greatosprey","activehyena","mintlocust","suitedakita","subtlestag","poeticraptor","validmoose","intentboar","normallion","vocalrodent","gamegoblin","crackkodiak","moralmidge","actualfeline","squarequail","betterturtle","loyaloryx","onechimp","robustsnipe","happyrobin","busyloon","viableparrot","socialbream","truedeer","frankloon","neatmole","honestparrot","livingmarmot","freeelk","gladurchin","caringskunk","pumpedquail"],"total":5323,"end_chrom":"X","end_left":88116180,"end_right":88122180,"sv_type":"DEL","length":22569593,"pos_left":65546587,"pos_right":65552587}[hiseq.clinical@cg-vm1 ~]$
dnil commented 2 years ago

Ah, its the new-years release extend-api. SNV and SV vars have different _id type, with SNVs having a unique str, and SVs where this is not quite possible due to overlaps etc have an ordinary mongodb BSON ObjectId for _id. That PR introduced a required Pydantic type of str for both, failing every SV call downstream.

We need both better tests and better error handling in Scout for this..