search

populationgenomics / variant-curation-portal

Web application for curating loss of function variants
MIT License
1 stars 0 forks source link

Request timeout when exporting results from large project #71

Closed EddieLF closed 4 months ago

EddieLF commented 4 months ago

There is a project with several thousand variant results, which will not export as either csv or json. The cloud run logs show that the request timeout (300s) is being reached so the request is failing - Log entry.

requestUrl: "https://curator.populationgenomics.org.au/api/project/7/results/export/"
...
severity: "ERROR"
textPayload: "The request has been terminated because it has reached the maximum request timeout. To change this limit, see https://cloud.google.com/run/docs/configuring/request-timeout"

The documentation link says this can be configured on the command line with gcloud run deploy --image IMAGE_URL --timeout=[TIMEOUT]. To resolve this, we could optimise the query used to gather the results, or increase the timeout from the default 300s.

EddieLF commented 4 months ago

@nevoodoo tagging you in for your django expertise 🙏

The project results API call is defined here - https://github.com/populationgenomics/variant-curation-portal/blob/main/curation_portal/views/project_results_export.py

No doubt it could be optimised, somehow...? The simpler band-aid solution would be to increase the timeout. @SamBryen needs these results exported by June 21. I'm not sure if optimising it will be worth our time, unless there is a simple solution you can see. Noting that this endpoint has remained unchanged upstream for the past few years.