search

va-big-data-genomics / trellis-mvp-functions

Trellis serverless data management framework for variant calling of VA MVP whole-genome sequencing data.
6 stars 1 forks source link

Add flag to db-query to always reload queries in dev env #31

Open pbilling opened 2 years ago

pbilling commented 2 years ago

Right now the list of predefined database queries is stored in cloud storage and loaded into global variables at cold start of a function instance. Subsequent function calls reuse the same global variables, saving time when executing the function. Global variables can persist on an instance for about 10 minutes after the last function invocation. More info: https://cloud.google.com/functions/docs/bestpractices/tips#functions-tips-scopes-python.

This can be confusing if a developer is changing the database query object and then running live tests, as the new queries will not be ingested until the current instance shuts down and a new one starts. My current solution to this issue is to delete and rebuild the db-query function every time I upload new queries. A less cumbersome solution might be to add a flag so that the queries are always reloaded on function invocation, when in a development environment.

pbilling commented 1 year ago

Initial implementation in 84ab21700ccf91c902098146afa6109b29395bb4 and eb51e96253b465736e5c79977744ed475cd31603