search

krishnanlab / geneplexus-app-v2

GenePlexus App - backend (cloud functions) and frontend
https://gene-plexus.netlify.app/
2 stars 0 forks source link

Stress test functions #2

Closed ChristopherMancuso closed 3 weeks ago

ChristopherMancuso commented 7 months ago

Attempt to see how robust the google cloud function can be. Some things to test are

ChristopherMancuso commented 6 months ago

@falquaddoomi did you get a chance to test out how much data can be attached to a gcp function. If all the data is read in at the start using a RAM disk, I'm not sure these functions are actually going to work. Basically for the machine learning functions, there is no way to break them down where they wouldn't have a lot of data in each function. Well, there is a way to do it but that would mean deploying more than 100 functions. If you haven't tested this out yet I can try it this week.

falquaddoomi commented 6 months ago

@falquaddoomi did you get a chance to test out how much data can be attached to a gcp function. If all the data is read in at the start using a RAM disk, I'm not sure these functions are actually going to work.

Hey @ChristopherMancuso, the amount of disk space that you can use is limited by the amount of RAM you allocate to the function, as described here: https://cloud.google.com/functions/docs/concepts/execution-environment#memory-file-system. Cloud functions can be configured to use up to 32GB of RAM, as specified here, https://cloud.google.com/functions/docs/configuring/memory.

There are options other than bundling the data with the function. We could for example run a database on a VM to store the data then have the cloud functions query it, rather than having them load the data from disk. Alternatively, we could go back to a non-cloud-function version of the API that runs on a VM rather than a cloud function, in which case disk space isn't an issue.

ChristopherMancuso commented 6 months ago

@falquaddoomi thanks for sending that. For some reason I thought the functions only went up to 2 GB. Looking at gcp function pricing it looks like even with an 8GB function , if each function only takes a minute that is only $0.00408 per invocation for compute time. They should still work!

ChristopherMancuso commented 3 weeks ago

I think we have this solved here and if all files stay the same size what we have work and we can still go to an even bigger google function if needed.