search

seqeralabs / wave

On-demand containers provisioning service
https://seqera.io/wave/
GNU Affero General Public License v3.0
35 stars 4 forks source link

Timeout on /containers when creating a container with cellpose #597

Closed odagayev closed 3 months ago

odagayev commented 3 months ago

Here is the build log: https://wave.seqera.io/view/builds/dded52ad814da816_1

I'm trying to build something with this package: https://github.com/MouseLand/cellpose

It appears that this container times out. I have a feeling that this takes a long time because of the CUDA dependencies that this package has. I'd imagine that many ML focused packages right now are running into a similar issue.

I would love to be able to address this issue for the ML community.

pditommaso commented 3 months ago

Currently all build have an hard limits to 15 minutes. It could looked into how to have different build profiles having a longer expiration time.

odagayev commented 3 months ago

What are the implications of doubling this build time? Is the 15 arbitrary or is there a reason for the hard limit?

pditommaso commented 3 months ago

One side the longer the build the easier for a malicious user to saturate our build cluster.

Also a too long build would may cause the invoking client (e.g. a nextflow pipeline job to hold for all build time doing anything and burning money)

ewels commented 3 months ago

How about having extended build times only for authenticated users? That might mitigate the risk of malicious users a little.

Another option is to make it an opt-in flag, so that longer build times are only used if specifically requested. This is what GitHub Actions does for example.

ewels commented 3 months ago

Suggestion from @pditommaso in weekly meeting: Quick patch to double timeout to 30 mins for when user is authed + freeze is set. This will apply to all builds in Seqera Containers @odagayev.

Can investigate more complex solutions at a later date if we continue to see problems.

odagayev commented 3 months ago

I'm afraid this won't work. This is for seqera.io/containers which is mostly unauthed users. Let me know if this is possible.

pditommaso commented 3 months ago

Yes, we are using an account for it

odagayev commented 3 months ago

Thank you guys. 🙏

CC: @FloWuenne

pditommaso commented 3 months ago

Solved by https://github.com/seqeralabs/wave/pull/600

pditommaso commented 3 months ago

This has been released. Build submitted by auth users and having freeze enable will have 25 minutes (instead of 15 min) build time.