Closed Puumanamana closed 1 year ago
pditommaso
commented
1 year ago The disk size is indeed defined in the code.
@jordeu you were mentioning about some constraints regarding this, however I think it should be parametrised somehow.
@Puumanamana. I believe your task is failing because the task uses the bootdisk as local temporary work directory.
Try adding process.scratch=false in your nextflow.config file
jordeu
commented
1 year ago Fusion is using local SSD that is different from the network storage boot disk. You cannot choose the size of local SSD (all of them are 375GB), only how many do you want. https://cloud.google.com/compute/docs/disks/local-ssd
The intended use of Fusion is to always set only one local SSD disk that is used as cache and then run Nextflow with process.scratch=false in this way you do not need to worry about each process local disk size, the process is run on top of Fusion with "infinite" capacity.
The default GCP quota is 6TB, so this will allow 16 concurrent processes by default. But I agree that like all other GCP quotas the defaults are really low.
Maybe we can make it optional if Fusion uses local SSD as cache or the default network boot disk. The problem of using the network boot disk is that then we are doubling the network usage and the performance of Fusion should decrease a bit but it may be still good enough for many use cases, we should do some testing before.
Puumanamana
commented
1 year ago @pditommaso, when trying to do a reproducible test, I forgot to set process.scratch = false indeed. But my original issue came from trying to run many concurrent processes in parallel with fusion and getting the following error (actually warning):
Mar-20 00:22:45.773 [Task monitor] WARN n.c.g.batch.GoogleBatchTaskHandler - Batch job cannot be run: VM in Managed Instance Group meets error: Batch Error: code - CODE_GCE_QUOTA_EXCEEDED, description - error count is 2, latest message example: Instance 'nf-8908fe23-167927-e3b48cca-de94-46750-group0-0-pn66' creation failed: Quota 'LOCAL_SSD_TOTAL_GB' exceeded. Limit: 30000.0 in region us-central1..I wanted to reduce the disk size requirement for my processes to be able to run more but I see now it's not possible. 16 is very restrictive (probably a bit more in my case but I'd like to run thousands in parallel). So maybe the fusion is not the right thing for my use case. @jordeu, I'm not familiar with network boot disks, would it still be considered as a fusion filesystem (all processes sharing the same disk space? Alternatively, is there anyway to mount multiple SSDs to increase the limit?
jordeu
commented
1 year ago Yes, Fusion can also work using a network disk as temporal cache (we've tested this at AWS with good results). I'll also test this setup on GCP and then we can evaluate to make the usage of local SSD optional.
jordeu
commented
1 year ago I was thinking how to make this configurable, a nice way would be to allow to define if you want to use local SSD disk or not at the level of process.
What do you think in allowing to define the disk like this:
process using_ssd {
disk 'local-ssd: 1'
...
}
process using_network_disk {
disk '10 GB'
...
}I need to do some benchmarks but most likely the results with network disk are good enough, so in this way fusion will be able to use SSD disk only on some processes while in other can use normal network disk.
The nice of this solution is that also makes sense without Fusion. So if you set disk 'local-ssd: 2' Nextflow will mount two local SSD disks as /tmp in the container and it will be use as scratch (like is doing with network disks).
The recommended way to run Fusion will be with process.disk = "local-ssd: 1" but any combination of them will also make sense.
I like that in this way we decouple the definition of what kind and size of disk we use as /tmp from the Fusion usage.
Puumanamana
commented
1 year ago That makes sense. Just a clarification though: at the moment with using google-batch, the boot disk (set with google.batch.bootDiskSize) is a SSD disk right? And still with google-batch, when setting disk on top of that, it just makes the boot disk larger. So with what you propose, the user would have more flexibility with type of disk to use, which seems great to me.
I'm also a bit confused by this whole process.scratch = true/false and how this plays into this new layout (does it have to always be false, and if not, what happens)
Really my main issue at the moment is to run hundreds of concurrent jobs, and whether I use fusion or not, I'm limited by the SSD quota (but I'm more limited with fusion).
jordeu
commented
1 year ago Just a clarification though: at the moment with using google-batch, the boot disk (set with
google.batch.bootDiskSize) is a SSD disk right?
Yes, but it's a network attached SSD disk. So the disk is an SSD but in another machine. While the "local-ssd" is an SSD in the same machine where your instance is running.
I'm also a bit confused by this whole
process.scratch = true/falseand how this plays into this new layout (does it have to always be false, and if not, what happens)
By default process.scratch is set to true and that means that Nextflow is going to use Google Storage utils to download/upload input/output files to/from a temporal folder in the container.
You can only set process.scratch = false when the working directory is a shared filesystem (and there is no need to download/upload files). This is not the case when you use an object storage like GS. What Fusion does is to provide this POSIX shared filesystem directly on top of object storage. So currently, there is no other way to setup a shared filesystem with Google Batch and Nextflow (it will be possible when #3630 is added).
Puumanamana
commented
1 year ago Thank you, good to know!
Bug report
When using fusion filesystems on google-batch, setting
diskdoesn't set the size of the SSD for the fusion and uses instead ~350GB SSD. Aside from cost, this is a problem for me since my GCP project have set quotas (I can't use more than some amount of SSD storage) which greatly limits the number of task I can run in parallel. My pipelines end up failing if too many tasks are submitted at the same time.Expected behavior and actual behavior
Actual: There's no way of setting the size of the SSD Expected: Either setting
diskwith fusion filesystem enabled sets the size of the SSD, or another directive is available to set the size of the SSD.Steps to reproduce the problem
The following script fails as expected when fusion is not enabled but not with fusion filesystem enabled (
-profile fusion)main.nf
nextflow.config
Program output
Without fusion, this returns I/O error for the 30GB input as expected (side question: is it expected here to have a null exit status?)
Output:
With fusion, no error:
In addition, here's the stdout for the fusion run:
We can see there's a NVMe disk attached with ~350 GB SSD
.nextflow.log (with fusion)
Environment
$SHELL --version): zsh 5.8 (x86_64-ubuntu-linux-gnu)