adamnovak
commented
2 years ago It definitely does look like your leader node does not have enough disk.
What do you mean by "I always see the /tmp is full"? Surely it's empty when you freshly launch the cluster, right? And then it fills up when you run your workflow?
The first error looks like what might happen if your disk fills up right at the end of a workflow, when it is uploading output files to your destination bucket, from the leader.
The second error looks like what might happen if your root filesystem is full and you try to launch a workflow: it tries to create a working directory for jobs that run on the leader, and it can't, because the disk is already full. When you got that, was the disk not full before you launched the workflow (and thus something earlier in the workflow startup process is filling it up)? Because I think input file import happens before that step, so it could be that importing the input files for the workflow is filling up your disk.
You can try using the TMPDIR environment variable to move where Toil keeps its temporary files, but that might not help if the leader node doesn't actually have a big enough disk for them to go on.
To make there be more space on the leader node, you can pass toil launch-cluster the --leaderStorage option, to tell it to assign the leader a larger-than-default EBS disk. You could also pick a leader node type that comes with its own local storage. For example, if you use an m5d.xlarge, the cluster will be set up with Toil's working directories on the 150G SSD that those instances come with. I'm don't think that /tmp will be there too, if /tmp is specifically filling up, but you could use TMPDIR to move it over to one of the directories (like /var/lib/toil) that is.
The real question is, why is the disk filling up on the leader? I think that what is happening is that our CWLTool-based file manipulations are causing local copies of the inputs and outputs to be created, when one might not expect to need the space for them. That's the good possible problem, because it should be easier to fix, and the workaround is to just make sure that your leader has room on disk for all the workflow inputs and outputs.
The bad possible problem is that the CWL intermediate jobs that execute on the leader might be getting file and directory objects staged to disk on the leader. @boyangzhao do you only see the errors you posted, or do workflows also sometimes run out of space in the middle?
To import the inputs, especially if there are Directory objects, we make a bunch of calls into CWLTool's file management functions, and they make a bunch of calls back into ToilFsAccess, and ToilFsAccess still falls back to making copies of things on disk to fulfill some requests. It also doesn't clean up after itself, I think partly because we initially designed it to work only in jobs, where cleanup would be handled automatically at the end of the job, and partly because the CWLTool StdFsAccess API doesn't have a hook for being notified when file paths you handed out are no longer needed. (This could also be why /tmp is filling up and not --workDir, because the logic to put temporary files in --workDir belongs to jobs and won't work on a ToilFsAccess in the leader itself.)
On the export side, I don't think we intend for files to be downloaded to disk before being uploaded to the destination bucket, but it might be happening anyway?
@mr-c I think we can address some of this by making ToilFsAccess a little smarter about not using _abs, and maybe getting all the ToilPathMappers and all the ToilFsAccesses to share a global cache with some kind of cleanup. But I think we might also need some help from the CWLTool side, because the APIs that StdFsAccess and PathMapper implement, and the guarantees they require, are not really fully documented as far as I can tell.
When can the path returned by StdFsAccess.realpath() safely be deleted by the StdFsAccess implementation? Does it ever need to outlive the StdFsAccess` object that it was retrieved from?
boyangzhao
unito-bot
mr-c
Hi it seems I'm keep running into space issues when trying to run jobs on AWS clusters using toil. I don't know how much is attributed to configurations on instance selection, and how much can I change with toil arguments.
I always see the /tmp is full (2GB and it's always at 100% usage), but I cannot change the size with umount/mount; I'm using the default ami image. I've tried using the
--workDirargument, but it seems this folder was always empty during the runs, and /tmp always full. And because /tmp becomes full, I can then no longer able to ssh to the node (toil ssh-cluster) as the error there would say no space left on /tmp.In one example, the job would finish, but has the last error (unclear if this is because /tmp is full). I use destBucket so I do see the files being written to the s3 buckets, but why the "no space left error" on the presumably leader node?
Other times, I'd get the following space error,
┆Issue is synchronized with this Jira Story ┆Issue Number: TOIL-1189