First custom run - Githubissues

ComparativeGenomicsToolkit / Comparative-Annotation-Toolkit

Apache License 2.0

171 stars 48 forks source link

First custom run #116

Open francicco opened 6 years ago

francicco commented 6 years ago

Hi,

I'm try to run a first test with my own data. It's about 3 very closely related genomes, already aligned with Cactus. I'm preparing the configuration file following the instruction. I have a bunch of question.

There are 5 different field in the conf file (ANNOTATION, BAM, INTRONBAM, FASTA_PROTEIN and ISO_SEQ_BAM), may I fill each fill these field with all information I have for each species? Let say I have 2 gft annotation for two species, can I use both info in [ANNOTATION]?
Let's say that for one I don't have a complete annotation, nor a red bam file, but only CDS sequences. If I can, in which field may I use this info? [INTRONBAM]?
for the [PROTEIN_FASTA] field can I point each species to the same fasta file made collecting all proteins for other species?

I guess it's enough for now, I'm sure I'l have other question, can I keep asking question using this platform?

Thanks a lot for your helpfulness. Cheers F

ifiddes commented 6 years ago

I am glad we got everything working!

Answers:

The only required field in the config file is one annotation GFF3 for the reference genome. Everything else is supplementary. You can provide additional GFF3 for additional species, however these will only be used to help guide AugustusCGP (but may make quite a big difference in that case).
For the CDS sequences, are these protein or DNA? I assume DNA because of the next question. In that case, I would align them to the genome and use them as [BAM] -- the main reason [INTRONBAM] exists is to reduce false positive genes called by noisy intragenic transcription. [INTRONBAM] does not give augustus exon hints, it only gives intron hints. Since these CDS sequences are presumably actually coding, they should define proper genic exons and can be hinted as such. One thing I am not sure of here is how the BAM filtering will handle data like this. You may want to inspect the resulting hints GFF in the browser to see if it makes sense before you go ahead and run CGP.
Yes, you can point each species to the same protein FASTA. This is an intended use case.

francicco commented 6 years ago

annotation GFF3 needs to be exactly of this format of GTF are also fine? About point 2. what about put the aligned coding regions in the ISO_SEQ_BAM field?

francicco commented 6 years ago

1 failed:
- 1 Gff3ToGenePred(...)

I guess so! ;)

ifiddes commented 6 years ago

Yes, it needs to be GFF3 format. The script programs/validate_gff3.py will tell you if your GFF3 will work with CAT.

Yes, you could put the aligned coding regions in [ISO_SEQ_BAM]. That would make a lot of sense if they are full length CDS. That field will influence all modes of augustus.

francicco commented 6 years ago

even if CDS has not UTRs?

I probably have to do some scripting to get a GFF3. validate_gff3.py will help!

ifiddes commented 6 years ago

Yes, even without UTR it will be fine. The main advantage that passing in data as an [ISO_SEQ_BAM] is that the individual sequences will be grouped into what augustus terms a 'hint group'. Hint groups must all be satisfied or discarded as a group. This means that the intron-exon structure of the CDS alignment will be considered in its entirety, which is useful for full length CDS.

AugustusCGP does not try to predict UTR sequences, it only does ORFs. AugustusTMR will add UTRs if it can, but generally won't add more UTR than the original reference annotation had.

francicco commented 6 years ago

Make sense. Tomorrow I'll focus on the annotation conversion. Thanks a lot for your help. F

ifiddes commented 6 years ago

If you are converting by hand, I would suggest following the convention that CAT is looking for. Each gene should have a gene_biotype tag saying what type it is -- protein_coding is the only one that is special, it will be treated differently than the others. Every transcript should have a transcript_biotype tag (that doesn't have to match the gene level), but also have the gene level tag as well. Use the Parent field to define relationships. Take a look at the GENCODE GFF3, their style is the easiest to parse.

The six key fields CAT is looking for is gene_biotype, transcript_biotype, gene_name, gene_id, transcript_id, transcript_name. The _name fields are optional and are human readable names for ease of view in the final assembly hub.

francicco commented 6 years ago

Manually... a python script. I'll look at GRCm38.mm10.subset.gff3 that is provided in test_data, generating the field you're mentioning. Arethree_prime_UTR, five_prime_UTR and CDS mandatory?

Is this the format? https://www.gencodegenes.org/releases/current.html

ifiddes commented 6 years ago

For that, I would take a look at the GFF3 spec. These things get a bit tricky. Generally, no, three_prime_UTR and five_prime_UTR are not necessary -- they will be inferred from a combination of CDS and exon fields associated with the same transcript that do not overlap (extra exon bases are considered UTR).

You can double check your work also using the gff3ToGenePred tool, which CAT uses internally as you saw. You can then use genePredToBed to easily make a BED to load into IGV or the browser that should tell you if it is being interpreted correctly. CAT uses genePred format internally, so if they look like what you expect there you should be good to go.

However, if your current GTF are valid, then you really shouldn't need to be messing with any of this. GTF and GFF3 are nearly the same file format, except that the tag-value pairs in the last column are delimited differently. GTF uses spaces between pairs and quotes values, while GFF3 uses semicolons and equal signs without quotes. That should be the only thing you have to change, besides potentially adding new tag-value pairs to get the six mentioned above.

francicco commented 6 years ago

Yes indeed!!! F

francicco commented 6 years ago

It seems ok, right?

(luigi_env)[fc464@login-e-16 programs]$ python validate_gff3.py /rds/user/fc464/hpc-work/TetramoriumProject/Annotations/GTFs/Loci_isoforms.All.DEF.ManualCurated.gff3
Found 29401 transcripts and 14844 genes
Found the following gene biotypes: Counter({'protein_coding': 14844})
Found the following transcript biotypes: Counter({'protein_coding': 29401})
Some example database rows:
                             GeneId          TranscriptName      GeneName     GeneBiotype TranscriptBiotype
TranscriptId                                                                                               
Talp_g17480.t12.m27674  Talp_g17480          wnd-t12.m27674           wnd  protein_coding    protein_coding
Talp_g3928.t1.m6669      Talp_g3928   LOC100115283-t1.m6669  LOC100115283  protein_coding    protein_coding
Talp_g5158.t1.m8585      Talp_g5158   LOC103316829-t1.m8585  LOC103316829  protein_coding    protein_coding
Talp_g3942.t1.m6702      Talp_g3942           armi-t1.m6702          armi  protein_coding    protein_coding
Talp_g17067.t11.m26981  Talp_g17067         tutl-t11.m26981          tutl  protein_coding    protein_coding
Talp_g1003.t1.m1805      Talp_g1003          FucTA-t1.m1805         FucTA  protein_coding    protein_coding
Talp_g238.t2.m431         Talp_g238             hts-t2.m431           hts  protein_coding    protein_coding
Talp_g12657.t1.m19685   Talp_g12657         fd59A-t1.m19685         fd59A  protein_coding    protein_coding
Talp_g12576.t2.m19579   Talp_g12576           sbr-t2.m19579           sbr  protein_coding    protein_coding
Talp_g13790.t1.m21558   Talp_g13790    PH4alphaPV-t1.m21558    PH4alphaPV  protein_coding    protein_coding
Talp_g1496.t1.m2670      Talp_g1496          msl-2-t1.m2670           msl  protein_coding    protein_coding
Talp_g13985.t4.m21840   Talp_g13985          bves-t4.m21840          bves  protein_coding    protein_coding
Talp_g11810.t2          Talp_g11810               CG5316-t2        CG5316  protein_coding    protein_coding
Talp_g17223.t1.m27224   Talp_g17223  LOC103317335-t1.m27224  LOC103317335  protein_coding    protein_coding
Talp_g9236.t1.m14502     Talp_g9236       CG32579-t1.m14502       CG32579  protein_coding    protein_coding
Talp_g3971.t4.m6754      Talp_g3971           Afti-t4.m6754          Afti  protein_coding    protein_coding
Talp_g6789.t1.m10913     Talp_g6789          Mcm6-t1.m10913          Mcm6  protein_coding    protein_coding
Talp_g2470.t1.m4285      Talp_g2470        OG_9491-t1.m4285       OG_9491  protein_coding    protein_coding
Talp_g9688.t3.m15238     Talp_g9688          AOX3-t3.m15238          AOX3  protein_coding    protein_coding
Talp_g439.t1.m770         Talp_g439            Kdm2-t1.m770          Kdm2  protein_coding    protein_coding
Some example gene rows:
                    TranscriptName      GeneName     GeneBiotype TranscriptBiotype
GeneId                                                                            
Talp_g1                RpS5a-t1.m1         RpS5a  protein_coding    protein_coding
Talp_g100          CG31522-t1.m180       CG31522  protein_coding    protein_coding
Talp_g1000   LOC100124084-t1.m1801  LOC100124084  protein_coding    protein_coding
Talp_g10000       CG2023-t1.m15774        CG2023  protein_coding    protein_coding
Talp_g10001          Nca-t1.m15775           Nca  protein_coding    protein_coding
Talp_g10003        Vps15-t3.m15778         Vps15  protein_coding    protein_coding
Talp_g10004    U4-U6-60K-t1.m15779            U4  protein_coding    protein_coding
Talp_g10005   epsilonCOP-t2.m15781    epsilonCOP  protein_coding    protein_coding
Talp_g10006        BRWD3-t3.m15784         BRWD3  protein_coding    protein_coding
Talp_g10007      CG10778-t1.m15787       CG10778  protein_coding    protein_coding

ifiddes commented 6 years ago

Looks good to me.

Ian Fiddes

On Oct 3, 2018, at 3:49 AM, Francesco Cicconardi notifications@github.com wrote:

It seems ok, right?

(luigi_env)[fc464@login-e-16 programs]$ python validate_gff3.py /rds/user/fc464/hpc-work/TetramoriumProject/Annotations/GTFs/Loci_isoforms.All.DEF.ManualCurated.gff3 Found 29401 transcripts and 14844 genes Found the following gene biotypes: Counter({'protein_coding': 14844}) Found the following transcript biotypes: Counter({'protein_coding': 29401}) Some example database rows: GeneId TranscriptName GeneName GeneBiotype TranscriptBiotype TranscriptId
Talp_g17480.t12.m27674 Talp_g17480 wnd-t12.m27674 wnd protein_coding protein_coding Talp_g3928.t1.m6669 Talp_g3928 LOC100115283-t1.m6669 LOC100115283 protein_coding protein_coding Talp_g5158.t1.m8585 Talp_g5158 LOC103316829-t1.m8585 LOC103316829 protein_coding protein_coding Talp_g3942.t1.m6702 Talp_g3942 armi-t1.m6702 armi protein_coding protein_coding Talp_g17067.t11.m26981 Talp_g17067 tutl-t11.m26981 tutl protein_coding protein_coding Talp_g1003.t1.m1805 Talp_g1003 FucTA-t1.m1805 FucTA protein_coding protein_coding Talp_g238.t2.m431 Talp_g238 hts-t2.m431 hts protein_coding protein_coding Talp_g12657.t1.m19685 Talp_g12657 fd59A-t1.m19685 fd59A protein_coding protein_coding Talp_g12576.t2.m19579 Talp_g12576 sbr-t2.m19579 sbr protein_coding protein_coding Talp_g13790.t1.m21558 Talp_g13790 PH4alphaPV-t1.m21558 PH4alphaPV protein_coding protein_coding Talp_g1496.t1.m2670 Talp_g1496 msl-2-t1.m2670 msl protein_coding protein_coding Talp_g13985.t4.m21840 Talp_g13985 bves-t4.m21840 bves protein_coding protein_coding Talp_g11810.t2 Talp_g11810 CG5316-t2 CG5316 protein_coding protein_coding Talp_g17223.t1.m27224 Talp_g17223 LOC103317335-t1.m27224 LOC103317335 protein_coding protein_coding Talp_g9236.t1.m14502 Talp_g9236 CG32579-t1.m14502 CG32579 protein_coding protein_coding Talp_g3971.t4.m6754 Talp_g3971 Afti-t4.m6754 Afti protein_coding protein_coding Talp_g6789.t1.m10913 Talp_g6789 Mcm6-t1.m10913 Mcm6 protein_coding protein_coding Talp_g2470.t1.m4285 Talp_g2470 OG_9491-t1.m4285 OG_9491 protein_coding protein_coding Talp_g9688.t3.m15238 Talp_g9688 AOX3-t3.m15238 AOX3 protein_coding protein_coding Talp_g439.t1.m770 Talp_g439 Kdm2-t1.m770 Kdm2 protein_coding protein_coding Some example gene rows: TranscriptName GeneName GeneBiotype TranscriptBiotype GeneId
Talp_g1 RpS5a-t1.m1 RpS5a protein_coding protein_coding Talp_g100 CG31522-t1.m180 CG31522 protein_coding protein_coding Talp_g1000 LOC100124084-t1.m1801 LOC100124084 protein_coding protein_coding Talp_g10000 CG2023-t1.m15774 CG2023 protein_coding protein_coding Talp_g10001 Nca-t1.m15775 Nca protein_coding protein_coding Talp_g10003 Vps15-t3.m15778 Vps15 protein_coding protein_coding Talp_g10004 U4-U6-60K-t1.m15779 U4 protein_coding protein_coding Talp_g10005 epsilonCOP-t2.m15781 epsilonCOP protein_coding protein_coding Talp_g10006 BRWD3-t3.m15784 BRWD3 protein_coding protein_coding Talp_g10007 CG10778-t1.m15787 CG10778 protein_coding protein_coding — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

francicco commented 6 years ago

CAT is running! So far, so good!!! F

francicco commented 6 years ago

I got this errors

ERROR: [pid 54657] Worker Worker(salt=467206512, workers=32, host=login-e-16, username=fc464, pid=165101) failed    ToilTask: GenerateHints for Talp using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1040, in run
    hints_db(self.hints_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db
    combined_hints = t.start(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob
    self._runJobSuccessors(jobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 333, in _runJobSuccessors
    self.issueJobs(successors)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 614, in issueJobs
    self.issueJob(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob
    self.checkResourceRequest(jobNode.memory, cores, jobNode.disk)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest
    raise InsufficientSystemResources('disk', disk, self.maxDisk)
InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 45097156608, Available: 16156389376
ERROR:luigi-interface:[pid 54657] Worker Worker(salt=467206512, workers=32, host=login-e-16, username=fc464, pid=165101) failed    ToilTask: GenerateHints for Talp using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1040, in run
    hints_db(self.hints_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db
    combined_hints = t.start(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob
    self._runJobSuccessors(jobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 333, in _runJobSuccessors
    self.issueJobs(successors)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 614, in issueJobs
    self.issueJob(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob
    self.checkResourceRequest(jobNode.memory, cores, jobNode.disk)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest
    raise InsufficientSystemResources('disk', disk, self.maxDisk)
InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 45097156608, Available: 16156389376
2018-10-04 17:16:52,065 - luigi-interface - ERROR - [pid 54657] Worker Worker(salt=467206512, workers=32, host=login-e-16, username=fc464, pid=165101) failed    ToilTask: GenerateHints for Talp using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1040, in run
    hints_db(self.hints_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db
    combined_hints = t.start(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob
    self._runJobSuccessors(jobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 333, in _runJobSuccessors
    self.issueJobs(successors)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 614, in issueJobs
    self.issueJob(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob
    self.checkResourceRequest(jobNode.memory, cores, jobNode.disk)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest
    raise InsufficientSystemResources('disk', disk, self.maxDisk)
InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 45097156608, Available: 16156389376

but the flag --maxDisk seems not to work

ifiddes commented 6 years ago

The problem isn't the --maxDisk flag, it is the drive you are trying to write to itself. Toil checks to see if the disk usage guessed for a job will exceed either the maxDisk OR the total space available on the disk. My guess is that your worker nodes have a small $TMPDIR, or that it is full. If all of your nodes share access to the same filesystem, you can set --workDir to a directory that exists on that shared filesystem and toil will place temporary files there.

On Thu, Oct 4, 2018 at 9:21 AM Francesco Cicconardi < notifications@github.com> wrote:

I got this errors

ERROR: [pid 54657] Worker Worker(salt=467206512, workers=32, host=login-e-16, username=fc464, pid=165101) failed ToilTask: GenerateHints for Talp using batchSystem singleMachine Traceback (most recent call last): File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run new_deps = self._run_get_new_deps() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps task_gen = self.task.run() File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/init.py", line 1040, in run hints_db(self.hints_args, toil_options) File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db combined_hints = t.start(job) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start return self._runMainLoop(rootJobGraph) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop jobCache=self._jobCache).run() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run self.innerLoop() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop self._processReadyJobs() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs self._processReadyJob(jobGraph, resultStatus) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob self._runJobSuccessors(jobGraph) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 333, in _runJobSuccessors self.issueJobs(successors) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 614, in issueJobs self.issueJob(job) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob self.checkResourceRequest(jobNode.memory, cores, jobNode.disk) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest raise InsufficientSystemResources('disk', disk, self.maxDisk) InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 45097156608, Available: 16156389376 ERROR:luigi-interface:[pid 54657] Worker Worker(salt=467206512, workers=32, host=login-e-16, username=fc464, pid=165101) failed ToilTask: GenerateHints for Talp using batchSystem singleMachine Traceback (most recent call last): File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run new_deps = self._run_get_new_deps() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps task_gen = self.task.run() File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/init.py", line 1040, in run hints_db(self.hints_args, toil_options) File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db combined_hints = t.start(job) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start return self._runMainLoop(rootJobGraph) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop jobCache=self._jobCache).run() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run self.innerLoop() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop self._processReadyJobs() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs self._processReadyJob(jobGraph, resultStatus) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob self._runJobSuccessors(jobGraph) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 333, in _runJobSuccessors self.issueJobs(successors) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 614, in issueJobs self.issueJob(job) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob self.checkResourceRequest(jobNode.memory, cores, jobNode.disk) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest raise InsufficientSystemResources('disk', disk, self.maxDisk) InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 45097156608, Available: 16156389376 2018-10-04 17:16:52,065 - luigi-interface - ERROR - [pid 54657] Worker Worker(salt=467206512, workers=32, host=login-e-16, username=fc464, pid=165101) failed ToilTask: GenerateHints for Talp using batchSystem singleMachine Traceback (most recent call last): File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run new_deps = self._run_get_new_deps() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps task_gen = self.task.run() File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/init.py", line 1040, in run hints_db(self.hints_args, toil_options) File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db combined_hints = t.start(job) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start return self._runMainLoop(rootJobGraph) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop jobCache=self._jobCache).run() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run self.innerLoop() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop self._processReadyJobs() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs self._processReadyJob(jobGraph, resultStatus) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob self._runJobSuccessors(jobGraph) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 333, in _runJobSuccessors self.issueJobs(successors) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 614, in issueJobs self.issueJob(job) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob self.checkResourceRequest(jobNode.memory, cores, jobNode.disk) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest raise InsufficientSystemResources('disk', disk, self.maxDisk) InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 45097156608, Available: 16156389376

but the flag --maxDisk seems not to work

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/116#issuecomment-427081758, or mute the thread https://github.com/notifications/unsubscribe-auth/AHdLXcURnyPJEjYwGtIIfWyKf1433pKyks5uhjWbgaJpZM4XEPUk .

-- Ian Fiddes, PhD

francicco commented 6 years ago

OOOOOok I think it's my falt, because I though I was in a different partition with more space, I'll change it! F

francicco commented 6 years ago

Augustus gave and error. It's definitely searching in the wrong spot. I'm gonna try to start from the very beginning

INFO:cat:Launching AugustusCGP toil pipeline.
ERROR: [pid 159229] Worker Worker(salt=031141483, workers=32, host=cpu-e-617, username=fc464, pid=152727) failed    ToilTask: AugustusCgp using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1450, in run
    cgp_args.cgp_cfg = self.prepare_cgp_cfg(pipeline_args)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1435, in prepare_cgp_cfg
    template = open(pipeline_args.augustus_cgp_cfg_template).read()
IOError: [Errno 2] No such file or directory: '/rds/user/fc464/hpc-work/TetramoriumProject/augustus_cfgs/cgp_extrinsic_template.cfg'
ERROR:luigi-interface:[pid 159229] Worker Worker(salt=031141483, workers=32, host=cpu-e-617, username=fc464, pid=152727) failed    ToilTask: AugustusCgp using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1450, in run
    cgp_args.cgp_cfg = self.prepare_cgp_cfg(pipeline_args)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1435, in prepare_cgp_cfg
    template = open(pipeline_args.augustus_cgp_cfg_template).read()
IOError: [Errno 2] No such file or directory: '/rds/user/fc464/hpc-work/TetramoriumProject/augustus_cfgs/cgp_extrinsic_template.cfg'

francicco commented 6 years ago

Other errors, which I don't understand

WARNING:toil.leader:h/p/jobBBfiIJ    IOError: [Errno [Errno 5] Input/output error] <class 'toil.job.Promise'>: ('fi
le:/home/fc464/rds/rds-shm37-helixmbodyw/Talpestre4Genomes.CAT.workingDir/toil/chaining/jobStore', 'm/Y/tmpW2fo52.t
mp')
2018-10-07 21:32:28,322 - toil.leader - WARNING - h/p/jobBBfiIJ    IOError: [Errno [Errno 5] Input/output error] <c
lass 'toil.job.Promise'>: ('file:/home/fc464/rds/rds-shm37-helixmbodyw/Talpestre4Genomes.CAT.workingDir/toil/chaini
ng/jobStore', 'm/Y/tmpW2fo52.tmp')
WARNING:toil.leader:h/p/jobBBfiIJ    ERROR:toil.worker:Exiting the worker because of a failed job on host cpu-e-695
2018-10-07 21:32:28,322 - toil.leader - WARNING - h/p/jobBBfiIJ    ERROR:toil.worker:Exiting the worker because of a failed job on host cpu-e-695

ERROR: [pid 451276] Worker Worker(salt=262158798, workers=32, host=cpu-e-695, username=fc464, pid=450778) failed    ToilTask: Chaining using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1089, in run
    chaining(chain_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/chaining.py", line 31, in chaining
    chain_file_ids = t.start(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 242, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
FailedJobsException
ERROR:luigi-interface:[pid 451276] Worker Worker(salt=262158798, workers=32, host=cpu-e-695, username=fc464, pid=450778) failed    ToilTask: Chaining using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1089, in run
    chaining(chain_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/chaining.py", line 31, in chaining
    chain_file_ids = t.start(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 242, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
FailedJobsException
2018-10-07 21:52:20,150 - luigi-interface - ERROR - [pid 451276] Worker Worker(salt=262158798, workers=32, host=cpu-e-695, username=fc464, pid=450778) failed    ToilTask: Chaining using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1089, in run
    chaining(chain_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/chaining.py", line 31, in chaining
    chain_file_ids = t.start(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 242, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
FailedJobsException

ERROR: [pid 90124] Worker Worker(salt=262158798, workers=32, host=cpu-e-695, username=fc464, pid=450778) failed    
ToilTask: GenerateHints for Talp using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1040, in run
    hints_db(self.hints_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db
    combined_hints = t.start(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob
    self._runJobSuccessors(jobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 333, in _runJobSuccessors
self.issueJobs(successors)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 614, in issueJobs
    self.issueJob(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob
    self.checkResourceRequest(jobNode.memory, cores, jobNode.disk)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest
    raise InsufficientSystemResources('disk', disk, self.maxDisk)
InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 45097156608, Available: 17179856896
ERROR:luigi-interface:[pid 90124] Worker Worker(salt=262158798, workers=32, host=cpu-e-695, username=fc464, pid=450778) failed    ToilTask: GenerateHints for Talp using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1040, in run
    hints_db(self.hints_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db
    combined_hints = t.start(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob
    self._runJobSuccessors(jobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 333, in _runJobSuccessors
    self.issueJobs(successors)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 614, in issueJobs
    self.issueJob(job)
File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob
    self.checkResourceRequest(jobNode.memory, cores, jobNode.disk)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest
    raise InsufficientSystemResources('disk', disk, self.maxDisk)
InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 45097156608, Available: 17179856896
2018-10-07 21:53:22,881 - luigi-interface - ERROR - [pid 90124] Worker Worker(salt=262158798, workers=32, host=cpu-e-695, username=fc464, pid=450778) failed    ToilTask: GenerateHints for Talp using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1040, in run
    hints_db(self.hints_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db
    combined_hints = t.start(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob
    self._runJobSuccessors(jobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 333, in _runJobSuccessors
    self.issueJobs(successors)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 614, in issueJobs
    self.issueJob(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob
    self.checkResourceRequest(jobNode.memory, cores, jobNode.disk)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest
    raise InsufficientSystemResources('disk', disk, self.maxDisk)
InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 45097156608, Available: 17179856896

In this last one it's asking for more space, but that is fake. He has 2TB at least available. A couple more of these type of errors then the final info:

INFO: 
===== Luigi Execution Summary =====

Scheduled 103 tasks of which:
* 28 ran successfully:
    - 1 FakePsl(...)
    - 1 FlatTranscriptFasta(...)
    - 2 GenerateHints(...)
    - 4 GenomeFasta(...)
    - 1 GenomeFiles(...)
    ...
* 3 failed:
    - 1 Chaining(...)
    - 2 GenerateHints(...)
* 72 were left pending, among these:
    * 72 had failed dependencies:
        - 3 AlignTranscriptDriverTask(...)
        - 1 AlignTranscripts(...)
        - 1 AssemblyHub(...)
        - 1 Augustus(...)
        - 1 AugustusCgp(...)
        ...

This progress looks :( because there were failed tasks

francicco commented 6 years ago

It looks like there is a problem with sort

ProcException: process exited 2: sort -k14,14 -k16,16n
DEBUG:luigi.scheduler:TransMapPsl_False_True_True_aa492fef2f task num failures is 1 and limit is 999999999
INFO: Informed scheduler that task   TransMapPsl_False_True_True_aa492fef2f   has status   FAILED
INFO:luigi-interface:Informed scheduler that task   TransMapPsl_False_True_True_aa492fef2f   has status   FAILED

ifiddes commented 6 years ago

The first error, in chaining, is not one I have seen before, but since it is an I/O error I suspect it is still disk space. The second error, the disk usage request one, is saying that the specific job is requesting 45GB and the drive has only 17GB. It can't be wrong about this, so that means that something is wrong and it is still pointing to your $TMPDIR somehow. For the sort error, that is weird. That one should be easily reproducible by hand, however. But I don't understand, because that stage cannot run until the chaining step is done. What does your $work_dir/chaining folder look like?

francicco commented 6 years ago

Hi @ifiddes,

This is how that dir looks like:

[fc464@login-e-10 chaining]$ ll
total 30M
-rw-rw-r-- 1 fc464 4.9M Oct  8 11:39 Talp-bkp55bd.chain
-rw-rw-r-- 1 fc464 5.6M Oct  8 11:39 Talp-Talp2.chain
-rw-rw-r-- 1 fc464  19M Oct  8 11:39 Talp-Timm.chain

ifiddes commented 6 years ago

Are those all of your target genomes? It seems like the chaining step ended up finishing despite the errors reported above. Some sort of race condition is at play here. So you are actually past the chaining step and on to the transMap step.

What happens if you run transMap by hand, doing something like pslMap -chainMapFile reference/*psl chaining/Talp-bkp55bd.chain /dev/stdout | pslMapPostChain /dev/stdin /dev/stdout? That will produce the input to the sort step.

francicco commented 6 years ago

so, Talp is the reference, bkp55bd, Talp2 and Timm are not reference.

if I run the command I got stuff

[fc464@login-e-10 Talpestre4Genomes.CAT.workingDir]$ pslMap -chainMapFile reference/*psl chaining/Talp-bkp55bd.chain /dev/stdout | pslMapPostChain /dev/stdin /dev/stdout | head 
2291    0   0   0   0   0   8   1690    +   Talp_g500.t1.m888   2291    0   2291    flattened_line_3026 39462   19486   23467   9   103,639,174,208,136,355,234,221,221,    0,103,742,916,1124,1260,1615,1849,2070, 19486,20186,21296,21570,21854,22085,22521,22838,23246,
1349    0   0   0   0   0   0   0   -   Talp_g16434.t1.m26030   1349    0   1349    flattened_line_2389 52410   19027   20376   1   1349,   0,  19027,
3835    0   0   0   0   0   4   1236    -   Talp_g15707.t5.m24869   3835    0   3835    flattened_line_5895 15309   246 5317    5   245,223,1655,1392,320,  0,245,468,2123,3515,    246,492,716,2383,4997,
4168    0   0   0   0   0   11  6964    -   Talp_g7005.t1.m11205    4168    0   4168    flattened_line_1791 70505   52534   63666   12  156,156,150,140,1559,355,283,240,117,717,260,35,    0,156,312,462,602,2161,2516,2799,3039,3156,3873,4133,   52534,52796,53037,53461,58485,60109,60780,61223,61551,61763,62555,63631,
887 0   0   0   0   0   1   83  +   Talp_g18156.t2.m28762   887 0   88flattened_line_1653   76574   60968   61938   2   24,863, 0,24,   60968,61075,
4714    0   0   0   3   10  3   4   +   Talp_g5275.t1.m8761 4724    0   4724    flattened_line_161  265454  105624  110342  7   371,1185,405,56,420,2040,237,   0,372,1557,1962,2019,2447,4487, 105624,105995,107181,107587,107643,108063,110105,
435 0   0   0   6   702 0   0   -   Talp_g5275.t1.m8761 4724    0   1137    flattened_line_3975 27588   25250   25685   7   133,67,11,66,19,28,111, 3587,3721,3790,3805,3873,3897,4613, 25250,25383,25450,25461,25527,25546,25574,
434 0   0   0   6   703 0   0   -   Talp_g5275.t1.m8761 4724    0   1137    flattened_line_7766 9833    7349    7783    7   102,97,11,66,19,28,111, 3587,3691,3790,3805,3873,3897,4613, 7349,7451,7548,7559,7625,7644,7672,
5495    0   0   0   1   5   15  11406   -   Talp_g6386.t2.m10305    5941    441 5941    flattened_line_2928 41110   19928   36829   17  2227,500,635,109,170,93,110,128,133,242,146,253,197,192,94,77,189,  0,2227,2732,3367,3476,3646,3739,3849,3977,4110,4352,4498,4751,4948,5140,5234,5311,  19928,22166,22666,23603,23832,24159,24540,24717,25170,25616,28072,29246,30631,30920,31259,31581,36640,
441 0   0   0   0   0   0   0   +   Talp_g6386.t2.m10305    5941    0   44flattened_line_4775   21362   20538   20979   1   441,    0,  20538,

ifiddes commented 6 years ago

Hmm, weird. And then if you turn it into the full command, does it fail:

pslMap -chainMapFile reference/*psl chaining/Talp-bkp55bd.chain /dev/stdout | pslMapPostChain /dev/stdin /dev/stdout | sort -k14,14 -k16,16n | pslRecalcMatch /dev/stdin genome_files/Talp.2bit reference/*fasta stdout | sort -k1,10?

If that works, it may have been some sort of transient problem related to this new data race... and that would suggest that if you just start the pipeline again it will work this time.

francicco commented 6 years ago

No fasta file!!!!

[fc464@login-e-10 Talpestre4Genomes.CAT.workingDir]$ pslMap -chainMapFile reference/*psl chaining/Talp-bkp55bd.chain /dev/stdout | pslMapPostChain /dev/stdin /dev/stdout | sort -k14,14 -k16,16n | pslRecalcMatch /dev/stdin genome_files/Talp.2bit reference/*fasta stdout | sort -k1,10
Couldn't open reference/*fasta , No such file or directory

francicco commented 6 years ago

there is a fasta file,

-rw-rw-r-- 1 fc464 101M Oct 8 08:56 Loci_isoforms.All.DEF.ManualCurated.fa

but it's a .fa and not fasta.

ifiddes commented 6 years ago

Sorry, I was doing it from memory. Replace that with .fa

francicco commented 6 years ago

flattened_line_0 is not in genome_files/Talp.2bit

francicco commented 6 years ago

which is true, flattened_line_0 should be in bkp55bd

francicco commented 6 years ago

changing slightly your command:

[fc464@login-e-10 Talpestre4Genomes.CAT.workingDir]$ pslMap -chainMapFile reference/*psl chaining/Talp-bkp55bd.chain /dev/stdout | pslMapPostChain /dev/stdin /dev/stdout | sort -k14,14 -k16,16n | pslRecalcMatch /dev/stdin genome_files/bkp55bd.2bit reference/*fa stdout | sort -k1,10 | head
0   0   0   1   0   0   0   0   -   Talp_g13634.t1  1325    1324    1325    flattened_line_18823    2049    1024    1025    1   1,  0,  1024,
0   0   0   1   0   0   0   0   -   Talp_g9130.t1   1396    1395    1396    flattened_line_20321    1835    498 499 1   1,  0,  498,
0   0   0   32  0   0   0   0   +   Talp_g17365.t1.m27446   3611    3579    3611    flattened_line_4197 25428   8135    8167    1   32, 3579,   8135,
0   0   0   32  0   0   0   0   +   Talp_g17365.t2.m27447   3603    3571    3603    flattened_line_4197 25428   8135    8167    1   32, 3571,   8135,
0   0   0   32  0   0   0   0   +   Talp_g17365.t3.m27448   4858    4826    4858    flattened_line_4197 25428   8135    8167    1   32, 4826,   8135,
0   0   0   32  0   0   0   0   +   Talp_g17365.t4.m27449   4924    4892    4924    flattened_line_4197 25428   8135    8167    1   32, 4892,   8135,
0   0   1   0   0   0   0   0   +   Talp_g11448.t2.m17872   987 986 98flattened_line_7229   11029   162 163 1   1,  986,    162,
0   0   1   0   0   0   0   0   -   Talp_g2043.t1.m3667 4017    2196    2197    flattened_line_12885    3956    3537    3538    1   1,  1820,   3537,
0   0   1   0   0   0   0   0   +   Talp_g2044.t3.m3670 4524    3696    3697    flattened_line_12885    3956    3537    3538    1   1,  3696,   3537,
0   0   1   0   0   0   0   0   +   Talp_g2044.t4.m3671 5026    4198    4199    flattened_line_12885    3956    3537    3538    1   1,  4198,   3537,

ifiddes commented 6 years ago

Ok, so it should be working again... if you restart the pipeline does it still crash on that sort step?

francicco commented 6 years ago

I think so, I did it several times. And it looks like it restarts from the beginning. I searched for a flag to resume it, but I haven't found it. Is there?

ifiddes commented 6 years ago

It shouldn't restart from the beginning. It will check each job for completion and not run it if it is completed. Can you restart it once now and let it run until it fails and send me the log after it does so? Thanks

francicco commented 6 years ago

I'll do! F

francicco commented 6 years ago

ERROR: [pid 135174] Worker Worker(salt=666997825, workers=32, host=login-e-9, username=fc464, pid=134709) failed    ToilTask: GenerateHints for bkp55bd using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1040, in run
    hints_db(self.hints_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/hints_db.py", line 84, in hints_db
    combined_hints = t.restart()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 814, in restart
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 395, in _processReadyJob
    self.issueJob(JobNode.fromJobGraph(jobGraph))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob
    self.checkResourceRequest(jobNode.memory, cores, jobNode.disk)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest
    raise InsufficientSystemResources('disk', disk, self.maxDisk)
InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 25769803776, Available: 15414497280
ERROR:luigi-interface:[pid 135174] Worker Worker(salt=666997825, workers=32, host=login-e-9, username=fc464, pid=134709) failed    ToilTask: GenerateHints for bkp55bd using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1040, in run
    hints_db(self.hints_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/hints_db.py", line 84, in hints_db
    combined_hints = t.restart()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 814, in restart
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 395, in _processReadyJob
    self.issueJob(JobNode.fromJobGraph(jobGraph))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob
    self.checkResourceRequest(jobNode.memory, cores, jobNode.disk)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest
    raise InsufficientSystemResources('disk', disk, self.maxDisk)
InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 25769803776, Available: 15414497280
2018-10-09 00:00:43,218 - luigi-interface - ERROR - [pid 135174] Worker Worker(salt=666997825, workers=32, host=login-e-9, username=fc464, pid=134709) failed    ToilTask: GenerateHints for bkp55bd using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1040, in run
    hints_db(self.hints_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/hints_db.py", line 84, in hints_db
    combined_hints = t.restart()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 814, in restart
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 395, in _processReadyJob
    self.issueJob(JobNode.fromJobGraph(jobGraph))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob
    self.checkResourceRequest(jobNode.memory, cores, jobNode.disk)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest
    raise InsufficientSystemResources('disk', disk, self.maxDisk)
InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 25769803776, Available: 15414497280

INFO: 
===== Luigi Execution Summary =====

Scheduled 68 tasks of which:
* 8 complete ones were encountered:
    - 1 Chaining(...)
    - 1 EvaluateTransMap(...)
    - 2 GenerateHints(...)
    - 1 GenomeFiles(...)
    - 1 PrepareFiles(...)
    ...
* 2 failed:
    - 2 GenerateHints(...)
* 58 were left pending, among these:
    * 58 had failed dependencies:
        - 3 AlignTranscriptDriverTask(...)
        - 1 AlignTranscripts(...)
        - 1 AssemblyHub(...)
        - 1 Augustus(...)
        - 1 AugustusCgp(...)
        ...

This progress looks :( because there were failed tasks

===== Luigi Execution Summary =====

Cat.log

francicco commented 6 years ago

[fc464@login-e-9 AnnotationTetramorium]$ quota
Filesystem  GiBytes    quota   limit   grace    files    quota    limit   grace User/group
/home          31.1     40.0    40.0       -    ----- No ZFS File Quotas  ----- U:fc464
/rds-d1       299.2      0.0     0.0       -   747711        0        0       - G:fc464
/rds-d4       688.0   1024.0  1126.4       -   721028  1048576  1048576       - G:fc464
/rds-d1      1466.9      0.0     0.0       -     4435        0        0       - G:rds
/rds-d2       806.2      0.0     0.0       -     2430        0        0       - G:rds
/rds-d1      2518.9   5120.0  5120.0       -   766872  2621440  2621440       - G:rds-shm37-helixmbodyw
/rcs1           0.0   1024.0  1024.0       -        2   102400   102400       - G:rcs-wjt27-test1
/rcs1         437.3   5120.0  5120.0       -   119492   512000   512000       - G:rcs-shm37-helixmbody

ifiddes commented 6 years ago

Well despite failing previously, this log shows that transMap ran fine on the subsequent runs. The only thing failing now is hint generation, for the same max disk problems. You said you have set --workDir to a location that is on one of these drives you are showing I assume. I am confused, however, because that log is showing that wherever toil is thinking to write temporary files to only has ~15GB available, which is why it is raising an exception. This could be happening transiently, as things get cleaned up after a failure, so your quota check might be misleading. It may also be that toil is not honoring the request to use --workDir properly... @joelarmstrong have you encountered this?

francicco commented 6 years ago

I'm rerunning it from the beginning

francicco commented 6 years ago

Nothing to be done

2018-10-09 08:00:17,646 - luigi-interface - ERROR - [pid 438235] Worker Worker(salt=477280113, workers=32, host=login-e-11, username=fc464, pid=437270) failed    ToilTask: GenerateHints for Talp using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1040, in run
    hints_db(self.hints_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/hints_db.py", line 84, in hints_db
    combined_hints = t.restart()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 814, in restart
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 395, in _processReadyJob
    self.issueJob(JobNode.fromJobGraph(jobGraph))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob
    self.checkResourceRequest(jobNode.memory, cores, jobNode.disk)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest
    raise InsufficientSystemResources('disk', disk, self.maxDisk)
InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 45097156608, Available: 17167187968
2018-10-09 08:00:19,951 - luigi-interface - ERROR - [pid 438536] Worker Worker(salt=477280113, workers=32, host=login-e-11, username=fc464, pid=437270) failed    ToilTask: GenerateHints for bkp55bd using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/__init__.py", line 1040, in run
    hints_db(self.hints_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/Comparative-Annotation-Toolkit/cat/hints_db.py", line 84, in hints_db
    combined_hints = t.restart()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 814, in restart
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 395, in _processReadyJob
    self.issueJob(JobNode.fromJobGraph(jobGraph))
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob
    self.checkResourceRequest(jobNode.memory, cores, jobNode.disk)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest
    raise InsufficientSystemResources('disk', disk, self.maxDisk)
InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 25769803776, Available: 17167187968

Cat.log

ifiddes commented 6 years ago

I'm not really sure -- Toil certainly seems convinced that you don't have enough space. I can create a branch that always requests a tiny amount of disk space, and then you can see if it crashes due to running out of space somewhere?

I have made a new branch storage_bug that does this https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/tree/storage_bug. Can you try checking this out as before and seeing what happens?

francicco commented 6 years ago

Same error

Uninstalling cat-0.1:
  Would remove:
    /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/cat.egg-link
Proceed (y/n)? y
  Successfully uninstalled cat-0.1
Cloning into 'Comparative-Annotation-Toolkit'...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 3230 (delta 0), reused 1 (delta 0), pack-reused 3226
Receiving objects: 100% (3230/3230), 14.59 MiB | 13.89 MiB/s, done.
Resolving deltas: 100% (2127/2127), done.
Obtaining file:///rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit
Requirement already satisfied: pyfasta>=0.5.2 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.5.2)
Requirement already satisfied: toil>=3.5 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (3.17.0)
Requirement already satisfied: luigi>=2.5 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (2.7.8)
Requirement already satisfied: seaborn>=0.7 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.9.0)
Requirement already satisfied: pandas>=0.18 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.23.4)
Requirement already satisfied: frozendict in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.2)
Requirement already satisfied: configobj>=5.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (5.0.6)
Requirement already satisfied: sqlalchemy>=1.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.2.11)
Requirement already satisfied: ete3 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (3.1.1)
Requirement already satisfied: pysam>=0.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.15.1)
Requirement already satisfied: numpy>=1.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.15.1)
Requirement already satisfied: scipy>=0.18.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.1.0)
Requirement already satisfied: bx-python>=0.7.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.8.2)
Requirement already satisfied: bd2k-python-lib==1.14a1.dev48 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.14a1.dev48)
Requirement already satisfied: requests==2.18.4 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (2.18.4)
Requirement already satisfied: dill==0.2.7.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (0.2.7.1)
Requirement already satisfied: subprocess32<=3.5.2 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (3.5.2)
Requirement already satisfied: future in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (0.16.0)
Requirement already satisfied: six>=1.10.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (1.11.0)
Requirement already satisfied: docker==2.5.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (2.5.1)
Requirement already satisfied: python-dateutil in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (2.7.3)
Requirement already satisfied: tornado<5,>=4.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from luigi>=2.5->cat==0.1) (4.5.3)
Requirement already satisfied: python-daemon<3.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from luigi>=2.5->cat==0.1) (2.2.0)
Requirement already satisfied: enum34>1.1.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from luigi>=2.5->cat==0.1) (1.1.6)
Requirement already satisfied: matplotlib>=1.4.3 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from seaborn>=0.7->cat==0.1) (2.2.3)
Requirement already satisfied: pytz>=2011k in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from pandas>=0.18->cat==0.1) (2018.5)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (1.22)
Requirement already satisfied: idna<2.7,>=2.5 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (2.6)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (2018.8.24)
Requirement already satisfied: websocket-client>=0.32.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (0.53.0)
Requirement already satisfied: backports.ssl-match-hostname>=3.5; python_version < "3.5" in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (3.5.0.1)
Requirement already satisfied: ipaddress>=1.0.16; python_version < "3.3" in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (1.0.22)
Requirement already satisfied: docker-pycreds>=0.2.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (0.3.0)
Requirement already satisfied: singledispatch in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from tornado<5,>=4.0->luigi>=2.5->cat==0.1) (3.4.0.3)
Requirement already satisfied: backports-abc>=0.4 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from tornado<5,>=4.0->luigi>=2.5->cat==0.1) (0.5)
Requirement already satisfied: setuptools in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from python-daemon<3.0->luigi>=2.5->cat==0.1) (40.4.1)
Requirement already satisfied: lockfile>=0.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from python-daemon<3.0->luigi>=2.5->cat==0.1) (0.12.2)
Requirement already satisfied: docutils in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from python-daemon<3.0->luigi>=2.5->cat==0.1) (0.14)
Requirement already satisfied: cycler>=0.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (0.10.0)
Requirement already satisfied: backports.functools-lru-cache in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (1.5)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (2.2.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (1.0.1)
Installing collected packages: cat
  Running setup.py develop for cat
Successfully installed cat

2018-10-09 23:58:40,397 - luigi-interface - ERROR - [pid 186248] Worker Worker(salt=478308608, workers=32, host=login-e-9, username=fc464, pid=33934) failed    ToilTask: GenerateHints for Talp using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/__init__.py", line 1040, in run
    hints_db(self.hints_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db
    combined_hints = t.start(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob
--
2018-10-09 23:59:31,593 - luigi-interface - ERROR - [pid 437928] Worker Worker(salt=478308608, workers=32, host=login-e-9, username=fc464, pid=33934) failed    ToilTask: GenerateHints for bkp55bd using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/__init__.py", line 1040, in run
    hints_db(self.hints_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db
    combined_hints = t.start(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob
[fc464@login-e-9 AnnotationTetramorium]$ grep -A35 ERROR Error
2018-10-09 23:58:40,397 - luigi-interface - ERROR - [pid 186248] Worker Worker(salt=478308608, workers=32, host=login-e-9, username=fc464, pid=33934) failed    ToilTask: GenerateHints for Talp using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/__init__.py", line 1040, in run
    hints_db(self.hints_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db
    combined_hints = t.start(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob
    self._runJobSuccessors(jobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 333, in _runJobSuccessors
    self.issueJobs(successors)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 614, in issueJobs
    self.issueJob(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob
    self.checkResourceRequest(jobNode.memory, cores, jobNode.disk)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest
    raise InsufficientSystemResources('disk', disk, self.maxDisk)
InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 45097156608, Available: 15414611968
2018-10-09 23:58:41,571 - toil.leader - INFO - Issued job 'setup_hints' m/6/jobHRCg2q with job batch system ID: 0 and cores: 1, disk: 4.0 G, and memory: 8.0 G
2018-10-09 23:58:45,641 - toil.leader - INFO - Job ended successfully: 'setup_hints' m/6/jobHRCg2q
2018-10-09 23:58:45,643 - toil.leader - INFO - Issued job 'merge_bams' 5/f/jobXz69l_ with job batch system ID: 1 and cores: 1, disk: 8.0 G, and memory: 8.0 G
--
2018-10-09 23:59:31,593 - luigi-interface - ERROR - [pid 437928] Worker Worker(salt=478308608, workers=32, host=login-e-9, username=fc464, pid=33934) failed    ToilTask: GenerateHints for bkp55bd using batchSystem singleMachine
Traceback (most recent call last):
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run
    new_deps = self._run_get_new_deps()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps
    task_gen = self.task.run()
  File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/__init__.py", line 1040, in run
    hints_db(self.hints_args, toil_options)
  File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db
    combined_hints = t.start(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run
    self.innerLoop()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop
    self._processReadyJobs()
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs
    self._processReadyJob(jobGraph, resultStatus)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob
    self._runJobSuccessors(jobGraph)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 333, in _runJobSuccessors
    self.issueJobs(successors)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 614, in issueJobs
    self.issueJob(job)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob
    jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob
    self.checkResourceRequest(jobNode.memory, cores, jobNode.disk)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest
    raise InsufficientSystemResources('disk', disk, self.maxDisk)
InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 25769803776, Available: 15414611968

But maybe I'm not istalling it properly. Is it the right branch? F

ifiddes commented 6 years ago

You need to check out the branch storage_bug (git checkout storage_bug && git pull). Also, delete the toil directory so that it starts with a clean run.

On Wed, Oct 10, 2018 at 12:25 AM Francesco Cicconardi < notifications@github.com> wrote:

Same error

Uninstalling cat-0.1: Would remove: /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/cat.egg-link Proceed (y/n)? y Successfully uninstalled cat-0.1 Cloning into 'Comparative-Annotation-Toolkit'... remote: Enumerating objects: 4, done. remote: Counting objects: 100% (4/4), done. remote: Compressing objects: 100% (4/4), done. remote: Total 3230 (delta 0), reused 1 (delta 0), pack-reused 3226 Receiving objects: 100% (3230/3230), 14.59 MiB | 13.89 MiB/s, done. Resolving deltas: 100% (2127/2127), done. Obtaining file:///rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit Requirement already satisfied: pyfasta>=0.5.2 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.5.2) Requirement already satisfied: toil>=3.5 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (3.17.0) Requirement already satisfied: luigi>=2.5 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (2.7.8) Requirement already satisfied: seaborn>=0.7 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.9.0) Requirement already satisfied: pandas>=0.18 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.23.4) Requirement already satisfied: frozendict in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.2) Requirement already satisfied: configobj>=5.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (5.0.6) Requirement already satisfied: sqlalchemy>=1.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.2.11) Requirement already satisfied: ete3 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (3.1.1) Requirement already satisfied: pysam>=0.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.15.1) Requirement already satisfied: numpy>=1.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.15.1) Requirement already satisfied: scipy>=0.18.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.1.0) Requirement already satisfied: bx-python>=0.7.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.8.2) Requirement already satisfied: bd2k-python-lib==1.14a1.dev48 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.14a1.dev48) Requirement already satisfied: requests==2.18.4 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (2.18.4) Requirement already satisfied: dill==0.2.7.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (0.2.7.1) Requirement already satisfied: subprocess32<=3.5.2 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (3.5.2) Requirement already satisfied: future in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (0.16.0) Requirement already satisfied: six>=1.10.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (1.11.0) Requirement already satisfied: docker==2.5.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (2.5.1) Requirement already satisfied: python-dateutil in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (2.7.3) Requirement already satisfied: tornado<5,>=4.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from luigi>=2.5->cat==0.1) (4.5.3) Requirement already satisfied: python-daemon<3.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from luigi>=2.5->cat==0.1) (2.2.0) Requirement already satisfied: enum34>1.1.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from luigi>=2.5->cat==0.1) (1.1.6) Requirement already satisfied: matplotlib>=1.4.3 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from seaborn>=0.7->cat==0.1) (2.2.3) Requirement already satisfied: pytz>=2011k in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from pandas>=0.18->cat==0.1) (2018.5) Requirement already satisfied: urllib3<1.23,>=1.21.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (1.22) Requirement already satisfied: idna<2.7,>=2.5 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (2.6) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (3.0.4) Requirement already satisfied: certifi>=2017.4.17 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (2018.8.24) Requirement already satisfied: websocket-client>=0.32.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (0.53.0) Requirement already satisfied: backports.ssl-match-hostname>=3.5; python_version < "3.5" in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (3.5.0.1) Requirement already satisfied: ipaddress>=1.0.16; python_version < "3.3" in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (1.0.22) Requirement already satisfied: docker-pycreds>=0.2.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (0.3.0) Requirement already satisfied: singledispatch in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from tornado<5,>=4.0->luigi>=2.5->cat==0.1) (3.4.0.3) Requirement already satisfied: backports-abc>=0.4 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from tornado<5,>=4.0->luigi>=2.5->cat==0.1) (0.5) Requirement already satisfied: setuptools in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from python-daemon<3.0->luigi>=2.5->cat==0.1) (40.4.1) Requirement already satisfied: lockfile>=0.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from python-daemon<3.0->luigi>=2.5->cat==0.1) (0.12.2) Requirement already satisfied: docutils in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from python-daemon<3.0->luigi>=2.5->cat==0.1) (0.14) Requirement already satisfied: cycler>=0.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (0.10.0) Requirement already satisfied: backports.functools-lru-cache in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (1.5) Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (2.2.1) Requirement already satisfied: kiwisolver>=1.0.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (1.0.1) Installing collected packages: cat Running setup.py develop for cat Successfully installed cat

2018-10-09 23:58:40,397 - luigi-interface - ERROR - [pid 186248] Worker Worker(salt=478308608, workers=32, host=login-e-9, username=fc464, pid=33934) failed ToilTask: GenerateHints for Talp using batchSystem singleMachine Traceback (most recent call last): File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run new_deps = self._run_get_new_deps() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps task_gen = self.task.run() File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/init.py", line 1040, in run hints_db(self.hints_args, toil_options) File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db combined_hints = t.start(job) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start return self._runMainLoop(rootJobGraph) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop jobCache=self._jobCache).run() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run self.innerLoop() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop self._processReadyJobs() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs self._processReadyJob(jobGraph, resultStatus) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob

2018-10-09 23:59:31,593 - luigi-interface - ERROR - [pid 437928] Worker Worker(salt=478308608, workers=32, host=login-e-9, username=fc464, pid=33934) failed ToilTask: GenerateHints for bkp55bd using batchSystem singleMachine Traceback (most recent call last): File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run new_deps = self._run_get_new_deps() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps task_gen = self.task.run() File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/init.py", line 1040, in run hints_db(self.hints_args, toil_options) File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db combined_hints = t.start(job) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start return self._runMainLoop(rootJobGraph) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop jobCache=self._jobCache).run() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run self.innerLoop() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop self._processReadyJobs() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs self._processReadyJob(jobGraph, resultStatus) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob [fc464@login-e-9 AnnotationTetramorium]$ grep -A35 ERROR Error 2018-10-09 23:58:40,397 - luigi-interface - ERROR - [pid 186248] Worker Worker(salt=478308608, workers=32, host=login-e-9, username=fc464, pid=33934) failed ToilTask: GenerateHints for Talp using batchSystem singleMachine Traceback (most recent call last): File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run new_deps = self._run_get_new_deps() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps task_gen = self.task.run() File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/init.py", line 1040, in run hints_db(self.hints_args, toil_options) File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db combined_hints = t.start(job) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start return self._runMainLoop(rootJobGraph) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop jobCache=self._jobCache).run() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run self.innerLoop() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop self._processReadyJobs() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs self._processReadyJob(jobGraph, resultStatus) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob self._runJobSuccessors(jobGraph) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 333, in _runJobSuccessors self.issueJobs(successors) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 614, in issueJobs self.issueJob(job) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob self.checkResourceRequest(jobNode.memory, cores, jobNode.disk) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest raise InsufficientSystemResources('disk', disk, self.maxDisk) InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 45097156608, Available: 15414611968 2018-10-09 23:58:41,571 - toil.leader - INFO - Issued job 'setup_hints' m/6/jobHRCg2q with job batch system ID: 0 and cores: 1, disk: 4.0 G, and memory: 8.0 G 2018-10-09 23:58:45,641 - toil.leader - INFO - Job ended successfully: 'setup_hints' m/6/jobHRCg2q 2018-10-09 23:58:45,643 - toil.leader - INFO - Issued job 'mergebams' 5/f/jobXz69l with job batch system ID: 1 and cores: 1, disk: 8.0 G, and memory: 8.0 G

2018-10-09 23:59:31,593 - luigi-interface - ERROR - [pid 437928] Worker Worker(salt=478308608, workers=32, host=login-e-9, username=fc464, pid=33934) failed ToilTask: GenerateHints for bkp55bd using batchSystem singleMachine Traceback (most recent call last): File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 199, in run new_deps = self._run_get_new_deps() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/luigi/worker.py", line 139, in _run_get_new_deps task_gen = self.task.run() File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/init.py", line 1040, in run hints_db(self.hints_args, toil_options) File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/hints_db.py", line 81, in hints_db combined_hints = t.start(job) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 783, in start return self._runMainLoop(rootJobGraph) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/common.py", line 1057, in _runMainLoop jobCache=self._jobCache).run() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 212, in run self.innerLoop() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 521, in innerLoop self._processReadyJobs() File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 445, in _processReadyJobs self._processReadyJob(jobGraph, resultStatus) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 415, in _processReadyJob self._runJobSuccessors(jobGraph) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 333, in _runJobSuccessors self.issueJobs(successors) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 614, in issueJobs self.issueJob(job) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/leader.py", line 593, in issueJob jobBatchSystemID = self.batchSystem.issueBatchJob(jobNode) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/singleMachine.py", line 236, in issueBatchJob self.checkResourceRequest(jobNode.memory, cores, jobNode.disk) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/batchSystems/abstractBatchSystem.py", line 251, in checkResourceRequest raise InsufficientSystemResources('disk', disk, self.maxDisk) InsufficientSystemResources: Requesting more disk than either physically available, or enforced by --maxDisk. Requested: 25769803776, Available: 15414611968

But maybe I'm not istalling it properly. Is it the right branch? F

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/ComparativeGenomicsToolkit/Comparative-Annotation-Toolkit/issues/116#issuecomment-428466247, or mute the thread https://github.com/notifications/unsubscribe-auth/AHdLXTjuNxNsbB2YmzKjptufnzPay62_ks5ujaDfgaJpZM4XEPUk .

-- Ian Fiddes, PhD

francicco commented 6 years ago

Hi @ifiddes

###### Wed 10 Oct 16:11:20 BST 2018: Activate Luigi
Uninstalling cat-0.1:
  Would remove:
    /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/cat.egg-link
Proceed (y/n)? y
  Successfully uninstalled cat-0.1
Cloning into 'Comparative-Annotation-Toolkit'...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 3230 (delta 0), reused 1 (delta 0), pack-reused 3226
Receiving objects: 100% (3230/3230), 14.59 MiB | 14.47 MiB/s, done.
Resolving deltas: 100% (2127/2127), done.
Branch storage_bug set up to track remote branch storage_bug from origin.
Switched to a new branch 'storage_bug'
Already up-to-date.
Obtaining file:///rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit
Requirement already satisfied: pyfasta>=0.5.2 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.5.2)
Requirement already satisfied: toil>=3.5 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (3.17.0)
Requirement already satisfied: luigi>=2.5 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (2.7.8)
Requirement already satisfied: seaborn>=0.7 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.9.0)
Requirement already satisfied: pandas>=0.18 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.23.4)
Requirement already satisfied: frozendict in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.2)
Requirement already satisfied: configobj>=5.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (5.0.6)
Requirement already satisfied: sqlalchemy>=1.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.2.11)
Requirement already satisfied: ete3 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (3.1.1)
Requirement already satisfied: pysam>=0.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.15.1)
Requirement already satisfied: numpy>=1.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.15.1)
Requirement already satisfied: scipy>=0.18.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.1.0)
Requirement already satisfied: bx-python>=0.7.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.8.2)
Requirement already satisfied: bd2k-python-lib==1.14a1.dev48 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.14a1.dev48)
Requirement already satisfied: requests==2.18.4 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (2.18.4)
Requirement already satisfied: dill==0.2.7.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (0.2.7.1)
Requirement already satisfied: subprocess32<=3.5.2 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (3.5.2)
Requirement already satisfied: future in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (0.16.0)
Requirement already satisfied: six>=1.10.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (1.11.0)
Requirement already satisfied: docker==2.5.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (2.5.1)
Requirement already satisfied: python-dateutil in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (2.7.3)
Requirement already satisfied: tornado<5,>=4.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from luigi>=2.5->cat==0.1) (4.5.3)
Requirement already satisfied: python-daemon<3.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from luigi>=2.5->cat==0.1) (2.2.0)
Requirement already satisfied: enum34>1.1.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from luigi>=2.5->cat==0.1) (1.1.6)
Requirement already satisfied: matplotlib>=1.4.3 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from seaborn>=0.7->cat==0.1) (2.2.3)
Requirement already satisfied: pytz>=2011k in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from pandas>=0.18->cat==0.1) (2018.5)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (1.22)
Requirement already satisfied: idna<2.7,>=2.5 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (2.6)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (2018.8.24)
Requirement already satisfied: websocket-client>=0.32.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (0.53.0)
Requirement already satisfied: backports.ssl-match-hostname>=3.5; python_version < "3.5" in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (3.5.0.1)
Requirement already satisfied: ipaddress>=1.0.16; python_version < "3.3" in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (1.0.22)
Requirement already satisfied: docker-pycreds>=0.2.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (0.3.0)
Requirement already satisfied: singledispatch in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from tornado<5,>=4.0->luigi>=2.5->cat==0.1) (3.4.0.3)
Requirement already satisfied: backports-abc>=0.4 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from tornado<5,>=4.0->luigi>=2.5->cat==0.1) (0.5)
Requirement already satisfied: setuptools in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from python-daemon<3.0->luigi>=2.5->cat==0.1) (40.4.1)
Requirement already satisfied: lockfile>=0.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from python-daemon<3.0->luigi>=2.5->cat==0.1) (0.12.2)
Requirement already satisfied: docutils in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from python-daemon<3.0->luigi>=2.5->cat==0.1) (0.14)
Requirement already satisfied: cycler>=0.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (0.10.0)
Requirement already satisfied: backports.functools-lru-cache in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (1.5)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (2.2.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (1.0.1)
Installing collected packages: cat
  Running setup.py develop for cat
Successfully installed cat

I'm having a number of these errors

10-10 16:12:50 cat.hints_db INFO  All BAMs validated for Talp2. Beginning Toil hints pipeline
10-10 16:12:50 cat.hints_db WARNING BAM /home/fc464/rds/rds-shm37-helixmbodyw/TetramoriumProject/Annotations/bams/Talp.RNAseq.FinalAlignment.ConcatenatedAllSupported.Aligned.sortedByCoord.out.bam does not have the following sequence/length pairs in its header: scaffold_1707-1545.
10-10 16:12:50 cat.hints_db INFO  All BAMs validated for Timm. Beginning Toil hints pipeline
10-10 16:12:55 toil.statsAndLogging WARNING Got message from job at time 10-10-2018 16:12:55: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance  of failure due to incorrectly requested resources. Job 'generate_annotation_hints' c/q/job0oN0C3 used 2137.50% (21.4 MB [22413312B] used, 1.0 MB [1048576B] requested) at the end of its run.
10-10 16:12:55 toil.statsAndLogging WARNING Got message from job at time 10-10-2018 16:12:55: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance  of failure due to incorrectly requested resources. Job 'generate_annotation_hints' c/q/job0oN0C3 used 2137.50% (21.4 MB [22413312B] used, 1.0 MB [1048576B] requested) at the end of its run.
2018-10-10 16:12:55,154 - toil.statsAndLogging - WARNING - Got message from job at time 10-10-2018 16:12:55: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance  of failure due to incorrectly requested resources. Job 'generate_annotation_hints' c/q/job0oN0C3 used 2137.50% (21.4 MB [22413312B] used, 1.0 MB [1048576B] requested) at the end of its run.
10-10 16:12:56 cat   INFO  Finished GenerateHints Toil pipeline for Talp2.
10-10 16:12:57 luigi-interface INFO  [pid 384569] Worker Worker(salt=533712739, workers=32, host=login-e-10, username=fc464, pid=383584) done      ToilTask: GenerateHints for Talp2 using batchSystem singleMachine
10-10 16:12:59 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_02857668f4   has status   DONE
10-10 16:12:59 cat   INFO  Finished GenerateHints Toil pipeline for Timm.
10-10 16:12:59 luigi-interface INFO  [pid 384562] Worker Worker(salt=533712739, workers=32, host=login-e-10, username=fc464, pid=383584) done      ToilTask: GenerateHints for Timm using batchSystem singleMachine
10-10 16:13:00 luigi-interface INFO  Informed scheduler that task   GenerateHints_False_True_True_ad9e6b880e   has status   DONE
10-10 16:13:16 cat.hints_db INFO  All BAMs validated for bkp55bd. Beginning Toil hints pipeline
10-10 16:13:24 toil.statsAndLogging WARNING Got message from job at time 10-10-2018 16:13:24: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance  of failure due to incorrectly requested resources. Job 'setup_hints' E/O/job9uDJIe used 564510.16% (5.5 GB [5919318016B] used, 1.0 MB [1048576B] requested) at the end of its run.
10-10 16:13:24 toil.statsAndLogging WARNING Got message from job at time 10-10-2018 16:13:24: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance  of failure due to incorrectly requested resources. Job 'setup_hints' E/O/job9uDJIe used 564510.16% (5.5 GB [5919318016B] used, 1.0 MB [1048576B] requested) at the end of its run.
2018-10-10 16:13:24,671 - toil.statsAndLogging - WARNING - Got message from job at time 10-10-2018 16:13:24: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance  of failure due to incorrectly requested resources. Job 'setup_hints' E/O/job9uDJIe used 564510.16% (5.5 GB [5919318016B] used, 1.0 MB [1048576B] requested) at the end of its run.
10-10 16:13:48 cat.hints_db INFO  All BAMs validated for Talp. Beginning Toil hints pipeline
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite
    raise RuntimeError("The termination flag is set, exiting")
RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot join current thread',) in <bound method CachingFileStore.__del__ of <toil.fileStore.CachingFileStore object at 0x7f8d95b46990>> ignored
10-10 16:14:00 toil.leader WARNING The job seems to have left a log file, indicating failure: 'setup_hints' V/l/jobonkzlb
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb    INFO:toil:Running Toil version 3.17.0-585383ed5c4453b556269818571d3c7419c613b0.
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb    WARNING:toil.resource:'JTRES_f89c4ca814d7a4a857ec3cb77eeb1ff6' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb    WARNING:toil.resource:'JTRES_f89c4ca814d7a4a857ec3cb77eeb1ff6' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb    CRITICAL:toil.fileStore:CACHE: Unable to acquire lock on /tmp/toil-7a495da5-69ca-4b3f-9e5b-373e7b352e45-619e78b9-3876-4391-8308-e60915f5c769/cache-7a495da5-69ca-4b3f-9e5b-373e7b352e45/.cacheLock
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb    Traceback (most recent call last):
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1350, in _runner
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb        returnValues = self._run(jobGraph, fileStore)
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1295, in _run
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb        return self.run(fileStore)
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1479, in run
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb      File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/hints_db.py", line 100, in setup_hints
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb        bam_path = job.fileStore.readGlobalFile(bam_file_id)
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 684, in readGlobalFile
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb        '/.'.join(os.path.split(cachedFileName)))
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb      File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 346, in readFile
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb        shutil.copyfile(jobStoreFilePath, localFilePath)
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb      File "/usr/lib64/python2.7/shutil.py", line 84, in copyfile
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb        copyfileobj(fsrc, fdst)
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb      File "/usr/lib64/python2.7/shutil.py", line 52, in copyfileobj
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb        fdst.write(buf)
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb    IOError: [Errno 28] No space left on device
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb    ERROR:toil.worker:Exiting the worker because of a failed job on host login-e-10
10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'setup_hints' V/l/jobonkzlb with ID V/l/jobonkzlb to 1

ifiddes commented 6 years ago

‘No space left on device’. So toil was not wrong about you pointing to a location without enough space when the code was properly requesting space.

What batch system are you using? Somehow the —workDir flag is not properly pointing your jobs away from your tmpdir. What happens if you do export TMPDIR=/some/path/with/space then try again?

Ian Fiddes

On Oct 10, 2018, at 8:18 AM, Francesco Cicconardi notifications@github.com wrote:

Hi @ifiddes

Wed 10 Oct 16:11:20 BST 2018: Activate Luigi

Uninstalling cat-0.1: Would remove: /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/cat.egg-link Proceed (y/n)? y Successfully uninstalled cat-0.1 Cloning into 'Comparative-Annotation-Toolkit'... remote: Enumerating objects: 4, done. remote: Counting objects: 100% (4/4), done. remote: Compressing objects: 100% (4/4), done. remote: Total 3230 (delta 0), reused 1 (delta 0), pack-reused 3226 Receiving objects: 100% (3230/3230), 14.59 MiB | 14.47 MiB/s, done. Resolving deltas: 100% (2127/2127), done. Branch storage_bug set up to track remote branch storage_bug from origin. Switched to a new branch 'storage_bug' Already up-to-date. Obtaining file:///rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit Requirement already satisfied: pyfasta>=0.5.2 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.5.2) Requirement already satisfied: toil>=3.5 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (3.17.0) Requirement already satisfied: luigi>=2.5 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (2.7.8) Requirement already satisfied: seaborn>=0.7 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.9.0) Requirement already satisfied: pandas>=0.18 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.23.4) Requirement already satisfied: frozendict in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.2) Requirement already satisfied: configobj>=5.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (5.0.6) Requirement already satisfied: sqlalchemy>=1.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.2.11) Requirement already satisfied: ete3 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (3.1.1) Requirement already satisfied: pysam>=0.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.15.1) Requirement already satisfied: numpy>=1.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.15.1) Requirement already satisfied: scipy>=0.18.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.1.0) Requirement already satisfied: bx-python>=0.7.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (0.8.2) Requirement already satisfied: bd2k-python-lib==1.14a1.dev48 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from cat==0.1) (1.14a1.dev48) Requirement already satisfied: requests==2.18.4 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (2.18.4) Requirement already satisfied: dill==0.2.7.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (0.2.7.1) Requirement already satisfied: subprocess32<=3.5.2 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (3.5.2) Requirement already satisfied: future in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (0.16.0) Requirement already satisfied: six>=1.10.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (1.11.0) Requirement already satisfied: docker==2.5.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (2.5.1) Requirement already satisfied: python-dateutil in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from toil>=3.5->cat==0.1) (2.7.3) Requirement already satisfied: tornado<5,>=4.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from luigi>=2.5->cat==0.1) (4.5.3) Requirement already satisfied: python-daemon<3.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from luigi>=2.5->cat==0.1) (2.2.0) Requirement already satisfied: enum34>1.1.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from luigi>=2.5->cat==0.1) (1.1.6) Requirement already satisfied: matplotlib>=1.4.3 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from seaborn>=0.7->cat==0.1) (2.2.3) Requirement already satisfied: pytz>=2011k in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from pandas>=0.18->cat==0.1) (2018.5) Requirement already satisfied: urllib3<1.23,>=1.21.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (1.22) Requirement already satisfied: idna<2.7,>=2.5 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (2.6) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (3.0.4) Requirement already satisfied: certifi>=2017.4.17 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from requests==2.18.4->toil>=3.5->cat==0.1) (2018.8.24) Requirement already satisfied: websocket-client>=0.32.0 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (0.53.0) Requirement already satisfied: backports.ssl-match-hostname>=3.5; python_version < "3.5" in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (3.5.0.1) Requirement already satisfied: ipaddress>=1.0.16; python_version < "3.3" in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (1.0.22) Requirement already satisfied: docker-pycreds>=0.2.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from docker==2.5.1->toil>=3.5->cat==0.1) (0.3.0) Requirement already satisfied: singledispatch in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from tornado<5,>=4.0->luigi>=2.5->cat==0.1) (3.4.0.3) Requirement already satisfied: backports-abc>=0.4 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from tornado<5,>=4.0->luigi>=2.5->cat==0.1) (0.5) Requirement already satisfied: setuptools in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from python-daemon<3.0->luigi>=2.5->cat==0.1) (40.4.1) Requirement already satisfied: lockfile>=0.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from python-daemon<3.0->luigi>=2.5->cat==0.1) (0.12.2) Requirement already satisfied: docutils in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from python-daemon<3.0->luigi>=2.5->cat==0.1) (0.14) Requirement already satisfied: cycler>=0.10 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (0.10.0) Requirement already satisfied: backports.functools-lru-cache in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (1.5) Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (2.2.1) Requirement already satisfied: kiwisolver>=1.0.1 in /home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages (from matplotlib>=1.4.3->seaborn>=0.7->cat==0.1) (1.0.1) Installing collected packages: cat Running setup.py develop for cat Successfully installed cat I'm having a number of these errors

10-10 16:12:50 cat.hints_db INFO All BAMs validated for Talp2. Beginning Toil hints pipeline 10-10 16:12:50 cat.hints_db WARNING BAM /home/fc464/rds/rds-shm37-helixmbodyw/TetramoriumProject/Annotations/bams/Talp.RNAseq.FinalAlignment.ConcatenatedAllSupported.Aligned.sortedByCoord.out.bam does not have the following sequence/length pairs in its header: scaffold_1707-1545. 10-10 16:12:50 cat.hints_db INFO All BAMs validated for Timm. Beginning Toil hints pipeline 10-10 16:12:55 toil.statsAndLogging WARNING Got message from job at time 10-10-2018 16:12:55: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job 'generate_annotation_hints' c/q/job0oN0C3 used 2137.50% (21.4 MB [22413312B] used, 1.0 MB [1048576B] requested) at the end of its run. 10-10 16:12:55 toil.statsAndLogging WARNING Got message from job at time 10-10-2018 16:12:55: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job 'generate_annotation_hints' c/q/job0oN0C3 used 2137.50% (21.4 MB [22413312B] used, 1.0 MB [1048576B] requested) at the end of its run. 2018-10-10 16:12:55,154 - toil.statsAndLogging - WARNING - Got message from job at time 10-10-2018 16:12:55: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job 'generate_annotation_hints' c/q/job0oN0C3 used 2137.50% (21.4 MB [22413312B] used, 1.0 MB [1048576B] requested) at the end of its run. 10-10 16:12:56 cat INFO Finished GenerateHints Toil pipeline for Talp2. 10-10 16:12:57 luigi-interface INFO [pid 384569] Worker Worker(salt=533712739, workers=32, host=login-e-10, username=fc464, pid=383584) done ToilTask: GenerateHints for Talp2 using batchSystem singleMachine 10-10 16:12:59 luigi-interface INFO Informed scheduler that task GenerateHints_False_True_True_02857668f4 has status DONE 10-10 16:12:59 cat INFO Finished GenerateHints Toil pipeline for Timm. 10-10 16:12:59 luigi-interface INFO [pid 384562] Worker Worker(salt=533712739, workers=32, host=login-e-10, username=fc464, pid=383584) done ToilTask: GenerateHints for Timm using batchSystem singleMachine 10-10 16:13:00 luigi-interface INFO Informed scheduler that task GenerateHints_False_True_True_ad9e6b880e has status DONE 10-10 16:13:16 cat.hints_db INFO All BAMs validated for bkp55bd. Beginning Toil hints pipeline 10-10 16:13:24 toil.statsAndLogging WARNING Got message from job at time 10-10-2018 16:13:24: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job 'setup_hints' E/O/job9uDJIe used 564510.16% (5.5 GB [5919318016B] used, 1.0 MB [1048576B] requested) at the end of its run. 10-10 16:13:24 toil.statsAndLogging WARNING Got message from job at time 10-10-2018 16:13:24: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job 'setup_hints' E/O/job9uDJIe used 564510.16% (5.5 GB [5919318016B] used, 1.0 MB [1048576B] requested) at the end of its run. 2018-10-10 16:13:24,671 - toil.statsAndLogging - WARNING - Got message from job at time 10-10-2018 16:13:24: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job 'setup_hints' E/O/job9uDJIe used 564510.16% (5.5 GB [5919318016B] used, 1.0 MB [1048576B] requested) at the end of its run. 10-10 16:13:48 cat.hints_db INFO All BAMs validated for Talp. Beginning Toil hints pipeline Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner self.run() File "/usr/lib64/python2.7/threading.py", line 765, in run self.target(*self.args, **self.__kwargs) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception in thread Thread-2: Traceback (most recent call last): File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner self.run() File "/usr/lib64/python2.7/threading.py", line 765, in run self.target(*self.args, **self.__kwargs) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 1471, in asyncWrite raise RuntimeError("The termination flag is set, exiting") RuntimeError: The termination flag is set, exiting

Exception RuntimeError: RuntimeError('cannot 10-10 16:14:00 toil.leader WARNING The 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb 10-10 16:14:00 toil.leader WARNING V/l/jobonkzlb — You are receiving this because you were Reply to this email directly, view it on join current thread',) in <bound method CachingFileStore.del of <toil.fileStore.CachingFileStore object at 0x7f8d95b46990>> ignored job seems to have left a log file, indicating failure: 'setup_hints' V/l/jobonkzlb INFO:toil.worker:---TOIL WORKER OUTPUT LOG--- INFO:toil:Running Toil version 3.17.0-585383ed5c4453b556269818571d3c7419c613b0. WARNING:toil.resource:'JTRES_f89c4ca814d7a4a857ec3cb77eeb1ff6' may exist, but is not yet referenced by the worker (KeyError from os.environ[]). WARNING:toil.resource:'JTRES_f89c4ca814d7a4a857ec3cb77eeb1ff6' may exist, but is not yet referenced by the worker (KeyError from os.environ[]). CRITICAL:toil.fileStore:CACHE: Unable to acquire lock on /tmp/toil-7a495da5-69ca-4b3f-9e5b-373e7b352e45-619e78b9-3876-4391-8308-e60915f5c769/cache-7a495da5-69ca-4b3f-9e5b-373e7b352e45/.cacheLock Traceback (most recent call last): File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/worker.py", line 314, in workerScript job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1350, in _runner returnValues = self._run(jobGraph, fileStore) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1295, in _run return self.run(fileStore) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/job.py", line 1479, in run rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs) File "/rds/project/shm37/rds-shm37-helixmbodyw/TetramoriumProject/Comparative-Annotation-Toolkit/cat/hints_db.py", line 100, in setup_hints bam_path = job.fileStore.readGlobalFile(bam_file_id) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/fileStore.py", line 684, in readGlobalFile '/.'.join(os.path.split(cachedFileName))) File "/home/fc464/software/luigi/luigi_env/lib/python2.7/site-packages/toil/jobStores/fileJobStore.py", line 346, in readFile shutil.copyfile(jobStoreFilePath, localFilePath) File "/usr/lib64/python2.7/shutil.py", line 84, in copyfile copyfileobj(fsrc, fdst) File "/usr/lib64/python2.7/shutil.py", line 52, in copyfileobj fdst.write(buf) IOError: [Errno 28] No space left on device ERROR:toil.worker:Exiting the worker because of a failed job on host login-e-10 WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'setup_hints' V/l/jobonkzlb with ID V/l/jobonkzlb to 1 mentioned. GitHub, or mute the thread.

francicco commented 6 years ago

But there is space:

[fc464@login-e-9 Dpha.Assembly-105.52x-erate-0.105]$ quota
Filesystem  GiBytes    quota   limit   grace    files    quota    limit   grace User/group
/home          31.0     40.0    40.0       -    ----- No ZFS File Quotas  ----- U:fc464
/rds-d1         1.1      0.0     0.0       -    13679        0        0       - G:fc464
/rds-d4       688.0   1024.0  1126.4       -   721028  1048576  1048576       - G:fc464
/rds-d1      1466.9      0.0     0.0       -     4435        0        0       - G:rds
/rds-d2       806.2      0.0     0.0       -     2430        0        0       - G:rds
/rds-d1      1722.6   5120.0  5120.0       -   755354  2621440  2621440       - G:rds-shm37-helixmbodyw
/rcs1           0.0   1024.0  1024.0       -        2   102400   102400       - G:rcs-wjt27-test1
/rcs1         320.2   5120.0  5120.0       -    54006   512000   512000       - G:rcs-shm37-helixmbody

Almost 4TB rds-shm37-helixmbodyw available. Am I reading it wrong? F

ifiddes commented 6 years ago

I don’t know what that quota checking system is. We are talking about missing megabytes of space.

What is your default TMPDIR? (echo $TMPDIR). What happens if you type ‘df -h’?

Ian Fiddes

On Oct 10, 2018, at 8:26 AM, Francesco Cicconardi notifications@github.com wrote:

But there is space:

[fc464@login-e-9 Dpha.Assembly-105.52x-erate-0.105]$ quota Filesystem GiBytes quota limit grace files quota limit grace User/group /home 31.0 40.0 40.0 - ----- No ZFS File Quotas ----- U:fc464 /rds-d1 1.1 0.0 0.0 - 13679 0 0 - G:fc464 /rds-d4 688.0 1024.0 1126.4 - 721028 1048576 1048576 - G:fc464 /rds-d1 1466.9 0.0 0.0 - 4435 0 0 - G:rds /rds-d2 806.2 0.0 0.0 - 2430 0 0 - G:rds /rds-d1 1722.6 5120.0 5120.0 - 755354 2621440 2621440 - G:rds-shm37-helixmbodyw /rcs1 0.0 1024.0 1024.0 - 2 102400 102400 - G:rcs-wjt27-test1 /rcs1 320.2 5120.0 5120.0 - 54006 512000 512000 - G:rcs-shm37-helixmbody Am I reading it wrong? F

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

francicco commented 6 years ago

TMPDIR is empty

[fc464@login-e-10 Eisa.Assembly-76.13x-erate-0.105]$ df -h
Filesystem                                       Size  Used Avail Use% Mounted on
/dev/sda1                                         40G   24G   17G  59% /
devtmpfs                                          94G     0   94G   0% /dev
tmpfs                                             94G  613M   94G   1% /dev/shm
tmpfs                                             94G  315M   94G   1% /run
tmpfs                                             94G     0   94G   0% /sys/fs/cgroup
/dev/sda3                                         57G   33M   57G   1% /local
tmpfs                                             16G  8.6G  7.5G  54% /tmp
10.43.200.2:/volumes/pool2/darwin/home            22T   13T  8.9T  59% /home
10.43.200.2:/volumes/pool2/darwin/usr-local      3.4T  2.3T  1.1T  68% /usr/local
10.144.240.3@o2ib:10.144.240.4@o2ib:/biocld1     1.4P  409T  907T  32% /clincloud
10.144.240.30@o2ib:10.144.240.29@o2ib:/rcs1      231T  171T   49T  78% /rcs1
10.144.240.52@o2ib:10.144.240.53@o2ib:/mrc-bsu1  174T   32T  134T  20% /mrc-bsu/scratch
10.144.240.30@o2ib:10.144.240.29@o2ib:/rds-d1    1.4P  1.1P  317T  78% /rds-d1
10.47.240.68@o2ib1:10.47.240.69@o2ib1:/rds-d2    2.2P 1005T  1.2P  47% /rds-d2
10.47.240.68@o2ib1:10.47.240.69@o2ib1:/rds-d3    2.2P  1.2P  984T  56% /rds-d3
10.47.240.68@o2ib1:10.47.240.69@o2ib1:/rds-d4    2.2P  966T  1.3P  44% /rds-d4
10.47.240.68@o2ib1:10.47.240.69@o2ib1:/rds-d5    2.2P  1.5P  717T  68% /rds-d5

francicco commented 6 years ago

But this doesn't tell much because is the entire cluster

francicco commented 6 years ago

Right now I'm aligning millions of Pacbio seqs into several BAM files without any problem at all! F