Restart at stage 3

[joannarifkin]

Hi there!

Our cluster went down in the middle of error correction (logfile "rcorrector.log" attached rcorrector.log). I tried to restart it at stage 3 ("restart_rcorrector.sh" below) for the files that weren't completed, but I got an error message ("restart_rcorrector.log" attached restart_rcorrector.log) saying it couldn't open the dump file:

-c tmp_e7b917b99610dc6fc4e8b80c5f556d73.jf_dump Could not open file tmp_e7b917b99610dc6fc4e8b80c5f556d73.jf_dump

The existing dump file is called tmp_0bc1c07ce74e15c5324069f59af374e3.jf_dump. Can I just rename the file?

Thanks!

"restart_rcorrector.sh" script

perl /ohta/joanna.rifkin/rcorrector/run_rcorrector.pl -stage 3 -t 12 \ -1 \ NS.1393.003.NEBNext_dual_i7_150---NEBNext_dual_i5_150.RsgL3_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_151---NEBNext_dual_i5_151.RsaEB1_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_152---NEBNext_dual_i5_152.RsaEB2_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_153---NEBNext_dual_i5_153.RsaL2_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_154---NEBNext_dual_i5_154.RsaEB3_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_155---NEBNext_dual_i5_155.RsaL3_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_156---NEBNext_dual_i5_156.RscEB1_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_157---NEBNext_dual_i5_157.RscL1_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_158---NEBNext_dual_i5_158.RscEB2_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_159---NEBNext_dual_i5_159.RscL2_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_160---NEBNext_dual_i5_160.RscEB3_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_161---NEBNext_dual_i5_161.RscL3_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_162---NEBNext_dual_i5_162.RtrEB1_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_163---NEBNext_dual_i5_163.RtrL1_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_164---NEBNext_dual_i5_164.RtrEB2_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_165---NEBNext_dual_i5_165.RtrL2_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_166---NEBNext_dual_i5_166.RhPG13_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_167---NEBNext_dual_i5_167.RpiPG4_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_168---NEBNext_dual_i5_168.RhPT24_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_191---NEBNext_dual_i5_191.RhPG15_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_192---NEBNext_dual_i5_192.RhPG16_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_97---NEBNext_dual_i5_97.RhaPG2_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_98---NEBNext_dual_i5_98.RhaPG3_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_99---NEBNext_dual_i5_99.RhaPG4_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_A12---NEBNext_dual_i5_A12.RthLF3_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_A1---NEBNext_dual_i5_A1.RacEB1_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_B12---NEBNext_dual_i5_B12.RthEBF1_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_C12---NEBNext_dual_i5_C12.RthEBF2_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_C1---NEBNext_dual_i5_C1.RhPG18_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_D10---NEBNext_dual_i5_D10.RthEB1_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_D12---NEBNext_dual_i5_D12.RthEBF4_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_D1---NEBNext_dual_i5_D1.RthLM1_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_E10---NEBNext_dual_i5_E10.RthL1_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_E12---NEBNext_dual_i5_E12.RthEBF3_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_E1---NEBNext_dual_i5_E1.RthLF1_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_F10---NEBNext_dual_i5_F10.RtrEB1a_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_F12---NEBNext_dual_i5_F12.RthEBM1_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_F1---NEBNext_dual_i5_F1.RthLF2_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_G12---NEBNext_dual_i5_G12.RsaEB13_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_G1---NEBNext_dual_i5_G1.RthLF4_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_H12---NEBNext_dual_i5_H12.RthEB1F1_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_H1---NEBNext_dual_i5_H1.RthL1F1_R1.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_149---NEBNext_dual_i5_149.RsgEB3_R1.cor.fq.gz \ -2 \ NS.1393.003.NEBNext_dual_i7_150---NEBNext_dual_i5_150.RsgL3_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_151---NEBNext_dual_i5_151.RsaEB1_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_152---NEBNext_dual_i5_152.RsaEB2_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_153---NEBNext_dual_i5_153.RsaL2_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_154---NEBNext_dual_i5_154.RsaEB3_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_155---NEBNext_dual_i5_155.RsaL3_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_156---NEBNext_dual_i5_156.RscEB1_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_157---NEBNext_dual_i5_157.RscL1_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_158---NEBNext_dual_i5_158.RscEB2_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_159---NEBNext_dual_i5_159.RscL2_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_160---NEBNext_dual_i5_160.RscEB3_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_161---NEBNext_dual_i5_161.RscL3_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_162---NEBNext_dual_i5_162.RtrEB1_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_163---NEBNext_dual_i5_163.RtrL1_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_164---NEBNext_dual_i5_164.RtrEB2_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_165---NEBNext_dual_i5_165.RtrL2_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_166---NEBNext_dual_i5_166.RhPG13_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_167---NEBNext_dual_i5_167.RpiPG4_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_168---NEBNext_dual_i5_168.RhPT24_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_191---NEBNext_dual_i5_191.RhPG15_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_192---NEBNext_dual_i5_192.RhPG16_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_97---NEBNext_dual_i5_97.RhaPG2_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_98---NEBNext_dual_i5_98.RhaPG3_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_99---NEBNext_dual_i5_99.RhaPG4_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_A12---NEBNext_dual_i5_A12.RthLF3_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_A1---NEBNext_dual_i5_A1.RacEB1_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_B12---NEBNext_dual_i5_B12.RthEBF1_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_C12---NEBNext_dual_i5_C12.RthEBF2_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_C1---NEBNext_dual_i5_C1.RhPG18_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_D10---NEBNext_dual_i5_D10.RthEB1_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_D12---NEBNext_dual_i5_D12.RthEBF4_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_D1---NEBNext_dual_i5_D1.RthLM1_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_E10---NEBNext_dual_i5_E10.RthL1_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_E12---NEBNext_dual_i5_E12.RthEBF3_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_E1---NEBNext_dual_i5_E1.RthLF1_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_F10---NEBNext_dual_i5_F10.RtrEB1a_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_F12---NEBNext_dual_i5_F12.RthEBM1_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_F1---NEBNext_dual_i5_F1.RthLF2_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_G12---NEBNext_dual_i5_G12.RsaEB13_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_G1---NEBNext_dual_i5_G1.RthLF4_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_H12---NEBNext_dual_i5_H12.RthEB1F1_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_H1---NEBNext_dual_i5_H1.RthL1F1_R2.cor.fq.gz,\ NS.1393.003.NEBNext_dual_i7_149---NEBNext_dual_i5_149.RsgEB3_R2.cor.fq.gz

[mourisl]

Yes, you can.

I think the change of the identifier name is because for gzipped input, rcorrector will pipe the results to a temporary file assigned by the system which might change in another run. I will fix this issue. Thank you!

[joannarifkin]

Thank you so much!

On Tue, Nov 10, 2020 at 3:10 PM Li Song notifications@github.com wrote:

Yes, you can.

I think the change of the identifier name is because for gzipped input, rcorrector will pipe the results to a temporary file assigned by the system which would change in another run. I will fix this issue. Thank you!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mourisl/Rcorrector/issues/27#issuecomment-724939090, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFPL6CSRBGY37DKRIND7BU3SPGM4PANCNFSM4TRCYW2A .

-- Joanna Rifkin PhD

Department of Ecology and Evolutionary Biology The University of Toronto 25 Willcocks St. Toronto, ON M5S 3B2

[joannarifkin]

If I rename the files to match the error message, I seem to get empty output files but no error messages. What should I do? I could just restart from the beginning, but it's been running for about 4 days ...

Thanks again!

[mourisl]

I just checked the code and your log, it seems the input files to the two runs were different, that's why the unique identifier names were different.

I think you may put wrong files for the restart run. For example NS.1393.003.NEBNext_dual_i7_150---NEBNext_dual_i5_150.RsgL3_R1.cor.fq.gz is the first file in the restart_rcorrector , it is not in the original run where the file should be NS.1393.003.NEBNext_dual_i7_100---NEBNext_dual_i5_100.RhaPG6_R1.fastq.gz. Furthermore, your read files in the restart_run have the suffix "cor.fq.gz" suggesting those files are from the output of Rcorrector, not the raw file.

[joannarifkin]

Yep, caught that about five minutes after I posted. My bad!

On Tue, Nov 10, 2020 at 4:05 PM Li Song notifications@github.com wrote:

I just checked the code and your log, it seems the input files to the two runs were different, that's why the unique identifier names were different.

I think you may put wrong files for the restart run. For example NS.1393.003.NEBNext_dual_i7_150---NEBNext_dual_i5_150.RsgL3_R1.cor.fq.gz is the first file in the restart_rcorrector , it is not in the original run where the file should be NS.1393.003.NEBNext_dual_i7_100---NEBNext_dual_i5_100.RhaPG6_R1.fastq.gz. Furthermore, your read files in the restart_run have the suffix "cor.fq.gz" suggesting those files are from the output of Rcorrector, not the raw file.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mourisl/Rcorrector/issues/27#issuecomment-724965781, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFPL6CW6WPOMPRCGW5A6V4LSPGTJ5ANCNFSM4TRCYW2A .

-- Joanna Rifkin PhD

Department of Ecology and Evolutionary Biology The University of Toronto 25 Willcocks St. Toronto, ON M5S 3B2