Pipeline hangs at PHANOTATE step

[cmerenstein]

I've been running Cenote Taker2 on a single sample, and it's been stuck on the PHANOTATE step for a bit over 24 hours now, without any visible update or running process.

When I run ps -a I see that I have several phanotate.py processes running, but via top they all appear to be sleeping.

I've had this happen on several other samples - at one point I ran about 10k Cenote-Taker2 runs, I'd say it happened in about 1-5% of samples.

I saw another issue about this from months ago, but the fix was just to update to the latest version. I did a complete re-install of Cenote-Taker2 and still have the same issue. I've attached the log file here.

Any help getting it to finish the PHANOTATE step would be massively appreciated!! Carter

cenote_log.txt

[mtisza1]

Hi Carter,

Thanks for opening the issue. I haven't seen this before, so I'll need to think about it. I'll probably have to push this until next week. For now, can you send me the file LIN_seqs_for_phanotate.txt in the no_end_contigs_with_viral_domain/ directory? Also, are there any files ending in *phan.fasta that were generated in the no_end_contigs_with_viral_domain/ directory? If so, can you send me a list of those files as well.

Also, did you rerun your 1-5% of runs that ended with this error and did they run fine the second time?

This seems different than past issues, so can you link an issue number that seems similar in your reply?

Thanks,

Mike

[cmerenstein]

Thanks for the quick reply!

I've attached the file you mentioned. There are also 44 files in that dir with *phan.fasta, most of them are a few kb, some are empty.

I don't think I re-ran all the runs that ended with that error, I was scanning a lot of files and then dropped that project once after a certain point and didn't go back to the failures. If it would help though, I can check on a few and rereun them now.

This was the issue I was thinking of: https://github.com/mtisza1/Cenote-Taker2/issues/14 where it sounds like a for loop issue in the PHANOTATE step made it slow, but perhaps that's not as similar as I thought looking back.

Thanks again, Carter

LIN_seqs_for_phanotate.txt

[mtisza1]

OK, quick question:

Do you think you maxed out the memory usage and the computer froze?

[cmerenstein]

I would be surprised if that's happening, but I did limit the memory to 32 gb. Is it possible that it needs more than that? The sample isn't very large, there are only about 4,000 contigs.

[mtisza1]

Hi,

I was never able to come up with a clever way to reproduce this. Let me know if you've been able to gain any additional insights. Otherwise, I'll close this issue.

Best,

Mike

[cmerenstein]

I no longer see this issue in the newest version, closing.