search

steineggerlab / ufcg

UFCG: Universal Fungal Core Genes
https://ufcg.steineggerlab.com
GNU General Public License v3.0
29 stars 0 forks source link

ERROR! java.io.IOException: Unable to delete file: .nfs #2

Open Sebastien-Raguideau opened 1 year ago

Sebastien-Raguideau commented 1 year ago

I am encountering small issue with makes me unable to use ufcg on more than a file at a time.

After profiling a genome and writing results there is an attempt to delete temporary files, this fail in a non deterministic way and may be partially related to my server file system speed in closing handle on files.

I suppose ufcg attempt to remove these files too quickly or without having closed them beforehand? In any way, it interrupt execution and I need to work around it by launching ufcg on a unique genome at a time.

endixk commented 1 year ago

Ideally such parallel execution should run without an error, as each of them use randomly generated headers. Still, running multiple processes with writing/deleting operations on the same working directory is always problematic (especially when the directory is mounted across the network).

I would recommend you to use the batch input mode with sufficient number of CPU threads instead (profile -i <dir> -t <cpu>), which can perform fairly similar compared to the parallel execution.

Sebastien-Raguideau commented 1 year ago

Sorry, I was not clear I do use batch execution and the issue interrupts it. So instead I've had to go through external parallel execution of ufcg. When doing external parallel execution, the same issue arises but since it does not impact results and just exit code, it doesn't matter.

I don't code in java but this issue should be easy to fix. At the end of profiling, you delete temporary files too quickly after closing them. Or at least too quickly for my filesystem. The fix would be to add a waiting time of 1 second before removing the temporary directory folder.

I you don't, batch input mode becomes meaningless since your pipeline stops when unable to delete temporary folder. Alternatively you could try to catch that error.

endixk commented 1 year ago

I got the point now. Thank you for the detailed description. As you mentioned this won't be too difficult to fix. I will try to deal with this soon.

endixk commented 1 year ago

@Sebastien-Raguideau Could you please reproduce the error with the newest version?