search

vlasmirnov / MAGUS

Graph Clustering Merger
MIT License
32 stars 13 forks source link

The PermissionError #14

Open macelik opened 2 years ago

macelik commented 2 years ago

Hello again. :)

I am having a trouble, and I could not troubleshoot it. I am recieving The PermissionError: [errno 13] permission denied when I try it with my file which has nearly 600k sequences. MAGNUS is running on WSL2 with 40 cores and 400GB RAM. The code i am using is python ./MAGUS-master/magus.py -d outputs1 -i SGNRDiscarded.fasta -o aligned1.txt -t clustal. The thing is it runs alright without problems with the sample file that is provided (in the same directory). I have tried to chown the directory and as well as with sudo command. What i recognised is that the first time it threw the error the output directory was around 1.10GB and then I run again with sudo command and the time i threw the same error, the output directory was 1.45GB. So the same error but i think at different stages. At the moment I opened the WSL2 with Admin priviliges (right click and select run with admin priv) and running it again. In the link below you can find the captured error.png and log.txt.

PermissionError

vlasmirnov commented 2 years ago

Thanks for reaching out again. This is really weird, I admit I've never seen this before. My first instinct is that the disk (or your allocation on the disk) is running out of space and disallowed further writing? That being said, MAGUS allows you to start the same run and reuse existing files, so it should be able to pick up where it left off. Does the problem persist if you do this?

macelik commented 2 years ago

Nope, I have always deleted the output directory and then restart it again just in case. :) I ll give it shot to this too. There is plenty of disk space. I thought maybe windows is going to sleep mode, I have checked thats not the case either.

Oh another thing is, I watch the processors on htop and sometimes all threads go empty and MAGUS just hangs there at a step without doing anything and then I just press Enter on the command window and then it proceeds very quickly to do the tasks. I do not know if this is WSL thing or something. But again, it goes all smooth with other files (tried it with ~10k sequences) with no problem.

vlasmirnov commented 2 years ago

Yeah, I did my best to make it restarteable for massive runs, especially for cases like this. Problems might happen if the process is hard-killed before it has a chance to update the task files (where it records the things its working on/waiting on). If a restarted run seems to hang, it might be useful to delete the task files and rerun it, so it looks for existing files with no assumptions.

Also for large but easy (low sequence heterogeneity) alignments, it might be worth increasing the number of subsets --maxnumsubsets (I think the default is 25, and you could bump it up to 100). That might make it go faster on easy alignments. On hard alignments, that will slow it down.

I did most of my testing on WSL as well, so its weird that it's behaving like that. Let me know if problems persist, and I'll do my best to figure out what's going on.

macelik commented 2 years ago

Oh hello there, So I opened WSL2 with admin priv and run MAGUS, it threw a different error besides the one I mentioned above. I have added the log.txt in here.

I have followed what you said, I restart the MAGUS with the same command and now seems fine (has been been almost 7 hours and seems ok).

I would like to ask something, previously when I aligned the file with clustalo the sequences in aligned output went to over 6k in length while my average unaligned seq lengths were 1250, and with mafft it was even worse it extend it to over 30k. Clearly, there were sequences that were screwing the whole alignment. I tried to inspect and see but when you have 600k sequences it becomes hard. :)) I think MAGUS comes very handy in these kinda situations, we can easily inspect the lengths in subalignment directory in sets and that would give us an idea. I am hoping that would be really useful. Would you agree or you would advice a different approach?

vlasmirnov commented 2 years ago

Yeah, that's a good question. You could try inspecting the subset alignments, as well as the backbone alignments (if the backbones are long, it will produce a long merged alignment). That should identify the culprit. We've been working on some additional functionality to deal with sequence length heterogeneity by choosing backbones in a smarter way, but that's still in development. Let me know if that helps, and how your experiment goes.