search

scipion-em / scipion-em-motioncorr

Plugin to use Motioncor2 program within the Scipion framework
GNU General Public License v3.0
1 stars 0 forks source link

Working with scratch directory set #98

Open frozenfas opened 8 months ago

frozenfas commented 8 months ago

I am having a issue with motioncorr failing when I use a scratch directory.

I set the path to the scratch using the "SCIPION_SCRATCH" variable in the config file.

This means when running the motioncorr protocol it makes a symbolic link from tmp to the specified scratch and goes there to work, i believe. when specifying where to write the output mrc it uses a relative path like "../../extra/XXX". This leaves the output somewhere on the scratch and not in the scipion directory where the next step looks for it, resulting in a failure.

It could be I am not using the scratch correctly or maybe it is better to use a absolute path for the output?

azazellochg commented 8 months ago

@frozenfas SCIPION_SCRATCH is unfortunately not a thoroughly tested feature of Scipion. While what you are reporting is indeed a bug, however, I'm not sure of any speed-up of putting tmp on an SSD drive. This means that every movie will have to be copied first to SSD scratch and then the results will be copied back. These extra two transfers will remove any SSD speed advantage. This is why I suppose neither Cryosparc nor Relion offers such an option. In the case of particle stacks the situation is different. So, my suggestion would be not to use it for motion correction protocols.

frozenfas commented 8 months ago

Thanks, this does make sense. I had it turned on for other reasons and just remembered about it when I encountered the error here.

delarosatrevin commented 8 months ago

I have thought for some time about scratch at this point and it could be useful. But might require some extra communication between jobs. For example, the output from motioncor could be written to scratch. This may make it faster if working with a shared filesystem and save GPU time for computing. The result could be copied/moved by another thread not using the GPU. The micrograph in scratch could be reused for CTF estimation and particle extraction.

On Sun, Feb 25, 2024 at 9:23 AM Sean Connell @.***> wrote:

Thanks, this does make sense. I had it turned on for other reasons and just remembered about it when I encountered the error here.

— Reply to this email directly, view it on GitHub https://github.com/scipion-em/scipion-em-motioncorr/issues/98#issuecomment-1962973408, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACAM6WAFEJJY7VJEUN7QTUTYVNJOLAVCNFSM6AAAAABDXZ64U6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRSHE3TGNBQHA . You are receiving this because you are subscribed to this thread.Message ID: @.***>