search

I2PC / scipion

Scipion is an image processing framework to obtain 3D models of macromolecular complexes using Electron Microscopy (3DEM)
http://scipion.i2pc.es
Other
76 stars 47 forks source link

'scipion - join sets' fails when data comes from two separate imports #2066

Closed JuhaHuiskonen closed 4 years ago

JuhaHuiskonen commented 4 years ago

I am combining data arising from two separate 'import - movies'. I have processed everything separately until now. When I combine two SetOfParticels using 'scipion - join sets' and try to run 'relion - 3D auto-refine' I get the following error:

00029: File "/projappl/project_2001566/apps/scipion/2.0/software/lib/python2.7/site-packages/relion/convert/convert.py", line 757, in convertBinaryFiles 00030: pw.utils.createLink(rootDir, outputRoot) 00031: File "/projappl/project_2001566/apps/scipion/2.0/pyworkflow/utils/path.py", line 250, in createLink 00032: relsource = join(relpath(sourcedir, destdir), basename(source)) 00033: File "/projappl/project_2001566/apps/scipion/2.0/software/lib/python2.7/posixpath.py", line 428, in relpath 00034: raise ValueError("no path specified") 00035: ValueError: no path specified 00036: Protocol failed: no path specified 00037: FAILED: convertInputStep, step 1

JuhaHuiskonen commented 4 years ago

update: even joining sets of particles that come from a single import function results in the same error in 'relion - 3D auto-refine'

pconesa commented 4 years ago

Thanks @JuhaHuiskonen , do the "output of the import protocols" look ok? Are images shown?

JuhaHuiskonen commented 4 years ago

Yes, everything else is normal to this point:I have imported a set of movies, and done all the typical steps (motion correction, CTF estimation, particle picking). As I had 3M coordinates after picking, I used 'scipion - split sets' to split it in 3 sets (this was the only way for me to continue as there were other issues with large sets). I kept processing (2D and 3D classification) these separately. Now I am joining particles with 'scipion - join sets'. In this new set I can see the particle images and as far as I can see the metadata is also ok. For some reason the conversion of this database to Relion seems to fail.

pconesa commented 4 years ago

uumm! I see ....we might need some input here from relion guys. @delarosatrevin @azazellochg

delarosatrevin commented 4 years ago

Thanks for reporting @JuhaHuiskonen. I think I know the root of this issue. My guess is that I introduce it when I changed how the symbolic links were generated. Before there was a symlink for each file, but these generate a lot of links (entries in the filesystem) that are really bad for parallel filesystems. So I changed to link only the root folder, but there seems to be not working in this case where binaries come from different locations. I will take a look as soon as possible.

I'm sorry for the inconveniences.

JuhaHuiskonen commented 4 years ago

Hi @delarosatrevin, that would make sense and nobody likes millions of symlinks.. But do you have a commit where this was introduced so that we could go back to the old way for now?

delarosatrevin commented 4 years ago

@JuhaHuiskonen tomorrow morning my first task will be checking this issue. Please let me try first to find a fix and if by tomorrow I'm not done, I will let you know the commit, it might also contain other changes that maybe is not that easy to revert only this part.

delarosatrevin commented 4 years ago

I'm closing this here as I already opened the issue in the scipion-em-relion repo

pconesa commented 4 years ago

@delarosatrevin , @JuhaHuiskonen thank you both.