Refactor/deduplication of alignment utils

alisterburt commented 2 years ago

Sorry @EuanPyle this is quite a large PR

main things are

deduplication of IMOD/AreTomo utils (single source of truth is the best)
added a new cli program for writing output which will be called by RELION

edit: I also messed up merging your PR without some changes I wanted so have included its commits here

alisterburt commented 2 years ago

if you could give it a once over that would be great!

EuanPyle commented 2 years ago

No problem! Will do in a min and run a few tests :)

EuanPyle commented 2 years ago

There is a problem somewhere: my AreTomo TSA stopped during the final TS and never wrote out the aligned_tilt_series.star. Weridly, rln counted this as a SUCCESS etc. job although I did get an error message about manually marking it as failed. TS and alignments remain in the external/ folder. I'll look in to what's going wrong!

EDIT: Think it's looking in the tilt_series folder for .star files rather than external/tilt_series, will eat lunch then think about it ;)

EDIT2: It's definitely a directory structure issue. Currently the alignment data is being placed in job???/external/TS_Name/external/TS_Name/

EuanPyle commented 2 years ago

Found the problem, it's aretomo specific. Will comment where it's gone wrong

alisterburt commented 2 years ago

@EuanPyle this was the only issue you came across? am I good to merge here now? :)

EuanPyle commented 2 years ago

Sorry, not yet! For some reason it keeps crashing in the middle of AreTomo. This may be my computer though! I'm going to restart it and re-run, can I let you know in 20 mins?

alisterburt commented 2 years ago

sure!

EuanPyle commented 2 years ago

Sorry, little computer issues, everythings going at a snails pace (that's being generous)

alisterburt commented 2 years ago

same here, no stress!!

EuanPyle commented 2 years ago

There's another issue, I think in the writing the final aligned_tilt_series.star as the error occurs immediately after the alignment of the final TS is finished. The error says MetaDataTable::read: File TiltSeriesAlign/job012/tilt_series/TS_001.star does not exist.

it's looking for the individual star files in the tilt_series folder rather than the alignment ones. Will have a gander to find it in the code, if it's not in our code it may be in rln? Not sure.

EuanPyle commented 2 years ago

lambda tilt_series_id: output_directory / 'tilt_series' / f'{tilt_series_id}.star'

in aretomo/_utils.py needs to be:

lambda tilt_series_id: output_directory / 'alignments' / f'{tilt_series_id}.star'

alisterburt commented 2 years ago

Changed back to writing files out into 'tilt-series' at Sjors' request (makes his life easier in RELION) - I think this version should now work with RELION (definitely does for IMOD, aretomo utils seem to be pointing to the right place)

alisterburt commented 2 years ago

I'm gonna merge here and we can iterate - thanks @EuanPyle !

EuanPyle commented 2 years ago

Sweet, will run a quick test on AreTomo and call it a day! Nice one!

alisterburt / tomo_preprocessing

Refactor/deduplication of alignment utils #34