search

I2PC / scipion

Scipion is an image processing framework to obtain 3D models of macromolecular complexes using Electron Microscopy (3DEM)
http://scipion.i2pc.es
Other
76 stars 47 forks source link

Clean-up feature for Scipion projects #2078

Open JuhaHuiskonen opened 4 years ago

JuhaHuiskonen commented 4 years ago

The file number in large projects can become hard to manage. It would make sense to have a feature to delete those files that are not registered as outputs required by Scipion and that are not generally needed.

Each protocol could define files:

The user could then select all the protocols they want to clean and select either of the two options (quick or full).

pconesa commented 4 years ago

Thanks Juga. We clearly need to do something. What I'm not sure is if it's more hassle for the developers to flag all possible output files or just delete those that are not final, like all iteration files in iterative protocols.

El mié., 6 may. 2020 16:39, Juha Huiskonen notifications@github.com escribió:

The file number in large projects can become hard to manage. It would make sense to have a feature to delete those files that are not registered as outputs required by Scipion and that are not generally needed.

Each protocol could define files:

  • that are necessary for Scipion workflow (will never be deleted)
  • files that are useful in some cases outside of Scipion but not used in the workflow (deleted only if 'full' clean is selected, e.g. those diagnostic files that can't be visualised by Scipion)
  • intermediate files that are very rarely needed (deleted if 'quick' or 'full' clean is selected, e.g. iterations 2...N-1)

The user could then select all the protocols they want to clean and select either of the two options (quick or full).

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/I2PC/scipion/issues/2078, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7ZYIL2ATS644MQDEOSF3RQFZCXANCNFSM4M2P7XPA .

JuhaHuiskonen commented 4 years ago

The protocols that increase the file count the most seem to be RELION's particle polishing and CTF refinement (especially in projects with many many movies - as these protocols create several diagnostic files for each movie). Refinement and classification runs produces significantly fewer files.

pconesa commented 4 years ago

@azazellochg , @delarosatrevin , It might be a good time now maybe to review the amount of files relion keeps. If you have the chance, rethink if files kept are really useful for a regular user.

I'm mention here @dmaluenda to be aware of this for xmipp methods.

delarosatrevin commented 4 years ago

Yes, I plan to have a look at it...but maybe it would be good time to define one cleanUp method in base Protocol class...and make then actionable from the GUI. So it is ready for when each protocol decides to implement it. I have some ideas for a general implementation. We could discuss about it.

On Thu, May 7, 2020 at 8:21 PM Pablo Conesa notifications@github.com wrote:

@azazellochg https://github.com/azazellochg , @delarosatrevin https://github.com/delarosatrevin , It might be a good time now maybe to review the amount of files relion keeps. If you have the chance, rethink if files kept are really useful for a regular user.

I'm mention here @dmaluenda https://github.com/dmaluenda to be aware of this for xmipp methods.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/I2PC/scipion/issues/2078#issuecomment-625419662, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACAM6WG3OMFHCKNUFMHECQTRQL33HANCNFSM4M2P7XPA .

pconesa commented 4 years ago

Anything that will make life easy for developers is welcome.

On 7/5/20 20:57, Jose Miguel de la Rosa Trevin wrote:

Yes, I plan to have a look at it...but maybe it would be good time to define one cleanUp method in base Protocol class...and make then actionable from the GUI. So it is ready for when each protocol decides to implement it. I have some ideas for a general implementation. We could discuss about it.

On Thu, May 7, 2020 at 8:21 PM Pablo Conesa notifications@github.com wrote:

@azazellochg https://github.com/azazellochg , @delarosatrevin https://github.com/delarosatrevin , It might be a good time now maybe to review the amount of files relion keeps. If you have the chance, rethink if files kept are really useful for a regular user.

I'm mention here @dmaluenda https://github.com/dmaluenda to be aware of this for xmipp methods.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/I2PC/scipion/issues/2078#issuecomment-625419662, or unsubscribe

https://github.com/notifications/unsubscribe-auth/ACAM6WG3OMFHCKNUFMHECQTRQL33HANCNFSM4M2P7XPA .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/I2PC/scipion/issues/2078#issuecomment-625437010, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAF7ZYLWRT7JYYNUJNRBUXDRQMABJANCNFSM4M2P7XPA.