servoz
commented
1 year ago Ok I looked at this ticket.
In fact, Project.cleanup_orphan_history() (and more exactly Project.get_orphan_history()) does its job very well.
The issue comes when we use a project that has been renamed or copied.
This is because in these cases, the path values in COLLECTION_BRICK and COLLECTION_HISTORY tables, for data already calculated, correspond to the old paths before the project was renamed. As a result, when detecting orphans, as we use the detection of the beginning of the current path (the current directory of the project, renamed), or the existence of a data from its path, the old data are detected as orphans...
I tried a fix (67ba9782ae1) but it is not universal (it works only if the old copy of the project still exists, which is not good because if we save a project under another name, we may want to delete the original project).
Finally, I think the cleanest fix to this issue is to write a method that will parse the project and rename the beginning of the paths with the new project name. This method can be used at the end of the procedure to copy a project under another name (File > Save project as) or when loading a new project (File > Open project).
To do ASAP ...
Working on mia_processes.bricks.stat.spm.Level1Design(), we observe that all history information for data not involved in the current calculation is lost after the run. The pipeline:
We start with a project that already contains the result of a first calculation for another pipeline:
After the run for the Level1Design, we observe in the DataBrowser:
I don't have time to work on this at all at the moment. I open this ticket to keep in mind that we will have to work on ASAP. I suspect that the obsolete / orphan brick cleaning stage is not well adapted to this case, without having really had the time to look into it in detail.