Open masilamani opened 6 years ago
I would like to see this enhancement implemented.
This would be similar to the --delete
option of rsync
.
This will require gathering a manifest from iRODS prior to comparing against the source tree being ingested.
The ingest tool does not currently know how to 'scan' an iRODS 'source', which would be a first step in implementing this feature. We have abstracted the 'scanner', so some of this groundwork has been laid in https://github.com/irods/irods_capability_automated_ingest/issues/207.
Rather than worrying about 'scanning' iRODS...
Maybe the initial worker, just prior to running the scandir
on the source, can ALSO
1) grab a full recursive listing from iRODS of the destination logical path...
2) and while walking the source, remove any 'seen' files from the destination listing...
3) and then when the scandir
is complete... any files (and directories?) still in the destination listing should be enqueued as 'deletes'...
We would need:
pre_data_obj_delete
post_data_obj_delete
pre_coll_delete
?post_coll_delete
?Some additional notes...
We've determined that empty source directories are created in iRODS when scanning filesystems (not S3). Therefore, it would be cleanest / expected for the --delete
option to 'clean them up' as well. This suggests that all three question marks in the above comment... should happen.
There is more code hidden at the end of 3) ... we need to implement the delete functions (using PRC) that would be enqueued.
Need tests
Recursive deletes for the iRODS side would ... not call pre/post_data_obj_delete since they would be 'hidden' by the recursive call. also... slower? Needs more discussion... how do we know what is 'down' there... hao did some math.
scandir
only sees one level deep a time (and then we run scandir
on subdirectories) - so our 'recursive listing' of iRODS would actually only need to return the listings immediately 'below' the named destination logical path. This would mean our memory footprint per task worker would be roughly 'scandir footprint' + ('number of data-objects and sub-collections' x 'length of logical path')
implementation detail - think about a good data structure for destination information for quick interrogation and deletion
possibly leverage the catalog with UNION/SET database math rather than just COLL_NAME LIKE %
delete could mean... unregister / trash / force ? how to expose those?
perhaps --delete
should be 'separate' from getting things INTO iRODS... would not have to interlace ingesting data with also deleting data from iRODS.
scandir
and compare... but no chance of worrying about register/force/put being confusing/interferingsome additional additional notes...
And so I think this means we need either
A1) UNREGISTER_SYNC DELETE_SYNC trash=[yes, no], default=yes
A2) UNREGISTER_SYNC REMOVE_SYNC trash=[yes, no], default=yes
B) PUT_SYNC delete=[trash, no-trash], default=trash REGISTER_SYNC delete=[unregister], default=unregister
Assuming B is possible (recursive delete being the puzzle)... I think B is more elegant... fewer moving parts / new names / new code.
two things today...
there are no priorities to queues in celery - so ignore the prior thoughts around that.
the below plan for enqueuing a recursive delete
today's initial experiment at 'recursive delete via celery' can cleanly enqueue deletions of all data objects recursively under a target path (no ordering issues / collisions).
irm -rf
( more than a 2x speedup )after removing all the data objects, we are left with a tree of empty collections in iRODS.
current best idea to remove the tree of empty collections is to also...
enqueue a task that runs a query against the iRODS database to find any data objects below the target collection (which are concurrently being removed by earlier enqueued tasks)
select count(DATA_ID) where COLL_NAME like '$target%'
select COLL_NAME where COLL_NAME like '$target%'
/
s? or genquery can do it?OR
I have a basic implementation working for this in a branch.
I've run into one difficulty that we have decided will not be handled in the initial effort. Any paths which have unencodeable characters (i.e. UnicodeEncodeError
) or use the character_map
event handler method cannot use this feature because the path construction logic is proving difficult to integrate. I've created an issue for this for handling at a later date: https://github.com/irods/irods_capability_automated_ingest/issues/261
Currently, this ingest supports for detecting only new and updated files from source directory and syncs to target collection in iRODS.
Similarly, Add delete detection with switch for logical delete or unregister in iRODS catalog . Based on the switch provided:
if -- unregister false, then set filesystem::deleted to Y
if -- unregister true, then unregister it from iRODS catalog
This can be optimized to compare the source directory with Redis cache itself if needed and then sync the changes to iRODS catalog.