Add delete detection with switch for logical delete or unregister in iRODS catalog

[masilamani]

Currently, this ingest supports for detecting only new and updated files from source directory and syncs to target collection in iRODS.

Similarly, Add delete detection with switch for logical delete or unregister in iRODS catalog . Based on the switch provided:

• if -- unregister false, then set filesystem::deleted to Y

• if -- unregister true, then unregister it from iRODS catalog

This can be optimized to compare the source directory with Redis cache itself if needed and then sync the changes to iRODS catalog.

[mathob]

I would like to see this enhancement implemented.

[trel]

This would be similar to the --delete option of rsync.

This will require gathering a manifest from iRODS prior to comparing against the source tree being ingested.

The ingest tool does not currently know how to 'scan' an iRODS 'source', which would be a first step in implementing this feature. We have abstracted the 'scanner', so some of this groundwork has been laid in https://github.com/irods/irods_capability_automated_ingest/issues/207.

[trel]

Rather than worrying about 'scanning' iRODS...

Maybe the initial worker, just prior to running the scandir on the source, can ALSO 1) grab a full recursive listing from iRODS of the destination logical path... 2) and while walking the source, remove any 'seen' files from the destination listing... 3) and then when the scandir is complete... any files (and directories?) still in the destination listing should be enqueued as 'deletes'...

We would need:

• new --delete flag
• new --event_handler method ... pre_data_obj_delete
• new --event_handler method ... post_data_obj_delete
• new --event_handler method ... pre_coll_delete ?
• new --event_handler method ... post_coll_delete ?

[trel]

Some additional notes...

• We've determined that empty source directories are created in iRODS when scanning filesystems (not S3). Therefore, it would be cleanest / expected for the --delete option to 'clean them up' as well. This suggests that all three question marks in the above comment... should happen.

• There is more code hidden at the end of 3) ... we need to implement the delete functions (using PRC) that would be enqueued.

• Need tests

• Recursive deletes for the iRODS side would ... not call pre/post_data_obj_delete since they would be 'hidden' by the recursive call. also... slower? Needs more discussion... how do we know what is 'down' there... hao did some math.

• scandir only sees one level deep a time (and then we run scandir on subdirectories) - so our 'recursive listing' of iRODS would actually only need to return the listings immediately 'below' the named destination logical path. This would mean our memory footprint per task worker would be roughly 'scandir footprint' + ('number of data-objects and sub-collections' x 'length of logical path')

• implementation detail - think about a good data structure for destination information for quick interrogation and deletion

• possibly leverage the catalog with UNION/SET database math rather than just COLL_NAME LIKE %

• delete could mean... unregister / trash / force ? how to expose those?

  • might want to tie these to certain ingest options (register vs put)?

• perhaps --delete should be 'separate' from getting things INTO iRODS... would not have to interlace ingesting data with also deleting data from iRODS.

  • would still have to scandir and compare... but no chance of worrying about register/force/put being confusing/interfering

[trel]

some additional additional notes...

• to keep the priorities separated... we will need two new queues (del_file and del_dir ... or better names)
  • but they fire 'after' our existing queues in terms of priority, we want them last
  • so the priority would be [restart, path, file, del_file, del_dir]

---

And so I think this means we need either

• A) new operation modes (with event handler method), or
• B) just a new event handler method

A1) UNREGISTER_SYNC DELETE_SYNC trash=[yes, no], default=yes

A2) UNREGISTER_SYNC REMOVE_SYNC trash=[yes, no], default=yes

B) PUT_SYNC delete=[trash, no-trash], default=trash REGISTER_SYNC delete=[unregister], default=unregister

Assuming B is possible (recursive delete being the puzzle)... I think B is more elegant... fewer moving parts / new names / new code.

[trel]

two things today...

• there are no priorities to queues in celery - so ignore the prior thoughts around that.

• the below plan for enqueuing a recursive delete

---

today's initial experiment at 'recursive delete via celery' can cleanly enqueue deletions of all data objects recursively under a target path (no ordering issues / collisions).

• with 50 objects per task, this took 45% of the time of serial irm -rf ( more than a 2x speedup )

after removing all the data objects, we are left with a tree of empty collections in iRODS.

current best idea to remove the tree of empty collections is to also...

enqueue a task that runs a query against the iRODS database to find any data objects below the target collection (which are concurrently being removed by earlier enqueued tasks)

• select count(DATA_ID) where COLL_NAME like '$target%'
• if > 0 found, re-enqueue same task, and exit (no sleeps, just fire when ready)
• if == 0, then enqueue deletes of all the subdirectories
  • run a query to get this listing (could be a large number... 10000)
  • select COLL_NAME where COLL_NAME like '$target%'
  • sort them from depth... count the /s? or genquery can do it?
  • enqueue one task, with ALL the recursive found sub-collections
    • from the bottom (by depth)
    • serially in that single task
    • would hit server policy for every subcollection removed (which is good/desired)

OR

• if == 0, then enqueue a single recursive delete of the top-level
  • which ... will not fire policy on server
    • could be an option/knob for the ingest tool
    • policy vs speed - but wouldn't be so much speed
  • or server pre-PEP on recursive remove to 'inspect' and report
    • or fire other policy for each of those...
    • seems like a lot of work on the admin

[alanking]

I have a basic implementation working for this in a branch.

I've run into one difficulty that we have decided will not be handled in the initial effort. Any paths which have unencodeable characters (i.e. UnicodeEncodeError) or use the character_map event handler method cannot use this feature because the path construction logic is proving difficult to integrate. I've created an issue for this for handling at a later date: https://github.com/irods/irods_capability_automated_ingest/issues/261