search

irods / irods_capability_automated_ingest

Other
12 stars 15 forks source link

Add ability for "PEP" event_handler methods to abort operation for a single path #184

Open alanking opened 1 year ago

alanking commented 1 year ago

The user should be allowed to abort the operation for any reason and define next steps for the ingest framework as part of the event_handler "PEP" implementations. Currently, if an Exception is raised from an event_handler method like pre_data_obj_create, the entire file chunk will fail and be placed in the retry queue.

This could take many forms. Our initial idea was to raise a custom Exception which contains information for the plugin framework for how to handle the file which is being aborted. The most obvious use case to us would be to create a new task with just that file that is immediately placed on the fail queue or the retry queue, and then the framework would continue to ingest the rest of the files in the current task.

This would serve as an alternative implementation for options such as --exclude_file_name which is not currently possible for this framework and may have other applications which we do not know about.

d-w-moore commented 1 year ago

This would serve as an alternative implementation for options such as --exclude_file_name which is not currently possible for this framework

What currently prevents --exclude_file_name from working in the ingest tool?

alanking commented 1 year ago

@d-w-moore - Please see https://github.com/irods/irods_capability_automated_ingest/pull/182