Ingest: Reuse of captured rules

kristenwilson commented 9 years ago

Needs to cover both user selected reuse in Refine and auto-ingest reuse (perhaps via TSV).

Will likely overlap with #333. Spec is probably required.

sosguthorpe commented 8 years ago

@kristenwilson / @ostephens Can either/both of you clarify what is expected here as part of 6.0? Issue #333 suggests that the automation and suggestion would be as part of 6.x and I'm looking for some clarification of what this issue means in respect to this specific release.

ostephens commented 8 years ago

@sosguthorpe @kristenwilson @ianibbo Apologies for the lengthy reply:

In the original proposal around macros and reuse I made 12 proposals. Out of these I think the following have been implemented:

P1 Properties of a Macro
P3 Display Macro JSON
P4 Replace Macro via JSON
P11 Finding and applying Macros in OpenRefine

We will not implement P5-P8 as these were an alternative approach to what we implemented under P3 and P4.

This leaves:

P2 Save Macro on Ingest
P9 Automatic application of Macros
P10 Application of Macros when creating project in OpenRefine
P12 Reusing Rules across GOKb Projects

P12 was always an optional extra, so I think we can forget this for now (and if it is desired raise it as a separate work stream).

This leaves P2, P9 and P10. In the original specification I had the idea that the way this would work would be:

P2 would link a Macro to some high-level property of the project so that the next time the same/similar file was uploaded then P9 would apply the same macro to the incoming file. (In the spec I mention the use of the 'Source' property to link the macro to, but I think the 'Ingest Profile' might be closer to what I actually had in mind)
P9 would enable a new file to be retrieved based on the Ingest Profile (or equivalent), and would apply the Macro - and either ingest the file, or push it to a user for further work if ingest wasn't possible
P10 was essentially another way of applying a common set of transforms to a known quantity

Having discussed this with @sosguthorpe, and some of the comments from @ianibbo regarding the way automating uploads would work I think now....

P10 could be deprecated in favour of @sosguthorpe's idea of automatically applying transforms to a file when specific properties were noted - e.g. column names
P9 needs some slight rethinking to bring it in line with the other automated file ingest work stream, but I think the basics are still correct - that is:
- Some way of knowing when a file is pushed through into GOKb to apply a set of transformations (be they defined directly in the GOKb application, or be they via an OpenRefine Macro)
- Where the automated process doesn't result in an ingestable file, some mechanism to create an OpenRefine project and a Review Request for someone to pick up the file and do further work on it.

Which means the work for 6.0 looks something like:

Develop/implement mechanism to automatically apply OpenRefine transformations where this can be deduced based on existing rules/well known scenarios
- Automated application of transformations should happen when a file is checked in for the first time in OpenRefine, and could also be applied in the automated processing
Develop/implement a way of applying the automatic OpenRefine transformations, and any manually specified Macros, to a file that is going through the automated ingest process
Develop/implement mechanism to handle situations where the automated ingest process does not manage to ingest the file and further work needs to be done before the file can be successfully ingested

However - it may be that I'm off base here depending on what @sosguthorpe and @ianibbo are hoping to do regarding the automated ingest process. But that's my best shot :)

sosguthorpe commented 8 years ago

Thanks for this Owen. This is definitely not a small undertaking and I think this should be moved to a future release under the 6 major version number. I believe we should focus on the other issues under the 6.0 milestone to get as many of those in as we can for the next release.

kristenwilson commented 8 years ago

Thanks, Owen and Steve. I think Owen's proposal sounds good in general. I will put this on our agenda for Friday to see if there's any additional discussion needed.

Steve, we had agreed to break the 6.X work down into two releases -- 6.0 and 6.1. I took an initial stab at sorting issues into the 6.0 release, but Ian had said that you two would look at it and move things around based on time, dependencies, etc. So feel free to keep doing that if it makes sense to you. We will also check in on this on Friday.

sosguthorpe commented 8 years ago

OK. No problem.

On 02/12/15 16:03, kristenwilson wrote:

Thanks, Owen and Steve. I think Owen's proposal sounds good in general. I will put this on our agenda for Friday to see if there's any additional discussion needed.

Steve, we had agreed to break the 6.X work down into two releases -- 6.0 and 6.1. I took an initial stab at sorting issues into the 6.0 release, but Ian had said that you two would look at it and move things around based on time, dependencies, etc. So feel free to keep doing that if it makes sense to you. We will also check in on this on Friday.

— Reply to this email directly or view it on GitHub https://github.com/k-int/gokb-phase1/issues/448#issuecomment-161345668.

kristenwilson commented 8 years ago

Closing this and adding a new issue for the rules work for the bulk data loader.

k-int / gokb-phase1

Ingest: Reuse of captured rules #448