MI-DPLA / combine

Combine /kämˌbīn/ - Metadata Aggregator Platform
MIT License
26 stars 11 forks source link

Create "valves" for input job(s), filtering all / valid / invalid records to be used #137

Closed ghukill closed 6 years ago

ghukill commented 6 years ago

Related to #119

Allowing the user to select all, invalid, or valid records from an input job, opens up a handful of useful workflows.

If a Job contains a large number of records, but a small number fail validation, it would be beneficial to allow a TransformationJob to operate over only the fail records, and then use a MergeJob to recombine. In some situations, this may be required, where a Transformation would alter valid records in an undesirable way.

The following two scenarios outline how the use of all / invalid / valid "valves" could be used:

img_20180330_123548

img_20180330_123540

First step will be to allow these valves on input jobs, then look into how the UI might help users merge the disparate pieces back together (which could, of course, be done manually).

ghukill commented 6 years ago

Consider, also, whether these "valves" should be at the Record Validation Scenario (RVS) level. Could have toggles for each RVS, where toggling "on" would mean to include. Default could be to all, or none. Noting here, but likely better to consider once in place.

ghukill commented 6 years ago

Mostly done. Closing this issue, preparing to merge with dev. Below is a graph showing a scenario of how this might help, where the Transform --> Merge Job.

screen shot 2018-04-02 at 12 43 24 pm

ghukill commented 6 years ago

Reopening - work on GUI for this addition still ongoing.

Todo

ghukill commented 6 years ago

Adding counts to edges and tables based on record validity

screen shot 2018-04-03 at 10 15 16 am

screen shot 2018-04-03 at 10 15 26 am

ghukill commented 6 years ago

Done.