search

jlolling / talendcomp_tJobInstance

Talend User Components to work with the cimt framework for batch jobs. These components are developed as cooperation between cimt AG and Jan Lolling
Apache License 2.0
14 stars 7 forks source link

Add Ignore counter to job_instance_status and tJobInstanceStop #5

Closed mattywausb closed 12 months ago

mattywausb commented 7 years ago

I would like to measure the ignored rows. "Ingnored" means, the incoming information shall not be copied to the destination on purpose. Having this counter one can measure the completnes of the transformation by checking input = output + reject + ignore

jlolling commented 7 years ago

Perhaps we can do that, my problem here is mostly the broken compatibility to existing projects (keep in mind it runs in about 100 projects and not only in ours). We should break the compatibility only once but than with all changes which we expect. The table should also not become wide, it will be expensive to read otherwise.

mattywausb commented 7 years ago

This extention is backwards compatible in the way, that old jobs will still run on the new datamodel. So all you need to do is: deploy new version of datamodel and components.

mattywausb commented 7 years ago

Today I again stumbled in to the requirement to add ignore counts. Just to keep it in mind, I added this note.