I want to write some tests for my current project: It sources data from an external Postgres DB and also from a flat file from S3. I don’t want to run Postgres DB in my simple test env nor do I want to connect to S3 - this is all well, I can just use the PDI datasets plugin. However, if I am not missing something, currently it is not possible test everything on a job level. So imagine I want to test the whole process, just replace the initial input data sourceS and final output data targets, nothing in between. There’s like 30 or so transformations for this project and maybe 10 jobs, all coordinated by one master job.
I just want to simulate at the bare minimum:
Cutover run
First delta run
Second delta run
So to summarise: It’s more like blackbox testing ... provide the whole process some input, let it do all the stuff in between it is meant to do (Jobs and transformations) and then compare the output it generated with the golden dataset.
It would also be convenient to trigger all this from the command line.
I want to write some tests for my current project: It sources data from an external Postgres DB and also from a flat file from S3. I don’t want to run Postgres DB in my simple test env nor do I want to connect to S3 - this is all well, I can just use the PDI datasets plugin. However, if I am not missing something, currently it is not possible test everything on a job level. So imagine I want to test the whole process, just replace the initial input data sourceS and final output data targets, nothing in between. There’s like 30 or so transformations for this project and maybe 10 jobs, all coordinated by one master job.
I just want to simulate at the bare minimum:
So to summarise: It’s more like blackbox testing ... provide the whole process some input, let it do all the stuff in between it is meant to do (Jobs and transformations) and then compare the output it generated with the golden dataset.
It would also be convenient to trigger all this from the command line.