search

mattcasters / pentaho-pdi-dataset

Set of PDI plugins to more easily work with data sets. We also want to provide unit testing capabilities through input data sets and golden data sets.
Apache License 2.0
30 stars 13 forks source link

Sorting #16

Closed cyberindy closed 7 years ago

cyberindy commented 7 years ago

I have many mappings/transformations where the order of rows is unpredictable - either by using switch/case steps or parallel streams. Unit tests for such transformations fail since the input rows and golden rows cannot be aligned properly. Sorting the datasets doesn't fix this, since the order of rows get mixed up at runtime.

I can work around this by inserting a sort step before each mapping output step. Since sorting is not required in the actual business logic though, I'd very much llike to save the overhead of sorting when the transformation is not executed as a unit test.

I think this issue could be solved if the unit test wouldn't sort the data sets but would insert an actual sort step into the generated unit test transformation (knowing that this implies unnecessary overhead for all other transformations with predictable row order).

Lastly: A big big thank you for your work on this wonderful plugin. I have long anticipated such functionality and am planning on using it heavily. This is definitely the way to go and takes ETL development to the right direction.

mattcasters commented 7 years ago

Thanks for this case cyberindy. The option to specify a sort order is now incorporated for all input and golden data sets. Really appreciate the feedback!