Closed mlandth closed 5 years ago
The Problem is that we completely ignore the in and out BlueprintRefs in the Backend when whe assemble the filters in the PipelineSupervisor. I'm on it.
Yes that is true. We completely ignore the new concept of in- and out-ports. I thought we could defer it until we re-build that mechanism to support branching and merging.
I think there is a point where we lost the order of stages - e.g. construction of stages is now asynchronous. Maybe you can do a quick fix to keep the order. Do as less as possible. As i said these parts get re-build anyway.
Ok. The problem is clear: We start a BlueprintMaterializer for each Stage. They finish their jobs in any order. And we just append their results in the order they finish.
That was my fault. 😞
Suggestion: Rewrite the withFilterStage function of the Pipeline class and sort the stages by their Ref/UUID. You get the right order from the PipelineBlueprint of the Pipeline.
Hey @mlandth let me know whether you could solve/fix the problem for now.
@kKdH For now this quickfix is working and all tests are passing until we rebuid the whole mechanism. Closing this issue.
I started a new IntegrationTest (WorkflowTest) to check actual filter operations in a pipeline. The WorkflowTest should build the following Pipeline: KafkaSource -> RetainFields -> RemoveFields -> RemoveFields -> KafkaSink
I'm inserting 3 Datasets before the RetainFields Filter. Each Dataset contains one Record with 7 Fields. (3x Text, 2x Number, 1x Boolean, 1x Health)
After the second RemoveFields Filter i am now extracting the Datasets and check that
Now to the Problem:
Sometimes the test passes. But for ~70% the test breaks with two different failures:
OR
I added some logs in the RetainFieldsFilterLogic and in the RemoveFieldsFilterLogic:
Order of onPull() logs from the passing test:
One order of onPull() logs from the failing test (empty list of datasets):
Is it possible that we ignore the order of filters when we create a Pipeline?
@kKdH Houston, we have a problem.