search

logbee / keyscore

Apache License 2.0
3 stars 0 forks source link

Strange behaviour in the Integration Test #50

Closed EndallBatan93 closed 5 years ago

EndallBatan93 commented 5 years ago

When inserting 2 Datasets in the first Pipeline, the expected count of extracted Datasets of the filter of the second Pipeline is 2 - but is actually 3. The problem does not extend to the actual dataflow in the pipeline. Only the extraction of datasets from a filter is not as expected.

EndallBatan93 commented 5 years ago

Definetly related to drain or pause because the correct number of datasets is inserted/extracted when inserting the datasets manually with rested into an running filter.

kKdH commented 5 years ago

Have you investigate when it happens that a datasets leaks? When the valve is already drained or when you open the valve again?

mlandth commented 5 years ago

After some further research i think that the drain function does not work as expected.

When you only pause and unpause the filter in the PipelineIntegrationTest, everything works as expected. Only when you activate the drain function (it doesn't matter if the filter was paused before or not) the PIT fails. When two datasets are inserted, one dataset passes even tough drain is activated.

mlandth commented 5 years ago

Ignoring how many datasets you insert in a "drained" filter, the first one always passes through the pipeline. All other datasets coming after the first one are actually drained.

mlandth commented 5 years ago

According to some enhanced logs, i think i found the problem.

[14:00.315UTC] Valve <374d089b-d33a-42a0-ad7d-481e60068e52> does now drain.

[14:00.404UTC] Inserting 2 datasets into valve <c200b0de-293d-4eac-82a5-b6f026fb07cc>
[14:00.404UTC] <c200b0de-293d-4eac-82a5-b6f026fb07cc> pushing out: Some(...)
[14:00.413UTC] Inserted 2 datasets into valve <c200b0de-293d-4eac-82a5-b6f026fb07cc>

[14:00.415UTC] <374d089b-d33a-42a0-ad7d-481e60068e52> pushing out: Some(...)

[14:00.514UTC] <374d089b-d33a-42a0-ad7d-481e60068e52> isDraining: true

Here are the logs of the InValve (c200b0de) and the OutValve (374d089b). The InValve is pushing out a dataset still before his state was updated and - more importantly - pushing out the first dataset before the drain parameter affects the behaviour of the OutValve.