TLDR; When emitting a Lineage Event for a Run that has older version (using the version facet) of input datasets, Marquez always use the latest version of the input datasets and it modifies the version facet of the input dataset to the previous version.
I removed some of the variables of the response payload so not to distract on the important bit. Notice how datasetA version 2 is now modified to version 1.
Notice how the version of the input dataset of 2nd Run points to the latest version of datasetA even though we explicitly specify version 1 when emitting the Lineage Event.
This is the step to replicate the issue reported on this slack discussion.
TLDR; When emitting a Lineage Event for a Run that has older version (using the version facet) of input datasets, Marquez always use the latest version of the input datasets and it modifies the version facet of the input dataset to the previous version.
Step to replicate can be found below
Creating datasetA version 1
Payload is below
1.create_run1_datasetA.json
Creating datasetB version 1
2.create_run1_datasetB.json
Notice that I want to explicitly use
version 1
ofdatasetA
Updating datasetA to version 2
3.create_run2_datasetA.json
Updating datasetB to version 2 but still using version 1 of dataset A
4.create_run2_datasetB.json
Querying datasetA versions
URL: http://localhost:3000/api/v1/namespaces/usa.california.sanjose/datasets/datasetA/versions
I removed some of the variables of the response payload so not to distract on the important bit. Notice how datasetA
version 2
is now modified toversion 1
.Querying the datasetB versions
Notice how the version of the input dataset of 2nd Run points to the latest version of datasetA even though we explicitly specify version 1 when emitting the Lineage Event.