Null pointer error on job event for static lineage

davidsharp7 commented 1 week ago

Problem

Emitting a JobEvent with input and/or output datasets causes a HTTP500 error in the API, which results from a nullPointerException in Marquez.

Closes: #2925

Solution

Make runuuid/Lineage event type nullable

on

insertInputDatasetFacetsFor insertOutputDatasetFacetsFor

Nullable

Following post which used to fail now posts correctly.

r=requests.post(url='http://localhost:8080/api/v1/lineage', 
                json={"eventTime": "2024-11-11T20:03:30.112534+00:00", 
                      "inputs": [{"facets": {}, 
                                  "inputFacets": {}, 
                                  "name": "users", 
                                  "namespace": "openlineage_experiments.py_static_tests"}], 
                      "job": {"facets": {"sql": {"_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.22.0/client/python", 
                                                 "_schemaURL": "https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/SqlJobFacet", 
                                                 "query": "SELECT * FROM users JOIN orders ON users.id = orders.user_id"}}, 
                              "name": "xjob_with_inputs_and_outputs", 
                              "namespace": "openlineage_experiments.py_static_tests"}, 
                      "outputs": [{"facets": {}, "name": "orders", 
                                   "namespace": "openlineage_experiments.py_static_tests", 
                                   "outputFacets": {}}], 
                      "producer": "https://github.com/openlineage-user", 
                      "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/JobEvent"})

One-line summary:

Make runuuid/Lineage event type nullable

Checklist

[x] You've signed-off your work
[x] Your changes are accompanied by tests (if relevant)
[x] Your change contains a small diff and is self-contained
[x] You've updated any relevant documentation (if relevant)
[x] You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
[x] You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
[x] You've included a header in any source code files (if relevant)

netlify[bot] commented 1 week ago

Deploy Preview for peppy-sprite-186812 ready!

Name	Link
Latest commit	4ccc0b8ea1a87b390998c8e689d41b07162b2485
Latest deploy log	https://app.netlify.com/sites/peppy-sprite-186812/deploys/673457bf7ea2be0008244786
Deploy Preview	https://deploy-preview-2974--peppy-sprite-186812.netlify.app
Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

codecov[bot] commented 1 week ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 81.18%. Comparing base (41d9d3d) to head (4ccc0b8).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #2974 +/- ## ============================================ + Coverage 81.02% 81.18% +0.16% - Complexity 241 1506 +1265 ============================================ Files 43 268 +225 Lines 996 7356 +6360 Branches 40 325 +285 ============================================ + Hits 807 5972 +5165 - Misses 160 1226 +1066 - Partials 29 158 +129 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

MarquezProject / marquez