acryldata / datahub-actions

DataHub Actions is a framework for responding to changes to your DataHub Metadata Graph in real time.
42 stars 47 forks source link

feat(actions): Adding support for MCL Timeseries #120

Closed jjoyce0510 closed 4 months ago

jjoyce0510 commented 4 months ago

Summary

In this PR, we add support for wiring in MCL Timeseries topic to datahub actions.

QA

Verified that timeseries events are streaming through using the hello_world action when enabled.

{
    "event_type": "MetadataChangeLogEvent_v1",
    "event": {
        "entityType": "assertion",
        "entityUrn": "urn:li:assertion:358c683782c93c2fc2bd4bdd4fdb0153",
        "changeType": "UPSERT",
        "aspectName": "assertionRunEvent",
        "aspect": {
            "value": "{\"result\":{\"type\":\"SUCCESS\",\"nativeResults\":{}},\"assertionUrn\":\"urn:li:assertion:358c683782c93c2fc2bd4bdd4fdb0153\",\"timestampMillis\":1675155843000,\"asserteeUrn\":\"urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)\",\"runId\":\"2021-12-28T12:00:00Z\",\"partitionSpec\":{\"type\":\"PARTITION\",\"partition\":\"{\\\"category\\\": \\\"catA\\\"}\"},\"batchSpec\":{\"limit\":10,\"nativeBatchId\":\"c8f12129f2e57412eee5fb8656154d05\",\"customProperties\":{\"data_asset_name\":\"data__foo1__asset\",\"datasource_name\":\"my_hive_datasource\"}},\"status\":\"COMPLETE\"}",
            "contentType": "application/json"
        },
        "systemMetadata": {
            "lastObserved": 1719854699981,
            "runId": "file-2024_07_01-10_24_59",
            "lastRunId": "no-run-id-provided"
        },
        "created": {
            "time": 1719854700775,
            "actor": "urn:li:corpuser:__datahub_system"
        }
    },
    "meta": {
        "kafka": {
            "topic": "MetadataChangeLog_Timeseries_v1",
            "offset": 8,
            "partition": 0
        }
    }
}
github-actions[bot] commented 4 months ago

Unit Test Results (build & test)

63 tests  ±0   63 :heavy_check_mark: ±0   3s :stopwatch: ±0s   1 suites ±0     0 :zzz: ±0    1 files   ±0     0 :x: ±0 

Results for commit 71ef03c0. ± Comparison against base commit 54a568b5.