linkedin / openhouse

Open Control Plane for Tables in Data Lakehouse
https://www.openhousedb.org/
BSD 2-Clause "Simplified" License
294 stars 50 forks source link

Handle unset execution conf in jobs service #114

Closed teamurko closed 4 months ago

teamurko commented 4 months ago

Summary

Fixed code to handle null execution conf field in JobConf

Changes

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

docker compose --profile with_jobs_scheduler run openhouse-jobs-scheduler -  --type SNAPSHOTS_EXPIRATION --cluster local --tablesURL http://openhouse-tables:8080 --jobsURL http://openhouse-jobs:8080
...
2024-06-03 17:52:53 2024-06-04 00:52:53 INFO  OtelConfig:70 - initializing open-telemetry sdk
2024-06-03 17:52:54 2024-06-04 00:52:54 INFO  Reflections:219 - Reflections took 153 ms to scan 1 urls, producing 4 keys and 10 values
2024-06-03 17:52:54 2024-06-04 00:52:54 INFO  JobsScheduler:108 - Starting scheduler
2024-06-03 17:52:54 2024-06-04 00:52:54 INFO  JobsScheduler:130 - Fetching task list based on the job type: SNAPSHOTS_EXPIRATION
2024-06-03 17:53:01 2024-06-04 00:53:01 INFO  OperationTasksBuilder:29 - metadata: dbName: db, tableName: test, creator: openhouse
2024-06-03 17:53:01 2024-06-04 00:53:01 INFO  JobsScheduler:140 - Submitting and running 1 jobs based on the job type: SNAPSHOTS_EXPIRATION
2024-06-03 17:53:01 2024-06-04 00:53:01 INFO  OperationTask:71 - Launching job for dbName: db, tableName: test, creator: openhouse
2024-06-03 17:53:06 2024-06-04 00:53:06 INFO  OperationTask:97 - Launched a job with id SNAPSHOTS_EXPIRATION_db_test_15c49365-2bb3-4ba6-89ae-5ce396ad67c0 for dbName: db, tableName: test, creator: openhouse
...
2024-06-03 17:58:42 2024-06-04 00:58:42 INFO  OperationTask:143 - Finished job for entity dbName: db, tableName: test, creator: openhouse: JobId SNAPSHOTS_EXPIRATION_db_test_15c49365-2bb3-4ba6-89ae-5ce396ad67c0, executionId 0, runTime 47116, queuedTime 17212, state SUCCEEDED
2024-06-03 17:58:42 2024-06-04 00:58:42 INFO  JobsScheduler:182 - Finishing scheduler for job type SNAPSHOTS_EXPIRATION, tasks stats: 1 created, 1 succeeded, 0 cancelled (timeout), 0 failed, 0 skipped (no state)

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Additional Information

For all the boxes checked, include additional details of the changes made in this pull request.