[query/joins] Datagen jobs never completes

[adithyachakilam]

Steps to reproduce:

• Just start a new docker setup
• run the entire notebook 03-query/11-joins.ipynb
• We will be struck at the triggering the users job.

Even though, clickstream job has finished and we move on to generating the user data, curl localhost:9999/jobs shows that clicks job still has two active sessions and below is the output .

[
  {
    "name": "clicks",
    "config_file": "clickstream/clickstream.json",
    "target": {
      "type": "file",
      "path": "/files/clicks.json"
    },
    "active_sessions": 2,
    "total_records": 1725122,
    "start_time": "2024-02-13 05:15:01",
    "run_time": 172881.522483,
    "status": "COMPLETE",
    "status_msg": "Running, Sim Clock: 2024-02-15 05:16:20.150717"
  },
  {
    "name": "users",
    "config_file": "clickstream/users_init.json",
    "target": {
      "type": "file",
      "path": "/files/users.json"
    },
    "active_sessions": 1967,
    "total_records": 1967,
    "start_time": "2024-02-15 05:16:19",
    "run_time": 20.169898,
    "status": "RUNNING",
    "status_msg": "Starting generator job."
  }
]

I tried playing around the concurrency, # of events and hours of data but the issue still exists.

[nick2432]

can i work on this?

[petermarshallio]

Hey @nick2432! You are more than welcome to. When posting your PR, feel free to add me @petermarshallio or @hevansDev as reviewers.