Messages are rescheduled & received at 00:00 UTC

odinnou commented 2 years ago

Hello, thanks for your great jobs! I'm facing a weird issue ... Every night at 00:00 UTC, some messages are rescheduled at 00:00 with an old scheduled-epoch (in my case 8 hours in past) and they are sent on the target topic.

I don't know I you have a suggestion about that?

Thanks in advance, kind regards

fkarakas commented 2 years ago

Hello and thank you. At midnight the scheduler is rescanning all the schedules in the topic and detects all schedules not triggered for the past day and will trigger this schedules at the beginning. Then it will manage the schedules for the current day.

This will allow to trigger the schedules un-triggered because the scheduler was down or not able to publish the message.

Your issue looks like the scheduler was down or he was not able to publish the message correctly...

Are you using the docker image or a GO program write of your own using the scheduler package ?

What are the id of your scheduled messages ? all different or the same ?

if you can make listing of all the messages in the schedules and history topic with their id, timestamp, epoch, and if there is a payload or not.

odinnou commented 2 years ago

Hello thank you for your quick response!

To add some context:

We use your docker image "etf1/kafka-message-scheduler:v0.0.6" with a config file on k8s (to specify username, password and partionier).
Our goal is to simulate a vehicle trip with scheduling GPS position each seconds (one kafka record per second for each vehicle).
It's only for testing: not for a production usecase.
All record values are AVRO serialized (but it seems that we don't need to specify any schema registry configuration).
We are scheduling more than 10k records a day (sometimes at the same second).
I try to force retention of the compacted schedules topic to only 5 hours (to avoid reschedule at 00:00 but this it not working).
Every scheduler-key are a unique identifier (a random uuid).

Regarding the log of the scheduler:

I notice a larger number of : 'time="2022-10-13T07:52:00Z" level=error msg="received an error with code Local: All broker connections are down: 5/5 brokers are down" but when I try a simulation it works!!

Regarding the history topic:

Sample record: key: 123456789123456 payload: raw avro timestamp: 1665619209862 headers:

[
  {
    "key": "scheduler-epoch",
    "stringValue": "1665562035"
  },
  {
    "key": "scheduler-target-topic",
    "stringValue": "telemetry.evt"
  },
  {
    "key": "scheduler-target-key",
    "stringValue": "123456789123456"
  },
  {
    "key": "scheduler-timestamp",
    "stringValue": "1665561745"
  },
  {
    "key": "scheduler-key",
    "stringValue": "d213d0a1-709f-4b2a-87cc-b881a93cab38"
  },
  {
    "key": "scheduler-topic",
    "stringValue": "schedules.evt"
  }
]

Regarding the schedules topic:

Sample record (not marching history sample record): key: 1c2ce0aa0-838c-4f22-92cf-926e1fd5e007 payload: empty timestamp: 1665619209627 headers:

[
  {
    "key": "scheduler-epoch",
    "stringValue": "1665562076"
  },
  {
    "key": "scheduler-target-topic",
    "stringValue": "telemetry.evt"
  },
  {
    "key": "scheduler-target-key",
    "stringValue": "456789123456789"
  },
  {
    "key": "scheduler-timestamp",
    "stringValue": "1665561782"
  },
  {
    "key": "scheduler-key",
    "stringValue": "c2ce0aa0-838c-4f22-92cf-926e1fd5e007"
  },
  {
    "key": "scheduler-topic",
    "stringValue": "schedules.evt"
  }
]

fkarakas Is it possible to avoid any reschedules at 00:00 ?

Thanks in advance, kind regards

fkarakas commented 1 year ago

perhaps same issue as https://github.com/etf1/kafka-message-scheduler/issues/53#issuecomment-1791730859 maybe the scheduler partitioner is different from your producer (written in java ?)

fkarakas commented 1 year ago

closing for now please feel free to reopen

etf1 / kafka-message-scheduler

Messages are rescheduled & received at 00:00 UTC #44