kestra-io / kestra

Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
https://kestra.io
Apache License 2.0
7.14k stars 428 forks source link

Upgrading kestra from 17.5 to 17.8 breaks triggered flows executions #4195

Closed aku closed 1 week ago

aku commented 2 weeks ago

Describe the issue

UPD: I've went back to 17.8 and the problem has disappeared.

I guess it would be nice to introduce some sort of compatibility checks between Kestra and plugins versions. Also, the system should report some compatibility issues during startup when upgrading/downgrading


UPD: the same problem happens after I went back to 17.5 version. this time it is a cron-based trigger

I have a couple of flows with kafka triggers and clickhouse triggers

e.g.:

triggers:
  - id: read_orders_from_kafka
    type: io.kestra.plugin.kafka.Trigger
    keyDeserializer: STRING
    valueDeserializer: JSON
    interval: PT5S
    topic: [TOPIC]
    groupId: [GROUP]
    maxRecords: 7000
    properties:
      bootstrap.servers: [REDACTED]
      sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required username="{{ secret('[REDACTED]') }}" password="{{ secret('[REDACTED]' )}}";
      sasl.mechanism: SCRAM-SHA-256
      security.protocol: SASL_PLAINTEXT
      auto.offset.reset: latest
      max.poll.records: 5000

Flows were running smoothly with kestra 17.4 and 17.5 Once I've updated to 17.8 following problem has happened:

A flow creates a new execution that stucks in "CREATED" state. If I click on the execution id in a global Executions tab or from the flow's Execution tab a message with "execution not found" pops up. However I can delete it by checking a box near to the execution id and clicking on DELETE button.

Also, I've checked that execution's data is available in S3 storage.

image image image

I had to downgrade back to 17.5 to make the flows running again. In 17.8 triggered executions stuck in CREATED state

Environment

aku commented 2 weeks ago

I've went back to 17.8 and the problem has disappeared.

I guess it would be nice to introduce some sort of compatibility checks between Kestra and plugins versions. Also, the system should report some compatibility issues during startup when upgrading/downgrading

anna-geller commented 1 week ago

thanks for the confirmation that the issue is now fixed 👍

regarding plugin versioning, we have an internal ticket for this; for transparency, this is planned as an EE feature (it's a governance feature falling into the EE category)