Open aballiet opened 5 months ago
Hi, Do you have some logs on the Kestra webserver side?
updated the issue @loicmathieu
There is a workaround with the following configuration:
micronaut:
server:
max-request-size: 1GB
netty:
server-type: full_content
But it would limit the request size to 1GB and silently ignore bigger requests, this can be an issue if you start a flow with total inputs size of more than 1GB.
We're still working with the Micronaut team to find a proper fix, see https://github.com/micronaut-projects/micronaut-core/issues/10677
@aballiet do you still have the issues?
Just leaving a note here - v0.16.10 is also affected.
@aballiet @yuri1969 just discovered a bug in Micronaut (not easily reproducible) with POST request including a payload when returning a 404 not found that hang leading to memory / CPU issue. I wonder if it could be the cause.
So, do you se any webhook that you call with a POST method and a payload? Do you sometimes, by inadvertance, use it with a flow or webhook key non-existent?
@loicmathieu As far as our deployment is concerned it starts executions exclusively by POSTing to /api/v1/executions/<namespace>/<flow_id>
- no webhooks are used. Nearly every flow defines two required inputs of the FILE type.
Occasional misconfiguration incidents lead to POSTing to a nonexistent <namespace>/<flow_id>
flow which might trigger HTTP 404 responses.
I stumbled upon a bug when an exception is returned when creating the execution; see https://github.com/kestra-io/kestra/pull/4836.
So if you encounter this issue after multiple calls to the create execution endpoint for a disabled flow or a flow that cannot be read from the database, this will be fixed by the PR. If it's not the case, we will need further investiguation.
Describe the issue
Webserver was working fine in 0.14.4, after upgrading to 0.15.8, webserver systematically crashes after few API calls (2 calls).
We tried a bunch of things and noticed the server struggles globally.
Here is an example of a flow which created a lot of scaling events on Kestra server and can make it crash :
Environment