apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.73k stars 4.58k forks source link

[Bug] [Master] dynamic task output param is not compatible with k8s task param #16509

Closed ChaoquanTao closed 3 weeks ago

ChaoquanTao commented 1 month ago

Search before asking

What happened

I orchestrate workflow with a k8s task followed by a dynamic task. When rerunning the workflow instance, the output param of the dynamic task will be applied to the k8s task, working as an environment variable in spec.template.spec.containers[0].env, with a pattern like dynamic.out(taskName). But the k8s environment variable name shouldn't contains brackets, so I got an error like this:

Message: Job.batch "00-uploader-mq-parse-7599" is invalid: spec.template.spec.containers[0].env[11].name: Invalid value: "dynamic.out(03-task-dispatcher)": a valid environment variable name must consist of alphabetic characters, digits, '_', '-', or '.', and must not start with a digit (e.g. 'my.env-name',  or 'MY_ENV.NAME',  or 'MyEnvName1', regex used for validation is '[-._a-zA-Z][-._a-zA-Z0-9]*'). Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.template.spec.containers[0].env[11].name, message=Invalid value: "dynamic.out(03-task-dispatcher)": a valid environment variable name must consist of alphabetic characters, digits, '_', '-', or '.', and must not start with a digit (e.g. 'my.env-name',  or 'MY_ENV.NAME',  or 'MyEnvName1', regex used for validation is '[-._a-zA-Z][-._a-zA-Z0-9]*'), reason=FieldValueInvalid, additionalProperties={})], group=batch, kind=Job, name=00-uploader-mq-parse-7599, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Job.batch "00-uploader-mq-parse-7599" is invalid: spec.template.spec.containers[0].env[11].name: Invalid value: "dynamic.out(03-task-dispatcher)": a valid environment variable name must consist of alphabetic characters, digits, '_', '-', or '.', and must not start with a digit (e.g. 'my.env-name',  or 'MY_ENV.NAME',  or 'MyEnvName1', regex used for validation is '[-._a-zA-Z][-._a-zA-Z0-9]*'), metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:704)
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:682)
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:633)
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:570)
    at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
    at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
    at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$retryWithExponentialBackoff$2(OperationSupport.java:611)
    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
    at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
    at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$4.onResponse(OkHttpClientImpl.java:268)
    at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)

What you expected to happen

rerunning workflow instance shouldn't occur an exception.

How to reproduce

  1. orchestrate a workflow, with k8s task first, followed by a dynamic task.
  2. execute workflow instance repeatedly,and the error occurs.

Anything else

No response

Version

3.2.x

Are you willing to submit PR?

Code of Conduct