kestra-io / plugin-azure

Apache License 2.0
3 stars 5 forks source link

`containerName` is not dynamic #116

Closed wrussell1999 closed 3 months ago

wrussell1999 commented 3 months ago

Expected Behavior

When using a task runner, which uses blobStorage, the containerName field is dynamic allowing you to put {{ vars.containerName }} as your value.

Property is dynamic so should allow expressions. Screenshot 2024-07-11 at 13 30 35

Actual Behaviour

Instead, it fails with the following error, suggesting that the expression is not being rendered correctly:

kestra-1    | 2024-07-11 12:12:01,661 INFO  WorkerThread f.a.m.1FRnAXHK8dcYfkJpmYAFfd Provided 1 input(s).
kestra-1    | 2024-07-11 12:12:01,664 ERROR WorkerThread c.a.s.blob.BlobContainerAsyncClient Illegal character in path at index 53: https://kestrademobatchstorage.blob.core.windows.net/{{ vars.containerName }}
kestra-1    | 2024-07-11 12:12:01,676 ERROR WorkerThread f.a.m.1FRnAXHK8dcYfkJpmYAFfd Illegal character in path at index 53: https://kestrademobatchstorage.blob.core.windows.net/{{ vars.containerName }}
kestra-1    | java.lang.IllegalArgumentException: Illegal character in path at index 53: https://kestrademobatchstorage.blob.core.windows.net/{{ vars.containerName }}
kestra-1    |   at java.base/java.net.URI.create(Unknown Source)
kestra-1    |   at com.azure.storage.blob.BlobContainerAsyncClient.<init>(BlobContainerAsyncClient.java:149)
kestra-1    |   at com.azure.storage.blob.BlobServiceAsyncClient.getBlobContainerAsyncClient(BlobServiceAsyncClient.java:161)
kestra-1    |   at com.azure.storage.blob.BlobServiceClient.getBlobContainerClient(BlobServiceClient.java:77)
kestra-1    |   at io.kestra.plugin.azure.storage.blob.models.BlobStorageForBatch.blobContainerClient(BlobStorageForBatch.java:53)
kestra-1    |   at io.kestra.plugin.azure.runner.Batch.run(Batch.java:168)
kestra-1    |   at io.kestra.plugin.scripts.exec.scripts.runners.CommandsWrapper.run(CommandsWrapper.java:159)
kestra-1    |   at io.kestra.plugin.scripts.python.Commands.run(Commands.java:281)
kestra-1    |   at io.kestra.plugin.scripts.python.Commands.run(Commands.java:19)
kestra-1    |   at io.kestra.core.runners.WorkerTaskThread.doRun(WorkerTaskThread.java:77)
kestra-1    |   at io.kestra.core.runners.AbstractWorkerThread.run(AbstractWorkerThread.java:56)
kestra-1    | Caused by: java.net.URISyntaxException: Illegal character in path at index 53: https://kestrademobatchstorage.blob.core.windows.net/{{ vars.containerName }}
kestra-1    |   at java.base/java.net.URI$Parser.fail(Unknown Source)
kestra-1    |   at java.base/java.net.URI$Parser.checkChars(Unknown Source)
kestra-1    |   at java.base/java.net.URI$Parser.parseHierarchical(Unknown Source)
kestra-1    |   at java.base/java.net.URI$Parser.parse(Unknown Source)
kestra-1    |   at java.base/java.net.URI.<init>(Unknown Source)
kestra-1    |   ... 11 common frames omitted

If you put the container name as a string directly on the containerName property, it works as expected.

Steps To Reproduce

No response

Environment Information

Example flow

Example from Azure task runner example

id: azure_batch_runner
namespace: company.team

variables:
  poolId: "poolId"
  containerName: "containerName"

tasks:
  - id: scrape_environment_info
    type: io.kestra.plugin.scripts.python.Commands
    containerImage: ghcr.io/kestra-io/pydata:latest
    taskRunner:
      type: io.kestra.plugin.azure.runner.Batch
      account: "{{ secret('AZURE_ACCOUNT') }}"
      accessKey: "{{ secret('AZURE_ACCESS_KEY') }}"
      endpoint: "{{ secret('AZURE_ENDPOINT') }}"
      poolId: "{{ vars.poolId }}"
      blobStorage:
        containerName: "{{ vars.containerName }}"
        connectionString: "{{ secret('AZURE_CONNECTION_STRING') }}"
    commands:
      - python main.py
    namespaceFiles:
      enabled: true
    outputFiles:
      - "environment_info.json"
    inputFiles:
      main.py: |
        import platform
        import socket
        import sys
        import json
        from kestra import Kestra

        print("Hello from Azure Batch and kestra!")

        def print_environment_info():
            print(f"Host's network name: {platform.node()}")
            print(f"Python version: {platform.python_version()}")
            print(f"Platform information (instance type): {platform.platform()}")
            print(f"OS/Arch: {sys.platform}/{platform.machine()}")

            env_info = {
                "host": platform.node(),
                "platform": platform.platform(),
                "OS": sys.platform,
                "python_version": platform.python_version(),
            }
            Kestra.outputs(env_info)

            filename = 'environment_info.json'
            with open(filename, 'w') as json_file:
                json.dump(env_info, json_file, indent=4)

        if __name__ == '__main__':
          print_environment_info()
wrussell1999 commented 3 months ago

Note: no issues for poolId