search

mage-ai / mage-ai

🧙 Build, run, and manage data pipelines for integrating and transforming data.
https://www.mage.ai/
Apache License 2.0
7.48k stars 680 forks source link

[BUG] secure_environment_variables for Azure executor #4872

Open mike-leuer opened 3 months ago

mike-leuer commented 3 months ago

Mage version

v0.9.68

Describe the bug

Azure block executor does not work with secure env variables.

To reproduce

  1. Create a azure container group with secret env variables with mage ai image.
  2. Run any pipeline

Expected behavior

No response

Screenshots

azure.core.exceptions.HttpResponseError: (InvalidEnvironmentVariable) The environment variable 'MAGE_DATABASE_CONNECTION_URL' in container 'mage-data-prep-block-19' of container group 'mage-data-prep-block-19' is invalid. One and only one property of 'value' and 'secureValue' can be specified in an environment variable.
Code: InvalidEnvironmentVariable
Message: The environment variable 'MAGE_DATABASE_CONNECTION_URL' in container 'mage-data-prep-block-19' of container group 'mage-data-prep-block-19' is invalid. One and only one property of 'value' and 'secureValue' can be specified in an environment variable.

Operating system

Terraform template https://github.com/mage-ai/mage-ai-terraform-templates With on lines added https://github.com/mage-ai/mage-ai-terraform-templates/blob/39b7e3b2eaef6d5f34dcaedabcb7e38f2c83fbb2/azure/main.tf#L71

secure_environment_variables = { "TEST" = "TEST" }

Additional context

No response

mike-leuer commented 3 months ago 

INFO:azure.core.pipeline.policies.http_logging_policy:Request URL: 'https://management.azure.com/subscriptions/e5b34700-2f7a-4982-9cac-19158cc3d5e0/resourceGroups/cgc-rnd-dev-rg-mageai/providers/Microsoft.ContainerInstance/containerGroups/mage-data-prep-block-19?api-version=REDACTED'
Request method: 'PUT'
Request headers:
    'Content-Type': 'application/json'
    'Content-Length': '1736'
    'Accept': 'application/json'
    'x-ms-client-request-id': '9b3c1eae-f11a-11ee-b02a-00155d2247ba'
    'User-Agent': 'azsdk-python-mgmt-containerinstance/10.1.0 Python/3.10.14 (Linux-5.10.102.2-microsoft-standard-x86_64-with-glibc2.36)'
    'Authorization': 'REDACTED'
A body is sent with the request
INFO:azure.core.pipeline.policies.http_logging_policy:Response status: 400
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Content-Length': '311'
    'Content-Type': 'application/json; charset=utf-8'
    'Expires': '-1'
    'x-ms-ratelimit-remaining-subscription-resource-requests-pt5m': 'REDACTED'
    'x-ms-ratelimit-remaining-subscription-resource-requests-pt1h': 'REDACTED'
    'x-ms-request-id': 'westus2:a5c621c5-350b-41ce-8574-aad89a5ce255'
    'x-ms-ratelimit-remaining-subscription-writes': '1199'
    'x-ms-correlation-request-id': 'REDACTED'
    'x-ms-routing-request-id': 'REDACTED'
    'Strict-Transport-Security': 'REDACTED'
    'X-Content-Type-Options': 'REDACTED'
    'X-Cache': 'REDACTED'
    'X-MSEdge-Ref': 'REDACTED'
    'Date': 'Tue, 02 Apr 2024 17:58:27 GMT'
Process Worker-2:20:1:
Run worker for job block_run_19
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.10/site-packages/newrelic/api/background_task.py", line 117, in wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/mage_ai/orchestration/queue/process_queue.py", line 253, in run
    start_session_and_run(args[1], *args[2], **args[3])
  File "/usr/local/lib/python3.10/site-packages/mage_ai/orchestration/db/process.py", line 15, in start_session_and_run
    results = target(*args)
  File "/usr/local/lib/python3.10/site-packages/mage_ai/orchestration/pipeline_scheduler_original.py", line 1161, in run_block
    return ExecutorFactory.get_block_executor(
  File "/usr/local/lib/python3.10/site-packages/mage_ai/data_preparation/executors/azure_container_instance_executor.py", line 28, in execute
    container_instance.run_job(
  File "/usr/local/lib/python3.10/site-packages/mage_ai/services/azure/container_instance/container_instance.py", line 79, in run_job
    result = aci_client.container_groups.begin_create_or_update(
  File "/usr/local/lib/python3.10/site-packages/azure/core/tracing/decorator.py", line 78, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/azure/mgmt/containerinstance/operations/_container_groups_operations.py", line 798, in begin_create_or_update
    raw_result = self._create_or_update_initial(
  File "/usr/local/lib/python3.10/site-packages/azure/mgmt/containerinstance/operations/_container_groups_operations.py", line 660, in _create_or_update_initial
    raise HttpResponseError(response=response, error_format=ARMErrorFormat)
azure.core.exceptions.HttpResponseError: (InvalidEnvironmentVariable) The environment variable 'MAGE_DATABASE_CONNECTION_URL' in container 'mage-data-prep-block-19' of container group 'mage-data-prep-block-19' is invalid. One and only one property of 'value' and 'secureValue' can be specified in an environment variable.
Code: InvalidEnvironmentVariable
Message: The environment variable 'MAGE_DATABASE_CONNECTION_URL' in container 'mage-data-prep-block-19' of container group 'mage-data-prep-block-19' is invalid. One and only one property of 'value' and 'secureValue' can be specified in an environment variable.```