[Bug] Older Python versions cannot auto-import activity return class in sandbox

devery59 commented 10 months ago

What are you really trying to do?

I'm Using Temporal on EKS Cluster and I try to connect temporal in other application and run workflow. When I use custom @dataclass as return value of activity, "Failed decoding arguments" error has occured.

Describe the bug

{
  "message": "Failed decoding arguments",
  "source": "",
  "stackTrace": "  File \"/usr/local/lib/python3.9/site-packages/temporalio/worker/_workflow_instance.py\", line 301, in activate\n    self._apply(job)\n\n  File \"/usr/local/lib/python3.9/site-packages/temporalio/worker/_workflow_instance.py\", line 373, in _apply\n    self._apply_resolve_activity(job.resolve_activity)\n\n  File \"/usr/local/lib/python3.9/site-packages/temporalio/worker/_workflow_instance.py\", line 490, in _apply_resolve_activity\n    ret_vals = self._convert_payloads(\n\n  File \"/usr/local/lib/python3.9/site-packages/temporalio/worker/_workflow_instance.py\", line 1232, in _convert_payloads\n    raise RuntimeError(\"Failed decoding arguments\") from err\n",
  "encodedAttributes": null,
  "cause": {
    "message": "<Custom dataclass path>",
    "source": "",
    "stackTrace": "  File \"/usr/local/lib/python3.9/site-packages/temporalio/worker/_workflow_instance.py\", line 1224, in _convert_payloads\n    return self._payload_converter.from_payloads(\n\n  File \"/usr/local/lib/python3.9/site-packages/temporalio/converter.py\", line 307, in from_payloads\n    values.append(converter.from_payload(payload, type_hint))\n\n  File \"/usr/local/lib/python3.9/site-packages/temporalio/converter.py\", line 569, in from_payload\n    obj = value_to_type(type_hint, obj, self._custom_type_converters)\n\n  File \"/usr/local/lib/python3.9/site-packages/temporalio/converter.py\", line 1361, in value_to_type\n    field_hints = get_type_hints(hint)\n\n  File \"/usr/local/lib/python3.9/typing.py\", line 1450, in get_type_hints\n    base_globals = sys.modules[base.__module__].__dict__\n\n  File \"/usr/local/lib/python3.9/site-packages/temporalio/worker/workflow_sandbox/_importer.py\", line 393, in __getitem__\n    return self.current[key]\n",
    "encodedAttributes": null,
    "cause": null,
    "applicationFailureInfo": {
      "type": "KeyError",
      "nonRetryable": false,
      "details": null
    }
  },
  "applicationFailureInfo": {
    "type": "RuntimeError",
    "nonRetryable": false,
    "details": null
  }
}

Minimal Reproduction

This is my execution code.

async with Worker(
        client,
        task_queue=TASK_QUEUE_NAME,
        workflows=[test_workflow],
        activities=[get_request_s3_dir],
    ):

        await client.execute_workflow(
            test_workflow.run,
            request,
            id=REQUEST_ID,
            task_queue=TASK_QUEUE_NAME,
            retry_policy=RetryPolicy(maximum_interval=timedelta(seconds=2), maximum_attempts=3),
        )

This is my workflow code.

@workflow.defn
class test_workflow:

    @workflow.run
    async def run(self, request: requestDataClass):
        request_s3_dir = await workflow.execute_activity(
            get_request_s3_dir,
            args = [BUCKET_NAME, request.request_id],
            start_to_close_timeout=timedelta(seconds=15),
        )

This is my activity code.

@activity.defn
async def get_request_s3_dir(bucket_name: str, request_id: str) -> customDataClass:
    uri = f"s3://{bucket_name}/{request_id}"
    return customDataClass.from_uri(uri)

This is my custom dataclass code.

@dataclass
class customDataClass:
    uri: str
    bucket_name: str
    object_key: str

    @classmethod
    def from_uri(cls, uri: str):
        ...
        return cls(uri, bucket_name, object_key)

Environment/Versions

OS and processor: Make Docker image and run pod in EKS
Temporal Version: 2.16.2(on my web) & temporalio == 1.3.0
base Image of Docker : python:3.9-slim-buster

Additional context

I'm using python 3.9 version to test on EKS. But when i use python 3.11 version as my Base Image or python setting there is no issue. Is there any difference between python 3.9 version's temporal env and 3.11 version's temporal env? ( I checked pip list versions. requirements version is same)

I know if i use in my workflow like below...

in python 3.9 verison ( above problem is solved)
with workflow.unsafe.imports_passed_through():
from file import customDataClass

in python 3.11 verison
< no need to import >

Let me know what is stable python version when i use temporal cluster and is there any best way to use pattern like this?

Thanks for your all effort.

cretz commented 10 months ago

\

Is the full error message just a path to the data class? Can I get the exact error message?

Is there any difference between python 3.9 version's temporal env and 3.11 version's temporal env?

None in our code, but Python may have changed how imports work between the two

I know if i use in my workflow like below...

So it appears in newer Python versions the sandbox importer can import your dataclass but in older Python versions you have to import it yourself

Minimal Reproduction

It may be easier with a full replication instead of just snippets, but I may have the idea here.

Regardless, I will see if I can investigate this. In the meantime, import the custom data class yourself as you have shown. Or upgrade to your Python version.

devery59 commented 10 months ago

@cretz Thanks for reply!

Is the full error message just a path to the data class? Can I get the exact error message?

Yes error message just a path to the data class

Regardless, I will see if I can investigate this. In the meantime, import the custom data class yourself as you have shown. Or upgrade to your Python version.

Thanks for your comment! I expect specific reason of above issue. Let me know if there is another thought.

cretz commented 10 months ago

Is it possible to provide a standalone replication? Maybe alter https://github.com/temporalio/samples-python/blob/main/hello/hello_activity.py until you can replicate the failure? (feel free to separate out to multiple files)

temporalio / sdk-python