AwsCredentials raises unhashable type: 'dict'

EmilRex commented 3 months ago

First check

[X] I added a descriptive title to this issue.
[X] I used the GitHub search to find a similar issue and didn't find it.
[X] I searched the Prefect documentation for this issue.
[X] I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

Using S3Bucket with AwsCredentials fails with TypeError: unhashable type: 'dict' seemingly no matter how I configure it. Not sure if I'm just hitting edge cases or what. Seems related to PrefectHQ/prefect-aws#372

Reproduction

from prefect import flow
from prefect_aws import S3Bucket, AwsCredentials
from prefect.testing.utilities import prefect_test_harness

@flow(
    persist_result=True,
    result_storage="s3-bucket/unhashable",
)
def unhashable():
    return "unhashable"

if __name__ == "__main__":
    with prefect_test_harness() as harness:
        credentials = AwsCredentials()
        credentials.save("unhashable")

        bucket = S3Bucket(
            bucket_name="unhashable",
            credentials=credentials,
        )
        bucket.save("unhashable")

        unhashable()

Error

10:38:48.699 | INFO    | prefect.engine - Created flow run 'cinnamon-flounder' for flow 'unhashable'
10:38:48.759 | ERROR   | Flow run 'cinnamon-flounder' - Crash detected! Execution was interrupted by an unexpected exception: TypeError: unhashable type: 'dict'
Traceback (most recent call last):
  File "/Users/emilchristensen/github/prefect-demos/flows/unhashable.py", line 25, in <module>
    unhashable()
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect/flows.py", line 1231, in __call__
    return enter_flow_run_engine_from_flow_call(
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect/engine.py", line 293, in enter_flow_run_engine_from_flow_call
    retval = from_sync.wait_for_call_in_loop_thread(
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect/_internal/concurrency/api.py", line 218, in wait_for_call_in_loop_thread
    return call.result()
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 318, in result
    return self.future.result(timeout=timeout)
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 179, in result
    return self.__get_result()
  File "/opt/homebrew/Cellar/python@3.9/3.9.19/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 389, in _run_async
    result = await coro
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect/client/utilities.py", line 100, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect/engine.py", line 385, in create_then_begin_flow_run
    state = await begin_flow_run(
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect/engine.py", line 551, in begin_flow_run
    terminal_or_paused_state = await orchestrate_flow_run(
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect/engine.py", line 929, in orchestrate_flow_run
    terminal_state = await return_value_to_state(
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect/states.py", line 305, in return_value_to_state
    return Completed(data=await result_factory.create_result(data))
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect/results.py", line 453, in create_result
    return await PersistedResult.create(
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect/results.py", line 690, in create
    await storage_block.write_path(key, content=blob.to_bytes())
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect_aws/s3.py", line 658, in write_path
    await run_sync_in_worker_thread(self._write_sync, path, content)
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 136, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect_aws/s3.py", line 668, in _write_sync
    s3_client = self._get_s3_client()
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect_aws/s3.py", line 470, in _get_s3_client
    return self.credentials.get_client("s3")
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect_aws/credentials.py", line 171, in get_client
    return _get_client_cached(ctx=self, client_type=client_type)
  File "/Users/emilchristensen/github/prefect-demos/.venv/lib/python3.9/site-packages/prefect_aws/credentials.py", line 122, in __hash__
    hash(self.aws_client_parameters),
TypeError: unhashable type: 'dict'

Versions

prefect-aws==0.4.17

Version:             2.19.5
API version:         0.8.4
Python version:      3.9.19
Git commit:          538dc374
Built:               Thu, Jun 13, 2024 2:50 PM
OS/Arch:             darwin/arm64
Profile:             sandbox
Server type:         cloud

Additional context

No response

TWeatherston commented 3 months ago

I've been experiencing the same issue when attempting to migrate from the soon to be deprecated S3 block to S3Bucket. I believe it's related to this issue https://github.com/PrefectHQ/prefect/issues/13349

desertaxle commented 3 months ago

This is fixed in prefect-aws==0.5.0rc1 by https://github.com/PrefectHQ/prefect/pull/13819, but the fix need to be ported to prefect-aws<0.5.0

Rupin817 commented 3 months ago

Any chance we could merge this in the prefect-aws<0.5.0?

bunchesofdonald commented 2 months ago

@Rupin817 prefect-aws==0.4.18 was just released and has a fix for this. Let us know if you run into more issues with it.

PrefectHQ / prefect