Closed Bryan-Meier closed 8 months ago
Hey @desertaxle , I'd be happy to take this up. A little bit about me. I'm Tejas, and I am just getting started with open source. I started using Prefect a couple of weeks ago, and really like it. So I would like to learn more about it, and help make it better. Earlier, I was trying to set up a simple ETL pipeline to learn about Prefect using this ECS Guide in the docs. The guide is a little out of date as well. I took help from @kevingrismore to complete building it out. So I'd like to work on improving the guide as well. :) As for this issue, so far, I've forked the repo, cloned it to my machine. I'd be happy to get some direction about what changes need to be made. 😄
I think in this instance the documentation just needs to be updated. If someone is new to Prefect and/or Python, they lean heavily on examples. When the examples are wrong or don't work it can be frustrating. The link below is where I found the documentation that provides a nonworking example. Let me know if you need more.
https://prefecthq.github.io/prefect-aws/s3/#prefect_aws.s3.S3Bucket.read_path
Thanks for taking this on @tejasanilshah!
@Bryan-Meier Ah I see, basically AwsCredentials
needs to be imported from credentials
and not directly from prefect_aws
, and this issue is requesting simply to update the documentation?
@Bryan-Meier Actually, I'm not sure if there would still be that error.
Since I see that in the __init__.py
there is a line like so:
from .credentials import AwsCredentials, MinIOCredentials
This would allow us to simply do this:
from prefect_aws import AwsCredentials
Can you share the error that you are facing?
@tejasanilshah, unless I am missing something if I run the code I posted, which is from the documentation while of course using a valid access_key, secret, bucket and path, I get the following:
Traceback (most recent call last):
File "C:\src\apps\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\src\apps\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "...\.vscode\extensions\ms-python.debugpy-2024.2.0-win32-x64\bundled\libs\debugpy\__main__.py", line 39, in <module>
cli.main()
File "...\.vscode\extensions\ms-python.debugpy-2024.2.0-win32-x64\bundled\libs\debugpy/..\debugpy\server\cli.py", line 430, in main
run()
File "...\.vscode\extensions\ms-python.debugpy-2024.2.0-win32-x64\bundled\libs\debugpy/..\debugpy\server\cli.py", line 284, in run_file
runpy.run_path(target, run_name="__main__")
File "...\.vscode\extensions\ms-python.debugpy-2024.2.0-win32-x64\bundled\libs\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_runpy.py", line 321, in run_path
return _run_module_code(code, init_globals, run_name,
File "...\.vscode\extensions\ms-python.debugpy-2024.2.0-win32-x64\bundled\libs\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_runpy.py", line 135, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "...\.vscode\extensions\ms-python.debugpy-2024.2.0-win32-x64\bundled\libs\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_runpy.py", line 124, in _run_code
exec(code, run_globals)
File "...\aws.py", line 15, in <module>
key_contents = s3_bucket_block.read_path(path="testing.txt")
File "c...\.env\lib\site-packages\prefect\utilities\asyncutils.py", line 259, in coroutine_wrapper
return call()
File "...\.env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 398, in __call__
return self.result()
File "...\.env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 284, in result
return self.future.result(timeout=timeout)
File "...\.env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 168, in result
return self.__get_result()
File "C:\src\apps\Python310\lib\concurrent\futures\_base.py", line 403, in __get_result
raise self._exception
File "...\.env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 355, in _run_async
result = await coro
File "...\.env\lib\site-packages\prefect_aws\s3.py", line 606, in read_path
return await run_sync_in_worker_thread(self._read_sync, path)
File "...\.env\lib\site-packages\prefect\utilities\asyncutils.py", line 95, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(
File "...\.env\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "...\.env\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "...\.env\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
result = context.run(func, *args)
File "...\.env\lib\site-packages\prefect_aws\s3.py", line 617, in _read_sync
s3_client.download_fileobj(Bucket=self.bucket_name, Key=key, Fileobj=stream)
File "...\.env\lib\site-packages\boto3\s3\inject.py", line 801, in download_fileobj
return future.result()
File "...\.env\lib\site-packages\s3transfer\futures.py", line 103, in result
return self._coordinator.result()
File "...\.env\lib\site-packages\s3transfer\futures.py", line 266, in result
raise self._exception
File "...\.env\lib\site-packages\s3transfer\tasks.py", line 269, in _main
self._submit(transfer_future=transfer_future, **kwargs)
File "...\.env\lib\site-packages\s3transfer\download.py", line 354, in _submit
response = client.head_object(
File "...\.env\lib\site-packages\botocore\client.py", line 553, in _api_call
return self._make_api_call(operation_name, kwargs)
File "...\.env\lib\site-packages\botocore\client.py", line 989, in _make_api_call
http, parsed_response = self._make_request(
File "..\.env\lib\site-packages\botocore\client.py", line 1015, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "..\.env\lib\site-packages\botocore\endpoint.py", line 119, in make_request
return self._send_request(request_dict, operation_model)
File "...\.env\lib\site-packages\botocore\endpoint.py", line 198, in _send_request
request = self.create_request(request_dict, operation_model)
File "...\.env\lib\site-packages\botocore\endpoint.py", line 134, in create_request
self._event_emitter.emit(
File "...\.env\lib\site-packages\botocore\hooks.py", line 412, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "...\.env\lib\site-packages\botocore\hooks.py", line 256, in emit
return self._emit(event_name, kwargs)
File "...\.env\lib\site-packages\botocore\hooks.py", line 239, in _emit
response = handler(**kwargs)
File "...\.env\lib\site-packages\botocore\signers.py", line 105, in handler
return self.sign(operation_name, request)
File "...\.env\lib\site-packages\botocore\signers.py", line 195, in sign
auth.add_auth(request)
File "...\.env\lib\site-packages\botocore\auth.py", line 418, in add_auth
raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials
The PR I've linked on this issue should address the corrections that are needed in the documentation. Thanks for calling it out!
When using the following example we are presented with an error of unable to locate credentials:
Expectation / Proposal
Expectation = No errors Proposal = Update documentation so that "credential" is used instead of "aws_credentials" in all areas such as what is shown in the S3Bucket part of the example above.