PrefectHQ / prefect-aws

Prefect integrations with AWS.
https://PrefectHQ.github.io/prefect-aws/
Apache License 2.0
84 stars 40 forks source link

Credential use examples are in correct in documentation #379

Closed Bryan-Meier closed 6 months ago

Bryan-Meier commented 7 months ago

When using the following example we are presented with an error of unable to locate credentials:

from prefect_aws import AwsCredentials
from prefect_aws.s3 import S3Bucket

aws_creds = AwsCredentials(
    aws_access_key_id=AWS_ACCESS_KEY_ID,
    aws_secret_access_key=AWS_SECRET_ACCESS_KEY
)

s3_bucket_block = S3Bucket(
    bucket_name="bucket",
    aws_credentials=aws_creds,
    basepath="subfolder"
)

key_contents = s3_bucket_block.read_path(path="subfolder/file1")

Expectation / Proposal

Expectation = No errors Proposal = Update documentation so that "credential" is used instead of "aws_credentials" in all areas such as what is shown in the S3Bucket part of the example above.

tejasanilshah commented 6 months ago

Hey @desertaxle , I'd be happy to take this up. A little bit about me. I'm Tejas, and I am just getting started with open source. I started using Prefect a couple of weeks ago, and really like it. So I would like to learn more about it, and help make it better. Earlier, I was trying to set up a simple ETL pipeline to learn about Prefect using this ECS Guide in the docs. The guide is a little out of date as well. I took help from @kevingrismore to complete building it out. So I'd like to work on improving the guide as well. :) As for this issue, so far, I've forked the repo, cloned it to my machine. I'd be happy to get some direction about what changes need to be made. 😄

Bryan-Meier commented 6 months ago

I think in this instance the documentation just needs to be updated. If someone is new to Prefect and/or Python, they lean heavily on examples. When the examples are wrong or don't work it can be frustrating. The link below is where I found the documentation that provides a nonworking example. Let me know if you need more.

https://prefecthq.github.io/prefect-aws/s3/#prefect_aws.s3.S3Bucket.read_path

Thanks for taking this on @tejasanilshah!

tejasanilshah commented 6 months ago

@Bryan-Meier Ah I see, basically AwsCredentials needs to be imported from credentials and not directly from prefect_aws, and this issue is requesting simply to update the documentation?

tejasanilshah commented 6 months ago

@Bryan-Meier Actually, I'm not sure if there would still be that error.

Since I see that in the __init__.py there is a line like so:

from .credentials import AwsCredentials, MinIOCredentials

This would allow us to simply do this:

from prefect_aws import AwsCredentials

Can you share the error that you are facing?

Bryan-Meier commented 6 months ago

@tejasanilshah, unless I am missing something if I run the code I posted, which is from the documentation while of course using a valid access_key, secret, bucket and path, I get the following:

Traceback (most recent call last):
  File "C:\src\apps\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\src\apps\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "...\.vscode\extensions\ms-python.debugpy-2024.2.0-win32-x64\bundled\libs\debugpy\__main__.py", line 39, in <module>
    cli.main()
  File "...\.vscode\extensions\ms-python.debugpy-2024.2.0-win32-x64\bundled\libs\debugpy/..\debugpy\server\cli.py", line 430, in main
    run()
  File "...\.vscode\extensions\ms-python.debugpy-2024.2.0-win32-x64\bundled\libs\debugpy/..\debugpy\server\cli.py", line 284, in run_file     
    runpy.run_path(target, run_name="__main__")
  File "...\.vscode\extensions\ms-python.debugpy-2024.2.0-win32-x64\bundled\libs\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "...\.vscode\extensions\ms-python.debugpy-2024.2.0-win32-x64\bundled\libs\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "...\.vscode\extensions\ms-python.debugpy-2024.2.0-win32-x64\bundled\libs\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "...\aws.py", line 15, in <module>
    key_contents = s3_bucket_block.read_path(path="testing.txt")
  File "c...\.env\lib\site-packages\prefect\utilities\asyncutils.py", line 259, in coroutine_wrapper
    return call()
  File "...\.env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 398, in __call__
    return self.result()
  File "...\.env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 284, in result
    return self.future.result(timeout=timeout)
  File "...\.env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 168, in result
    return self.__get_result()
  File "C:\src\apps\Python310\lib\concurrent\futures\_base.py", line 403, in __get_result
    raise self._exception
  File "...\.env\lib\site-packages\prefect\_internal\concurrency\calls.py", line 355, in _run_async
    result = await coro
  File "...\.env\lib\site-packages\prefect_aws\s3.py", line 606, in read_path
    return await run_sync_in_worker_thread(self._read_sync, path)
  File "...\.env\lib\site-packages\prefect\utilities\asyncutils.py", line 95, in run_sync_in_worker_thread     
    return await anyio.to_thread.run_sync(
  File "...\.env\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "...\.env\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread        
    return await future
  File "...\.env\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "...\.env\lib\site-packages\prefect_aws\s3.py", line 617, in _read_sync
    s3_client.download_fileobj(Bucket=self.bucket_name, Key=key, Fileobj=stream)
  File "...\.env\lib\site-packages\boto3\s3\inject.py", line 801, in download_fileobj
    return future.result()
  File "...\.env\lib\site-packages\s3transfer\futures.py", line 103, in result
    return self._coordinator.result()
  File "...\.env\lib\site-packages\s3transfer\futures.py", line 266, in result
    raise self._exception
  File "...\.env\lib\site-packages\s3transfer\tasks.py", line 269, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "...\.env\lib\site-packages\s3transfer\download.py", line 354, in _submit
    response = client.head_object(
  File "...\.env\lib\site-packages\botocore\client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "...\.env\lib\site-packages\botocore\client.py", line 989, in _make_api_call
    http, parsed_response = self._make_request(
  File "..\.env\lib\site-packages\botocore\client.py", line 1015, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "..\.env\lib\site-packages\botocore\endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "...\.env\lib\site-packages\botocore\endpoint.py", line 198, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "...\.env\lib\site-packages\botocore\endpoint.py", line 134, in create_request
    self._event_emitter.emit(
  File "...\.env\lib\site-packages\botocore\hooks.py", line 412, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "...\.env\lib\site-packages\botocore\hooks.py", line 256, in emit
    return self._emit(event_name, kwargs)
  File "...\.env\lib\site-packages\botocore\hooks.py", line 239, in _emit
    response = handler(**kwargs)
  File "...\.env\lib\site-packages\botocore\signers.py", line 105, in handler
    return self.sign(operation_name, request)
  File "...\.env\lib\site-packages\botocore\signers.py", line 195, in sign
    auth.add_auth(request)
  File "...\.env\lib\site-packages\botocore\auth.py", line 418, in add_auth
    raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials
kevingrismore commented 6 months ago

The PR I've linked on this issue should address the corrections that are needed in the documentation. Thanks for calling it out!