inigohidalgo / prefect-polygon-etl

ELT for Polygon.io financial data using Prefect
2 stars 0 forks source link

`deploy_all` fails on a fresh prefect instance due to missing credentials #9

Closed inigohidalgo closed 7 months ago

inigohidalgo commented 7 months ago
inigo@Desktop-Ubuntu ~/s/m/2/polygon_ETL (main)> python src/flows/deploy_all.py
Traceback (most recent call last):
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/prefect/client/orchestration.py", line 1438, in read_block_document_by_name
    response = await self._client.get(
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/httpx/_client.py", line 1801, in get
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/httpx/_client.py", line 1574, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/prefect/client/base.py", line 312, in send
    response.raise_for_status()
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/prefect/client/base.py", line 164, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url 'http://127.0.0.1:4201/api/block_types/slug/secret/block_documents/name/minio-access-key?include_secrets=true'
Response: {'detail': 'Block document not found'}
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/prefect/blocks/core.py", line 743, in _get_block_document
    block_document = await client.read_block_document_by_name(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/prefect/client/orchestration.py", line 1444, in read_block_document_by_name
    raise prefect.exceptions.ObjectNotFound(http_exc=e) from e
prefect.exceptions.ObjectNotFound

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/src/flows/deploy_all.py", line 7, in <module>
    from polygon_elt import get_aggregates
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/src/flows/polygon_elt/get_aggregates.py", line 29, in <module>
    key=pfbs.Secret.load("minio-access-key").get(),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 259, in coroutine_wrapper
    return call()
           ^^^^^^
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 431, in __call__
    return self.result()
           ^^^^^^^^^^^^^
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 317, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 178, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/inigo/.local/share/mise/installs/python/3.11.7/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 388, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/prefect/blocks/core.py", line 838, in load
    block_document, block_document_name = await cls._get_block_document(name)
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/inigo/stacks/mds/240128_polygon_ETL/polygon_ETL/.venv/lib/python3.11/site-packages/prefect/blocks/core.py", line 747, in _get_block_document
    raise ValueError(
ValueError: Unable to find block document named minio-access-key for block type secret