apache / iceberg-python

Apache PyIceberg
https://py.iceberg.apache.org/
Apache License 2.0
490 stars 180 forks source link

Add `view_exists` method to REST Catalog #1242

Open shiv-io opened 1 month ago

shiv-io commented 1 month ago

Part of the adding view support to the REST catalog: #818

Todo:

Please let me know what the appropriate place to add docs would be

shiv-io commented 1 month ago

When I tested catalog.view_exists('default.bar') with a local REST catalog, I got the following exception. This also occurs with the existing catalog.table_exists() method. Is this expected?

from pyiceberg.catalog.rest import RestCatalog

catalog = RestCatalog(
    name='local',
    **{
        'uri': 'http://0.0.0.0:8181'
    }
)

catalog.view_exists('default.bar')

# Traceback (most recent call last):
#   File "/Users/shivgupta/projects/iceberg-python/pyiceberg/catalog/rest.py", line 916, in view_exists
#     response.raise_for_status()
#   File "/Users/shivgupta/projects/iceberg-python/venv/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
#     raise HTTPError(http_error_msg, response=self)
# requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://0.0.0.0:8181/v1/namespaces/default/views/bar

# The above exception was the direct cause of the following exception:

# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File "/Users/shivgupta/projects/iceberg-python/venv/lib/python3.9/site-packages/tenacity/__init__.py", line 336, in wrapped_f
#     return copy(f, *args, **kw)
#   File "/Users/shivgupta/projects/iceberg-python/venv/lib/python3.9/site-packages/tenacity/__init__.py", line 475, in __call__
# >>> catalog.view_exists('default.bar')
# Traceback (most recent call last):
#   File "/Users/shivgupta/projects/iceberg-python/pyiceberg/catalog/rest.py", line 916, in view_exists
#     response.raise_for_status()
#   File "/Users/shivgupta/projects/iceberg-python/venv/lib/python3.9/site-packages/requests/models.py", line 1024, in raise_for_status
#     raise HTTPError(http_error_msg, response=self)
# requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: http://0.0.0.0:8181/v1/namespaces/default/views/bar

# The above exception was the direct cause of the following exception:

# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File "/Users/shivgupta/projects/iceberg-python/venv/lib/python3.9/site-packages/tenacity/__init__.py", line 336, in wrapped_f
#     return copy(f, *args, **kw)
#   File "/Users/shivgupta/projects/iceberg-python/venv/lib/python3.9/site-packages/tenacity/__init__.py", line 475, in __call__
#     do = self.iter(retry_state=retry_state)
#   File "/Users/shivgupta/projects/iceberg-python/venv/lib/python3.9/site-packages/tenacity/__init__.py", line 376, in iter
#     result = action(retry_state)
#   File "/Users/shivgupta/projects/iceberg-python/venv/lib/python3.9/site-packages/tenacity/__init__.py", line 398, in <lambda>
#     self._add_action_func(lambda rs: rs.outcome.result())
#   File "/Users/shivgupta/.pyenv/versions/3.9.16/lib/python3.9/concurrent/futures/_base.py", line 439, in result
#     return self.__get_result()
#   File "/Users/shivgupta/.pyenv/versions/3.9.16/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
#     raise self._exception
#   File "/Users/shivgupta/projects/iceberg-python/venv/lib/python3.9/site-packages/tenacity/__init__.py", line 478, in __call__
#     result = fn(*args, **kwargs)
#   File "/Users/shivgupta/projects/iceberg-python/pyiceberg/catalog/rest.py", line 918, in view_exists
#     self._handle_non_200_response(exc, {})
#   File "/Users/shivgupta/projects/iceberg-python/pyiceberg/catalog/rest.py", line 472, in _handle_non_200_response
#     raise exception(response) from exc
# pyiceberg.exceptions.BadRequestError: RESTError 400: Could not decode json payload: 

Note, I used the tabulario/iceberg-rest image to spin up a REST catalog server locally

sungwy commented 1 month ago

Hi @shiv-io - thank you for putting together this PR!

When I tested catalog.view_exists('default.bar') with a local REST catalog, I got the following exception. This also occurs with the existing catalog.table_exists() method. Is this expected?

Yes, we have seen issues with specific implementations of the REST Catalog exhibiting issues with certain endpoints. Which REST Catalog implementation/image are you using to run your local test?

shiv-io commented 1 month ago

@sungwy I used tabulario/iceberg-rest image to spin up a REST catalog server locally to test with

shiv-io commented 3 weeks ago

@sungwy thanks for reviewing! Let me know if there's anything else. I noticed we don't yet have docs for list_views (#817) and drop_views (#820), btw.

shiv-io commented 1 week ago

@sungwy just bumping this in case this fell off your radar!