search

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
2.03k stars 365 forks source link

`TableNotFoundError` on `DeltaTable.create(...)` #2195

Closed rspears74 closed 1 month ago

rspears74 commented 5 months ago

Environment

Delta-rs version: 0.15.1

Binding: Python

Environment:

Bug

What happened: I am calling DeltaTable.create with an S3 path. I get a TableNotFoundError.

What you expected to happen: I expect a table to be created at the path I'm specifying.

How to reproduce it: Use DeltaTable.create(...) (maybe with an S3 path?)

More details:

ion-elgreco commented 5 months ago

It would help if you can provide more details, what is the exact code you're running. Something that we can reproduce as well

rspears74 commented 5 months ago

There's not much more to show.

schema = '''{
  "name": "id",
  "type": "string",
  "nullable": false,
  "metadata": {}
}'''
storage = {"AWS...": "...", ...}
table = DeltaTable.create(delta_table_s3_path, schema, mode='overwrite', storage_options=storage)

The main point here is that DeltaTable.create should never throw a TableNotFoundError as it doesn't even make sense in context. It should either create a new table, or say that a table already exists, but then if you create it in overwrite mode, it should work.

rspears74 commented 5 months ago

This might have a similar problem as found in #2197 - which is essentially that it's unable to handle creating a new "folder" in s3.

ion-elgreco commented 4 months ago

@rspears74 the reason it throws a tableNotFoundError is because at the end of creation it's trying to read the table after creation.

I also use S3 but I don't run into this issue so perhaps it's some permission issue?

rspears74 commented 4 months ago

@ion-elgreco See this comment https://github.com/delta-io/delta-rs/issues/2195#issuecomment-1955192136

ion-elgreco commented 4 months ago

@rspears74 have you tried it against the latest version also?

It's not straightforward what the issue is because I don't have this experience with S3, albeit it's not the AWS s3

rspears74 commented 4 months ago

@ion-elgreco I have not, I'm no longer working with delta-rs (this was over a month ago, and the most recent version at the time was 0.15.1). I don't have time to investigate further right now, but I expect you could potentially reproduce my issue by trying to create a table in an S3 folder that doesn't yet exist - following the example in this comment in the other issue I opened. I've worked around these issues for the time being.