search

apache / iceberg

Apache Iceberg
https://iceberg.apache.org/
Apache License 2.0
6.19k stars 2.16k forks source link

While decrypting iceberg table data using aws encyption sdk getting unsupported version error #8497

Open Harsha-2264175 opened 1 year ago

Harsha-2264175 commented 1 year ago

I am performing Client-side encryption using aws encryption sdk on a csv file and after encryption storing the encrypted data into iceberg table. then storing the encrypted data from iceberg table in s3 in csv format. Performing decryption on the file stored in s3 and it is giving error that: "NotSupportedError: ('Unsupported version 120', ValueError'120 is not a valid Serialization Version'"

I wanted to know if iceberg tables support this format or not? is, it possible or not. Any support is helpful.

amogh-jahagirdar commented 1 year ago

Hey Harsha, I'm a bit confused on your workflow since Iceberg does not support CSV file format. Are you converting the CSV to Parquet and then doing addFiles? If you could clarify your workflow that would be useful.

Iceberg S3 integration currently supports the following server side encryption schemes: https://iceberg.apache.org/docs/latest/aws/#s3-server-side-encryption

Also just as an fyi: we're working on native encryption as part of the V3 spec https://github.com/apache/iceberg/projects/5

github-actions[bot] commented 5 days ago

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.