Open erikerlandson opened 2 years ago
+1
One of the use cases I had in mind for this was that the files I want to add are sitting on some s3 bucket. So in this use case,there needs to be a way to supply add_files
with s3 credentials, as parameters.
Similarly to the Hive connector's allow-register-partition-procedure
this should be disabled by default and opted in using a catalog property. The idea being that it should only be turned on if file system location based access control is in place.
Any updates on this? It will be great to have this procedure for importing a bunch of data directly avoiding any insert command.
+1
@martint Could you review the syntax of this procedure? The procedure name in Spark is add_files
. The detailed information is documented at https://iceberg.apache.org/docs/latest/spark-procedures/#add_files
The arguments should follow Trino conventions (e.g. source_table
should not be abused for locations), but the name looks good to me.
like so: https://github.com/RussellSpitzer/iceberg/blob/a4279fc5842046043f2afdc90f2428243958574d/spark3-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestAddFilesProcedure.java#L80