apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.13k stars 842 forks source link

[Bug] paimon oss needs Schema file created first #3519

Open Watson-kubernetes opened 3 weeks ago

Watson-kubernetes commented 3 weeks ago

Search before asking

Paimon version

0.8

Compute Engine

spark3.4

Minimal reproduce step

i want to test the performance of paimon in aliyun oss ,when i use the following code :
ds.write().format("paimon") .mode("append") .option("write.merge-schema", "true") .option("fs.oss.accessKeyId", accessId) .option("fs.oss.accessKeySecret", accessKey) .option("fs.oss.endpoint", ossEndpoint) .save(basePath); the console showed me that Exception in thread "main" java.lang.IllegalArgumentException: Schema file not found in location oss://hdtttt/mor3. Please create table first. at org.apache.paimon.table.FileStoreTableFactory.lambda$create$0(FileStoreTableFactory.java:61) at java.util.Optional.orElseThrow(Optional.java:290) at org.apache.paimon.table.FileStoreTableFactory.create(FileStoreTableFactory.java:59) at org.apache.paimon.table.FileStoreTableFactory.create(FileStoreTableFactory.java:45) at org.apache.paimon.spark.SparkSource.loadTable(SparkSource.scala:85) at org.apache.paimon.spark.SparkSource.getTable(SparkSource.scala:67)

What doesn't meet your expectations?

is there any options config that solves this problem? can paimon auto-initialize the table first commit, if it did not exist in the specified base path like hudi?

Anything else?

No response

Are you willing to submit a PR?

gsralex commented 2 weeks ago

please assign to me

Watson-kubernetes commented 1 week ago

@JingsongLi