Closed Zhujun-Vungle closed 5 years ago
Hi.. please leave us a note on the dev mailing list for faster response. https://hudi.apache.org/community.html
Did I missing some hudi config in writing pipeline?
For Spark SQL, you need to push the path filter as documented here. https://hudi.apache.org/querying_data.html#spark-ro-view Getting duplicates typically means, this is missing..
Why the commit is still inflight but parquet file can be generated?
So, the sequence of events is . We write parquet files and then upon successful writing of all attempted parquet files, we actually make the commit as completed. (i.e not inflight anymore). So this is normal. This is done to prevent queries from reading partially written parquet files..
In hudi, did the commit finish and then spark .save finish so there is no asynchronous problem?
.save() internally calls hoodieWriteClient.commit().
Is there any reader side config issue with hive output format ? since it's org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
OutputFormat is not really used anywhere by Hudi. So no.
Hope that helps. I will start a thread on the mailing list and we can continue there, if you have more follow ups.
Thank you so much !! Really helpful!!
Hi
Sorry , failed open https://lists.apache.org/list.html?dev@hudi.apache.org.
I have some follow up questions for this issue:
So, the sequence of events is . We write parquet files and then upon successful writing of all attempted parquet files, we actually make the commit as completed. (i.e not inflight anymore). So this is normal. This is done to prevent queries from reading partially written parquet files..
Does that mean:
Best,
@Zhujun-Vungle I responded on the mailing list.. hope that answers this
Gotcha, thank you so much!
Hi, Taking investigation on hudi, currently facing a issue when trying hudi global dedup
ENV
data source: kafka etl engine: spark streaming which simply ingest data per 10 minutes and write to s3 with global dedup enable data sink: s3 viewer: hive table which create like https://cwiki.apache.org/confluence/display/HUDI/Registering+sample+dataset+to+Hive+via+beeline
HUDI config
phenomenon
Order by last update time![Screen Shot 2019-04-16 at 4 22 00 PM](https://user-images.githubusercontent.com/22140275/56193814-195c8280-6064-11e9-8f09-c45f2ebc98b6.png)
The spark streaming running log, no delay at all![Screen Shot 2019-04-16 at 4 22 53 PM](https://user-images.githubusercontent.com/22140275/56193934-60e30e80-6064-11e9-990e-64cff58361b0.png)
question
After running pipeline for a while, I query the hive table(running in spark sql
select event_id,count(*) from hudi_table group by 1 having COUNT(*) > 1 order by 1 desc limit 10
), I got duplicate records. The duplicate records are coming from the inflight commit. SoRegards