Closed bobgalvao closed 4 years ago
Taking a look.
Quick question: Why is this value set to zero ? "hoodie.cleaner.commits.retained"
Also, The first error seems to be coming from parsing Json. Can you paste the schema in a gist file may be ?
Hello, The configuration parameter "hoodie.cleaner.commits.retained" was used in the test, please do not consider it. I'm sorry I didn't remove it.
In other processes that do not have this schema capture step, the error occurs.
Thank you!
@bobgalvao : Can you list your .hoodie/ and .hoodie/.aux folder and add it as attachment. It looks like you may be using hudi version 0.5.1. The stack-trace conforms to 0.5.1 and not to 0.5.0 Can you confirm this. Also, can you recheck if are not bringing in different versions of hudi accidentally ?
Regarding Problem 3: Does this file actually exist ? s3://bucket01/AWS_CASE/0/cfe57b0e-1ac0-4650-960a-615cdba323f9-0_0-1453-608455_20200521172025.parquet.
@bvaradar: I am using AWS EMR 5.29.0 and according to the release informed by AWS it is 5.0.0-inc. I am using the default installation provided by AWS. Reference: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/images/emr-releases-5x.png
Due to problems with S3, I started using HDFS, and I don't have files stored on S3. I will recreate the environment using S3 and as soon as I receive the errors, I send the information to you.
As I cannot reproduce easily. I may take a while to return. But as soon as the errors occur I will send them to you.
Thank you very much for your return.
@bvaradar I checked the versions in the EMR 5.29.0 cluster and they are in 0.5.0.
drwxr-xr-x 2 root root 4096 Jun 14 14:00 bin drwxr-xr-x 4 root root 4096 Jun 14 14:00 cli -rw-r--r-- 1 root root 2728594 Dec 14 2019 hudi-hadoop-mr-bundle-0.5.0-incubating.jar lrwxrwxrwx 1 root root 42 Jun 14 14:00 hudi-hadoop-mr-bundle.jar -> hudi-hadoop-mr-bundle-0.5.0-incubating.jar -rw-r--r-- 1 root root 1134353 Dec 14 2019 hudi-hive-bundle-0.5.0-incubating.jar lrwxrwxrwx 1 root root 37 Jun 14 14:00 hudi-hive-bundle.jar -> hudi-hive-bundle-0.5.0-incubating.jar -rw-r--r-- 1 root root 20967361 Dec 14 2019 hudi-spark-bundle-0.5.0-incubating.jar lrwxrwxrwx 1 root root 38 Jun 14 14:00 hudi-spark-bundle.jar -> hudi-spark-bundle-0.5.0-incubating.jar -rw-r--r-- 1 root root 23310250 Dec 14 2019 hudi-timeline-server-bundle-0.5.0-incubating.jar lrwxrwxrwx 1 root root 48 Jun 14 14:00 hudi-timeline-server-bundle.jar -> hudi-timeline-server-bundle-0.5.0-incubating.jar -rw-r--r-- 1 root root 39051878 Dec 14 2019 hudi-utilities-bundle-0.5.0-incubating.jar lrwxrwxrwx 1 root root 42 Jun 14 14:00 hudi-utilities-bundle.jar -> hudi-utilities-bundle-0.5.0-incubating.jar [hadoop@ip-10-xx-x-xxx hudi]$ pwd /usr/lib/hudi
@bobgalvao error-1 seems to be failing to read a .avro file, which I have not seen before.. this does not seem related to consistency issues actually, since s3 guarantees that partial objects are not exposed ...
error-2 is delta commit not having any data files for fetching the schema.. This should not happen in the recent 0.5.2 release even IIUC, since we are not writing schema with commit metadata always and reading out of it.. @bvaradar correct me if I am wrong.. but again does not seem related to s3
Correct, they do not seem to be related to consistency issues. I think hudi timeline might give some pointers to what is going on here.
I started processing using S3 and am waiting for errors to occur. I am thinking of migrating EMR to version 5.30.0 which is with version 0.5.2 of Hudi. Do you suggest we continue to evaluate the errors that occurred in version 0.5.0 or do I upgrade from version to 0.5.2? Today, my production process is running smoothly with HDFS. But I want to use S3 again. Guys, thanks for attention so far.
@bobgalvao :Can you start with 0.5.2 on S3 and see if you can repro the error. We haven't seen this error before and it is always good to use 0.5.2 as it has general bug fixes
@bvaradar: Perfect, I will start migrating to version 0.5.2 and signal you if an error occurs. Doubt: is it mandatory to use EMRFS when using Hudi with S3? Thank you.
@bobgalvao nope it’s not . If you are using s3, please consider turning on consistency checks https://hudi.apache.org/docs/configurations.html#withConsistencyCheckEnabled
Hi, I performed the migration to version 0.5.2 and no longer got the errors. Thanks for the support.
@bobgalvao we also have a 0.5.3 out, with a bunch of perf fixes.. Might be going there directly as we cook 0.6.0 :)
When I use hudi-0.7.0 and add clustering to my program, the same problem happened at the process time.But before I add clustering,no problem.
Hi,
I'm having a trouble using Apache Hudi with S3.
Steps to reproduce the behavior:
Environment Description:
AWS EMR: emr-5.29.0 Hudi version : 0.5.0-inc Spark version : 2.4.4 Hive version : 2.3.6 Hadoop version : 2.8.5 Storage : S3 Running on Docker? : No
The errors occur intermittently, making subsequent writing impossible for the error (error - 1) “Unrecognized token 'Objavro' ..” and for the error (error - 2 / 3) “Could not find any data file written for commit…” / "Failed to read schema from data...". In this last case, it normalize it in the next execution, but the streaming or batch processing ends with an error.
Due to the problem in the use of S3, I started using HDFS with the same code, where I had no problems with inconsistencies caused by S3.
I have already enabled EMRFS, but the same errors occur. I also enabled “hoodie.consistency.check.enabled” as recommended when using S3 storage. It seems to me to be related to the consistency of the S3.
I often get the errors below:
Error – 1 (when this error occurs it is no longer possible to use the Hudi dataset.):
Error – 2:
Error - 3:
Hudi-cli information:
Sample code used:
Build.sbt:
name := "aws-case-hudi" version := "0.1" scalaVersion := "2.11.12" libraryDependencies ++= Seq( "org.apache.spark" %% "spark-streaming" % "2.4.4" % "provided", "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.4.4", "org.apache.spark" %% "spark-sql" % "2.4.4", "org.apache.spark" %% "spark-core" % "2.4.4", "org.apache.hudi" % "hudi-spark-bundle" % "0.5.0-incubating", "org.apache.httpcomponents" % "httpclient" % "4.5.12", )
Any additional information I am available.
Best regards, bobgalvao