samelamin
commented
6 years ago Hi
Seems to be a problem with the transaction log. It's hard to be sure without looking at the full stack trace
The library is built using spark 2.1. I've been meaning to upgrade to 2.2 and I'll be doing that over the next few days
I would say try aiming it at a 2.1 cluster or build a custom package and see how it goes
Regards Sam
On Tue, 14 Nov 2017 at 23:50, mayankshah891 notifications@github.com wrote:
I am attempting to read a table from BigQuery and write it to another database.
On my first few tries, I was able to read the table from BigQuery, but when I attempted to write said table to the database, I got the following error:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 13.0 failed 4 times, most recent failure: Lost task 1.3 in stage 13.0 (TID 41, 10.102.241.112, executor 0): java.lang.IllegalStateException: Found known file 'data-000000000002.avro' with index 2, which isn't less than or equal to than endFileNumber 0!
Then, I checked our server, and it had actually worked! So I figured never mind, let me run the process all the way through again.
However, now I can't even read the initial table from BigQuery, with the exact same code. It returns this error:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 20.0 failed 4 times, most recent failure: Lost task 0.3 in stage 20.0 (TID 53, 10.102.249.174, executor 0): java.lang.IllegalStateException: Found known file 'data-000000000002.avro' with index 2, which isn't less than or equal to than endFileNumber 1!
Any clues? I'm using Databricks to import a table from BigQuery. The cluster I'm using is Databricks 3.3, which includes Spark 2.2.0 and Scala 2.11.
Thank you so much for your time and expertise
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/samelamin/spark-bigquery/issues/48, or mute the thread https://github.com/notifications/unsubscribe-auth/AEHLm6Fir7GYmnquj-vnDufO6AT2oNxlks5s2gsZgaJpZM4QeDPk .
mayankshah891
I am attempting to read a table from BigQuery and write it to another database.
On my first few tries, I was able to read the table from BigQuery, but when I attempted to write said table to the database, I got the following error:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 13.0 failed 4 times, most recent failure: Lost task 1.3 in stage 13.0 (TID 41, 10.102.241.112, executor 0): java.lang.IllegalStateException: Found known file 'data-000000000002.avro' with index 2, which isn't less than or equal to than endFileNumber 0!Then, I checked our server, and it had actually worked! So I figured never mind, let me run the process all the way through again.
However, now I can't even read the initial table from BigQuery, with the exact same code. It returns this error:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 20.0 failed 4 times, most recent failure: Lost task 0.3 in stage 20.0 (TID 53, 10.102.249.174, executor 0): java.lang.IllegalStateException: Found known file 'data-000000000002.avro' with index 2, which isn't less than or equal to than endFileNumber 1!Any clues? I'm using Databricks to import a table from BigQuery. The cluster I'm using is Databricks 3.3, which includes Spark 2.2.0 and Scala 2.11.
Thank you so much for your time and expertise