Closed kurtmaile closed 7 years ago
Hey Kurt, Seems the schemas do not match. If I were you I would try to save to a duplicate table and compare the schemas
It might be that there is a bug when inferring the schema because BigQuery Schemas do not match the Spark Struct Schema exactly
@kurtmaile do you still require help in this?
Hey thanks 'Ill close it and try what you suggested cheers!
Do you actually have a sample public dbricks notebook by any chance you can share? No probs if not @kurtmaile this is exactly why I am working on https://github.com/samelamin/docker-zeppelin, feel free to send a pr for a notebook that you might find beneficial
There are various public github datasets
Hi,
Thanks for your help on the previous issue.
Im getting the error below when trying to save a very simple dataframe (4 strings and 1 boolean - simple!) into a bigQuery table with matching column names, types and even the order (does order of dFrame cols need ot match order for bquery table? I assume not, but have the same order anyway).
dframe.flattenedEvent.saveAsBigQueryTable(...)
its such a simple example but cannot get it to work and get the error below - any ideas? I can read from othe tables, its just the writting/saving thats the issue.
java.io.IOException: Provided Schema does not match Table **:sdv.event at com.google.cloud.hadoop.io.bigquery.BigQueryUtils.waitForJobCompletion(BigQueryUtils.java:95) at com.samelamin.spark.bigquery.BigQueryClient.com$samelamin$spark$bigquery$BigQueryClient$$waitForJob(BigQueryClient.scala:143) at com.samelamin.spark.bigquery.BigQueryClient.load(BigQueryClient.scala:110) at com.samelamin.spark.bigquery.package$BigQueryDataFrame.saveAsBigQueryTable(package.scala:163) .......
Do you actually have a sample public dbricks notebook by any chance you can share? No probs if not
Cheers and thanks Kurt