Open pushkarva opened 1 year ago
Could you please let me know if the schema for the collection is constant for all the documents?
Could you please let me know if the schema for the collection is constant for all the documents?
Why would this be an issue? MongoDB is specifically built for variable data schemas
I'm also running into this issue - although I couldn't figure out the cause as java.lang.NullPointerException
is decidedly unhelpful. I tried setting a specific schema in BigQuery for the destination table but to no avail
We have made improvements to the template - now we'll avoid throwing NullPointerException and infer null values as STRINGs. There are still improvements to do -- such as allowing/reusing a given BigQuery schema.
I've got the same error while trying to import from the Atlas cluster with the MongoDB 7.2 version.
com.google.cloud.teleport.v2.common.UncaughtExceptionLogger - The template launch failed.
java.lang.NullPointerException: null
at com.google.cloud.teleport.v2.mongodb.templates.MongoDbUtils.getTableFieldSchema(MongoDbUtils.java:77)
at com.google.cloud.teleport.v2.mongodb.templates.MongoDbToBigQuery.run(MongoDbToBigQuery.java:121)
at com.google.cloud.teleport.v2.mongodb.templates.MongoDbToBigQuery.main(MongoDbToBigQuery.java:96)
Related Template(s)
MongoDB-to-BigQuery
What happened?
[Bug]: Dataflow - MongoDB-to-BigQuery batch mode failing with NullPointer for userOption FLATTEN
command:
gcloud dataflow flex-template run mongodbtobq01 --template-file-gcs-location gs://dataflow-templates-us-central1/latest/flex/MongoDB_to_BigQuery --region us-central1 --parameters mongoDbUri=mongodb+srv://xxxxxxxxxxxxx,database=xxxxxx-dev,collection=usermasters,outputTableSpec=skilled-duality-xxxxxx.xxxxxx.xxxxxxxxx,userOption=FLATTEN
Getting NullPointer: (It does load data as json into big query when i use userOption as NONE)
{"severity":"INFO","time":"2022/11/20 20:44:09.252706","line":"exec.go:66","message":"java.lang.NullPointerException"} {"severity":"INFO","time":"2022/11/20 20:44:09.252951","line":"exec.go:66","message":"\tat com.google.cloud.teleport.v2.mongodb.templates.MongoDbUtils.lambda$getTableFieldSchema$0(MongoDbUtils.java:58)"}
Beam Version
Newer than 2.35.0
Relevant log output