I am trying to setup an ETL to import data from mongo (4.0) to BigQuery.
It's important for us to have the schema of our mongo documents in bigquery. Therefore I tried to add the "infer" strategy to the tap. However this is resulting in the following exeception:
Run invocation could not be completed as block failed: Cannot start plugin tap-mongodb: Separator is found, but chunk is longer than limit
I have tried to increase the output buffer via the respective ENV-Var but that does not seem to have any effect.
I have figured out that the root cause is a quite big collection that stores documents.
Is there any option to exclude that collection from the schema generation and sync?
I am trying to setup an ETL to import data from mongo (4.0) to BigQuery.
It's important for us to have the schema of our mongo documents in bigquery. Therefore I tried to add the "infer" strategy to the tap. However this is resulting in the following exeception:
Run invocation could not be completed as block failed: Cannot start plugin tap-mongodb: Separator is found, but chunk is longer than limit
I have tried to increase the output buffer via the respective ENV-Var but that does not seem to have any effect.
ENV: