search

z3z1ma / tap-mongodb

Tap for mongodb emphasizing simplicity and leveraging Meltano SDK
MIT License
10 stars 4 forks source link

Tap fails with chunk exception #5

Open ZuSe opened 1 year ago

ZuSe commented 1 year ago

I am trying to setup an ETL to import data from mongo (4.0) to BigQuery.

It's important for us to have the schema of our mongo documents in bigquery. Therefore I tried to add the "infer" strategy to the tap. However this is resulting in the following exeception:

Run invocation could not be completed as block failed: Cannot start plugin tap-mongodb: Separator is found, but chunk is longer than limit

I have tried to increase the output buffer via the respective ENV-Var but that does not seem to have any effect.

ENV:

ZuSe commented 1 year ago

I have figured out that the root cause is a quite big collection that stores documents. Is there any option to exclude that collection from the schema generation and sync?