search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.07k stars 4.11k forks source link

[source-mongodb] `read operation. com.mongodb.MongoQueryException: Command failed with error 9` #36551

Open apulache opened 7 months ago

apulache commented 7 months ago

Connector Name

source-mongo

Connector Version

1.3.0

What step the error happened?

During the sync

Relevant information

After configure and test mongodb source successful during sync job failed

Relevant log output

2024-03-27 13:28:30 source > INFO main i.a.i.s.m.c.MongoDbCdcInitialSnapshotUtils(lambda$estimateInitialSnapshotSyncSize$5):132 Estimate for table: sample_data.inventory : {sync_row_count: 1000, sync_bytes: 271060, total_table_row_count: 1000, total_table_bytes: 271060}
2024-03-27 13:28:30 source > ERROR main i.a.i.s.m.MongoDbSource(read):163 Unable to perform sync read operation. com.mongodb.MongoQueryException: Command failed with error 9 (FailedToParse): 'Failed to parse: { find: "inventory", filter: {}, sort: { _id: 1 }, projection: { _id: 1, name: 1, year: 1, model: 1, price: 1, source: 1, variations: 1, _ab_cdc_cursor: 1, sale_frequency: 1, _ab_cdc_deleted_at: 1, _ab_cdc_updated_at: 1 }, allowDiskUse: true, $db: "sample_data", $clusterTime: { clusterTime: Timestamp(1711546108, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lsid: { id: UUID("79eed16d-c834-4934-aa13-1066cd51d9a4") }, $readPreference: { mode: "secondaryPreferred" } }. Unrecognized field 'allowDiskUse'.' on server 192.168.0.42:27017. The full response is {"operationTime": {"$timestamp": {"t": 1711546108, "i": 1}}, "ok": 0.0, "errmsg": "Failed to parse: { find: \"inventory\", filter: {}, sort: { _id: 1 }, projection: { _id: 1, name: 1, year: 1, model: 1, price: 1, source: 1, variations: 1, _ab_cdc_cursor: 1, sale_frequency: 1, _ab_cdc_deleted_at: 1, _ab_cdc_updated_at: 1 }, allowDiskUse: true, $db: \"sample_data\", $clusterTime: { clusterTime: Timestamp(1711546108, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lsid: { id: UUID(\"79eed16d-c834-4934-aa13-1066cd51d9a4\") }, $readPreference: { mode: \"secondaryPreferred\" } }. Unrecognized field 'allowDiskUse'.", "code": 9, "codeName": "FailedToParse", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1711546108, "i": 1}}, "signature": {"hash": {"$binary": {"base64": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", "subType": "00"}}, "keyId": 0}}}
    at com.mongodb.internal.operation.FindOperation.lambda$execute$1(FindOperation.java:328) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$0(OperationHelper.java:345) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:370) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$1(OperationHelper.java:344) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:370) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.OperationHelper.withSourceAndConnection(OperationHelper.java:343) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.FindOperation.lambda$execute$2(FindOperation.java:321) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.CommandOperationHelper.lambda$decorateReadWithRetries$3(CommandOperationHelper.java:192) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.async.function.RetryingSyncSupplier.get(RetryingSyncSupplier.java:67) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:332) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:72) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:153) ~[mongodb-driver-sync-4.10.2.jar:?]
    at com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:130) ~[mongodb-driver-sync-4.10.2.jar:?]
    at com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:90) ~[mongodb-driver-sync-4.10.2.jar:?]
    at com.mongodb.client.internal.MongoIterableImpl.cursor(MongoIterableImpl.java:95) ~[mongodb-driver-sync-4.10.2.jar:?]
    at io.airbyte.integrations.source.mongodb.InitialSnapshotHandler.lambda$getIterators$3(InitialSnapshotHandler.java:97) ~[io.airbyte.airbyte-integrations.connectors-source-mongodb-v2.jar:?]
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708) ~[?:?]
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
    at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
    at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
    at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
    at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
    at io.airbyte.integrations.source.mongodb.InitialSnapshotHandler.getIterators(InitialSnapshotHandler.java:108) ~[io.airbyte.airbyte-integrations.connectors-source-mongodb-v2.jar:?]
    at io.airbyte.integrations.source.mongodb.cdc.MongoDbCdcInitializer.createCdcIterators(MongoDbCdcInitializer.java:146) ~[io.airbyte.airbyte-integrations.connectors-source-mongodb-v2.jar:?]
    at io.airbyte.integrations.source.mongodb.MongoDbSource.read(MongoDbSource.java:155) [io.airbyte.airbyte-integrations.connectors-source-mongodb-v2.jar:?]
    at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:267) [airbyte-cdk-core-0.23.8.jar:?]
    at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:173) [airbyte-cdk-core-0.23.8.jar:?]
    at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125) [airbyte-cdk-core-0.23.8.jar:?]
    at io.airbyte.integrations.source.mongodb.MongoDbSource.main(MongoDbSource.java:52) [io.airbyte.airbyte-integrations.connectors-source-mongodb-v2.jar:?]

Stack Trace: com.mongodb.MongoQueryException: Command failed with error 9 (FailedToParse): 'Failed to parse: { find: "inventory", filter: {}, sort: { _id: 1 }, projection: { _id: 1, name: 1, year: 1, model: 1, price: 1, source: 1, variations: 1, _ab_cdc_cursor: 1, sale_frequency: 1, _ab_cdc_deleted_at: 1, _ab_cdc_updated_at: 1 }, allowDiskUse: true, $db: "sample_data", $clusterTime: { clusterTime: Timestamp(1711546108, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lsid: { id: UUID("79eed16d-c834-4934-aa13-1066cd51d9a4") }, $readPreference: { mode: "secondaryPreferred" } }. Unrecognized field 'allowDiskUse'.' on server 192.168.0.42:27017. The full response is {"operationTime": {"$timestamp": {"t": 1711546108, "i": 1}}, "ok": 0.0, "errmsg": "Failed to parse: { find: \"inventory\", filter: {}, sort: { _id: 1 }, projection: { _id: 1, name: 1, year: 1, model: 1, price: 1, source: 1, variations: 1, _ab_cdc_cursor: 1, sale_frequency: 1, _ab_cdc_deleted_at: 1, _ab_cdc_updated_at: 1 }, allowDiskUse: true, $db: \"sample_data\", $clusterTime: { clusterTime: Timestamp(1711546108, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lsid: { id: UUID(\"79eed16d-c834-4934-aa13-1066cd51d9a4\") }, $readPreference: { mode: \"secondaryPreferred\" } }. Unrecognized field 'allowDiskUse'.", "code": 9, "codeName": "FailedToParse", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1711546108, "i": 1}}, "signature": {"hash": {"$binary": {"base64": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", "subType": "00"}}, "keyId": 0}}}
    at com.mongodb.internal.operation.FindOperation.lambda$execute$1(FindOperation.java:328)
    at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$0(OperationHelper.java:345)
    at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:370)
    at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$1(OperationHelper.java:344)
    at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:370)
    at com.mongodb.internal.operation.OperationHelper.withSourceAndConnection(OperationHelper.java:343)
    at com.mongodb.internal.operation.FindOperation.lambda$execute$2(FindOperation.java:321)
    at com.mongodb.internal.operation.CommandOperationHelper.lambda$decorateReadWithRetries$3(CommandOperationHelper.java:192)
    at com.mongodb.internal.async.function.RetryingSyncSupplier.get(RetryingSyncSupplier.java:67)
    at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:332)
    at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:72)
    at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:153)
    at com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:130)
    at com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:90)
    at com.mongodb.client.internal.MongoIterableImpl.cursor(MongoIterableImpl.java:95)
    at io.airbyte.integrations.source.mongodb.InitialSnapshotHandler.lambda$getIterators$3(InitialSnapshotHandler.java:97)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
    at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
    at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
    at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
    at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
    at io.airbyte.integrations.source.mongodb.InitialSnapshotHandler.getIterators(InitialSnapshotHandler.java:108)
    at io.airbyte.integrations.source.mongodb.cdc.MongoDbCdcInitializer.createCdcIterators(MongoDbCdcInitializer.java:146)
    at io.airbyte.integrations.source.mongodb.MongoDbSource.read(MongoDbSource.java:155)
    at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:267)
    at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:173)
    at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125)
    at io.airbyte.integrations.source.mongodb.MongoDbSource.main(MongoDbSource.java:52)

2024-03-27 13:28:30 source > ERROR main i.a.c.i.b.AirbyteExceptionHandler(uncaughtException):64 Something went wrong in the connector. See the logs for more details. com.mongodb.MongoQueryException: Command failed with error 9 (FailedToParse): 'Failed to parse: { find: "inventory", filter: {}, sort: { _id: 1 }, projection: { _id: 1, name: 1, year: 1, model: 1, price: 1, source: 1, variations: 1, _ab_cdc_cursor: 1, sale_frequency: 1, _ab_cdc_deleted_at: 1, _ab_cdc_updated_at: 1 }, allowDiskUse: true, $db: "sample_data", $clusterTime: { clusterTime: Timestamp(1711546108, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lsid: { id: UUID("79eed16d-c834-4934-aa13-1066cd51d9a4") }, $readPreference: { mode: "secondaryPreferred" } }. Unrecognized field 'allowDiskUse'.' on server 192.168.0.42:27017. The full response is {"operationTime": {"$timestamp": {"t": 1711546108, "i": 1}}, "ok": 0.0, "errmsg": "Failed to parse: { find: \"inventory\", filter: {}, sort: { _id: 1 }, projection: { _id: 1, name: 1, year: 1, model: 1, price: 1, source: 1, variations: 1, _ab_cdc_cursor: 1, sale_frequency: 1, _ab_cdc_deleted_at: 1, _ab_cdc_updated_at: 1 }, allowDiskUse: true, $db: \"sample_data\", $clusterTime: { clusterTime: Timestamp(1711546108, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lsid: { id: UUID(\"79eed16d-c834-4934-aa13-1066cd51d9a4\") }, $readPreference: { mode: \"secondaryPreferred\" } }. Unrecognized field 'allowDiskUse'.", "code": 9, "codeName": "FailedToParse", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1711546108, "i": 1}}, "signature": {"hash": {"$binary": {"base64": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", "subType": "00"}}, "keyId": 0}}}
    at com.mongodb.internal.operation.FindOperation.lambda$execute$1(FindOperation.java:328) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$0(OperationHelper.java:345) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:370) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$1(OperationHelper.java:344) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:370) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.OperationHelper.withSourceAndConnection(OperationHelper.java:343) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.FindOperation.lambda$execute$2(FindOperation.java:321) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.CommandOperationHelper.lambda$decorateReadWithRetries$3(CommandOperationHelper.java:192) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.async.function.RetryingSyncSupplier.get(RetryingSyncSupplier.java:67) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:332) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:72) ~[mongodb-driver-core-4.10.2.jar:?]
    at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:153) ~[mongodb-driver-sync-4.10.2.jar:?]
    at com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:130) ~[mongodb-driver-sync-4.10.2.jar:?]
    at com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:90) ~[mongodb-driver-sync-4.10.2.jar:?]
    at com.mongodb.client.internal.MongoIterableImpl.cursor(MongoIterableImpl.java:95) ~[mongodb-driver-sync-4.10.2.jar:?]
    at io.airbyte.integrations.source.mongodb.InitialSnapshotHandler.lambda$getIterators$3(InitialSnapshotHandler.java:97) ~[io.airbyte.airbyte-integrations.connectors-source-mongodb-v2.jar:?]
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708) ~[?:?]
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
    at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
    at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
    at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
    at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
    at io.airbyte.integrations.source.mongodb.InitialSnapshotHandler.getIterators(InitialSnapshotHandler.java:108) ~[io.airbyte.airbyte-integrations.connectors-source-mongodb-v2.jar:?]
    at io.airbyte.integrations.source.mongodb.cdc.MongoDbCdcInitializer.createCdcIterators(MongoDbCdcInitializer.java:146) ~[io.airbyte.airbyte-integrations.connectors-source-mongodb-v2.jar:?]
    at io.airbyte.integrations.source.mongodb.MongoDbSource.read(MongoDbSource.java:155) ~[io.airbyte.airbyte-integrations.connectors-source-mongodb-v2.jar:?]
    at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:267) ~[airbyte-cdk-core-0.23.8.jar:?]
    at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:173) ~[airbyte-cdk-core-0.23.8.jar:?]
    at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125) ~[airbyte-cdk-core-0.23.8.jar:?]
    at io.airbyte.integrations.source.mongodb.MongoDbSource.main(MongoDbSource.java:52) ~[io.airbyte.airbyte-integrations.connectors-source-mongodb-v2.jar:?]

Stack Trace: com.mongodb.MongoQueryException: Command failed with error 9 (FailedToParse): 'Failed to parse: { find: "inventory", filter: {}, sort: { _id: 1 }, projection: { _id: 1, name: 1, year: 1, model: 1, price: 1, source: 1, variations: 1, _ab_cdc_cursor: 1, sale_frequency: 1, _ab_cdc_deleted_at: 1, _ab_cdc_updated_at: 1 }, allowDiskUse: true, $db: "sample_data", $clusterTime: { clusterTime: Timestamp(1711546108, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lsid: { id: UUID("79eed16d-c834-4934-aa13-1066cd51d9a4") }, $readPreference: { mode: "secondaryPreferred" } }. Unrecognized field 'allowDiskUse'.' on server 192.168.0.42:27017. The full response is {"operationTime": {"$timestamp": {"t": 1711546108, "i": 1}}, "ok": 0.0, "errmsg": "Failed to parse: { find: \"inventory\", filter: {}, sort: { _id: 1 }, projection: { _id: 1, name: 1, year: 1, model: 1, price: 1, source: 1, variations: 1, _ab_cdc_cursor: 1, sale_frequency: 1, _ab_cdc_deleted_at: 1, _ab_cdc_updated_at: 1 }, allowDiskUse: true, $db: \"sample_data\", $clusterTime: { clusterTime: Timestamp(1711546108, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lsid: { id: UUID(\"79eed16d-c834-4934-aa13-1066cd51d9a4\") }, $readPreference: { mode: \"secondaryPreferred\" } }. Unrecognized field 'allowDiskUse'.", "code": 9, "codeName": "FailedToParse", "$clusterTime": {"clusterTime": {"$timestamp": {"t": 1711546108, "i": 1}}, "signature": {"hash": {"$binary": {"base64": "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", "subType": "00"}}, "keyId": 0}}}
    at com.mongodb.internal.operation.FindOperation.lambda$execute$1(FindOperation.java:328)
    at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$0(OperationHelper.java:345)
    at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:370)
    at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$1(OperationHelper.java:344)
    at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:370)
    at com.mongodb.internal.operation.OperationHelper.withSourceAndConnection(OperationHelper.java:343)
    at com.mongodb.internal.operation.FindOperation.lambda$execute$2(FindOperation.java:321)
    at com.mongodb.internal.operation.CommandOperationHelper.lambda$decorateReadWithRetries$3(CommandOperationHelper.java:192)
    at com.mongodb.internal.async.function.RetryingSyncSupplier.get(RetryingSyncSupplier.java:67)
    at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:332)
    at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:72)
    at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:153)
    at com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:130)
    at com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:90)
    at com.mongodb.client.internal.MongoIterableImpl.cursor(MongoIterableImpl.java:95)
    at io.airbyte.integrations.source.mongodb.InitialSnapshotHandler.lambda$getIterators$3(InitialSnapshotHandler.java:97)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
    at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
    at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
    at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
    at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
    at io.airbyte.integrations.source.mongodb.InitialSnapshotHandler.getIterators(InitialSnapshotHandler.java:108)
    at io.airbyte.integrations.source.mongodb.cdc.MongoDbCdcInitializer.createCdcIterators(MongoDbCdcInitializer.java:146)
    at io.airbyte.integrations.source.mongodb.MongoDbSource.read(MongoDbSource.java:155)
    at io.airbyte.cdk.integrations.base.IntegrationRunner.readSerial(IntegrationRunner.java:267)
    at io.airbyte.cdk.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:173)
    at io.airbyte.cdk.integrations.base.IntegrationRunner.run(IntegrationRunner.java:125)
    at io.airbyte.integrations.source.mongodb.MongoDbSource.main(MongoDbSource.java:52)

2024-03-27 13:28:30 platform > readFromSource: source exception
io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 1
    at io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:362) ~[io.airbyte-airbyte-commons-worker-0.55.2.jar:?]
    at io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:234) ~[io.airbyte-airbyte-commons-worker-0.55.2.jar:?]
    at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-03-27 13:28:30 platform > readFromSource: done. (source.isFinished:true, fromSource.isClosed:false)
2024-03-27 13:28:30 platform > processMessage: done. (fromSource.isDone:true, forDest.isClosed:false)
2024-03-27 13:28:30 platform > writeToDestination: done. (forDest.isDone:true, isDestRunning:true)
2024-03-27 13:28:30 platform > thread status... timeout thread: false , replication thread: true
2024-03-27 13:28:30 destination > Airbyte message consumer: succeeded.
2024-03-27 13:28:30 destination > Migration finished with no explicit errors. Copying data from tmp tables to permanent
2024-03-27 13:28:30 destination > null
2024-03-27 13:28:30 destination > Removing tmp collections...
2024-03-27 13:28:30 destination > Finishing destination process...completed
2024-03-27 13:28:30 destination > Completed integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
2024-03-27 13:28:30 destination > completed destination: class io.airbyte.integrations.destination.mongodb.MongodbDestination
2024-03-27 13:28:30 platform > readFromDestination: done. (writeToDestFailed:false, dest.isFinished:true)
2024-03-27 13:28:30 platform > thread status... timeout thread: false , replication thread: true
2024-03-27 13:28:30 platform > sync summary: {
  "status" : "failed",
  "recordsSynced" : 0,
  "bytesSynced" : 0,
  "startTime" : 1711546107471,
  "endTime" : 1711546110591,
  "totalStats" : {
    "bytesCommitted" : 0,
    "bytesEmitted" : 0,
    "destinationStateMessagesEmitted" : 0,
    "destinationWriteEndTime" : 1711546110585,
    "destinationWriteStartTime" : 1711546107477,
    "meanSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBeforeSourceStateMessageEmitted" : 0,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "recordsEmitted" : 0,
    "recordsCommitted" : 0,
    "replicationEndTime" : 1711546110589,
    "replicationStartTime" : 1711546107471,
    "sourceReadEndTime" : 0,
    "sourceReadStartTime" : 1711546107477,
    "sourceStateMessagesEmitted" : 0
  },
  "streamStats" : [ {
    "streamName" : "inventory",
    "streamNamespace" : "sample_data",
    "stats" : {
      "bytesCommitted" : 0,
      "bytesEmitted" : 0,
      "recordsEmitted" : 0,
      "recordsCommitted" : 0
    }
  } ],
  "performanceMetrics" : {
    "processFromSource" : {
      "elapsedTimeInNanos" : 4079425,
      "executionCount" : 2,
      "avgExecTimeInNanos" : 2039712.5
    },
    "readFromSource" : {
      "elapsedTimeInNanos" : 3001309836,
      "executionCount" : 54,
      "avgExecTimeInNanos" : 5.5579811777777776E7
    },
    "processFromDest" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    },
    "writeToDest" : {
      "elapsedTimeInNanos" : 0,
      "executionCount" : 0,
      "avgExecTimeInNanos" : "NaN"
    },
    "readFromDest" : {
      "elapsedTimeInNanos" : 3076552124,
      "executionCount" : 103,
      "avgExecTimeInNanos" : 2.986943809708738E7
    }
  }
}
2024-03-27 13:28:30 platform > failures: [ {
  "failureOrigin" : "source",
  "failureType" : "system_error",
  "internalMessage" : "Command failed with error 9 (FailedToParse): 'Failed to parse: { find: \"inventory\", filter: {}, sort: { _?: 1 }, projection: { _?: 1, ?: 1, year: 1, model: 1, price: 1, source: 1, variations: 1, _ab_cdc_cursor: 1, sale_frequency: 1, _ab_cdc_deleted_at: 1, _ab_cdc_updated_at: 1 }, allowDiskUse: true, $db: \"sample_data\", $clusterTime: { clusterTime: Timestamp(1711546108, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lsid: { ?: UUID(\"79eed16d-c834-4934-aa13-1066cd51d9a4\") }, $readPreference: { mode: \"secondaryPreferred\" } }. Unrecognized field 'allowDiskUse'.' on server 192.168.0.42:27017. The full response is {\"operationTime\": {\"$timestamp\": {\"t\": 1711546108, \"i\": 1}}, \"ok\": 0.0, \"errmsg\": \"Failed to parse: { find: \\\"inventory\\\", filter: {}, sort: { _?: 1 }, projection: { _?: 1, ?: 1, year: 1, model: 1, price: 1, source: 1, variations: 1, _ab_cdc_cursor: 1, sale_frequency: 1, _ab_cdc_deleted_at: 1, _ab_cdc_updated_at: 1 }, allowDiskUse: true, $db: \\\"sample_data\\\", $clusterTime: { clusterTime: Timestamp(1711546108, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lsid: { ?: UUID(\\\"79eed16d-c834-4934-aa13-1066cd51d9a4\\\") }, $readPreference: { mode: \\\"secondaryPreferred\\\" } }. Unrecognized field 'allowDiskUse'.\", \"code\": 9, \"codeName\": \"FailedToParse\", \"$clusterTime\": {\"clusterTime\": {\"$timestamp\": {\"t\": 1711546108, \"i\": 1}}, \"signature\": {\"hash\": {\"$binary\": {\"base64\": \"AAAAAAAAAAAAAAAAAAAAAAAAAAA=\", \"subType\": \"00\"}}, \"keyId\": 0}}}",
  "externalMessage" : "Command failed with error 9 (FailedToParse): 'Failed to parse: { find: \"inventory\", filter: {}, sort: { _id: 1 }, projection: { _id: 1, name: 1, year: 1, model: 1, price: 1, source: 1, variations: 1, _ab_cdc_cursor: 1, sale_frequency: 1, _ab_cdc_deleted_at: 1, _ab_cdc_updated_at: 1 }, allowDiskUse: true, $db: \"sample_data\", $clusterTime: { clusterTime: Timestamp(1711546108, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lsid: { id: UUID(\"79eed16d-c834-4934-aa13-1066cd51d9a4\") }, $readPreference: { mode: \"secondaryPreferred\" } }. Unrecognized field 'allowDiskUse'.' on server 192.168.0.42:27017. The full response is {\"operationTime\": {\"$timestamp\": {\"t\": 1711546108, \"i\": 1}}, \"ok\": 0.0, \"errmsg\": \"Failed to parse: { find: \\\"inventory\\\", filter: {}, sort: { _id: 1 }, projection: { _id: 1, name: 1, year: 1, model: 1, price: 1, source: 1, variations: 1, _ab_cdc_cursor: 1, sale_frequency: 1, _ab_cdc_deleted_at: 1, _ab_cdc_updated_at: 1 }, allowDiskUse: true, $db: \\\"sample_data\\\", $clusterTime: { clusterTime: Timestamp(1711546108, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, lsid: { id: UUID(\\\"79eed16d-c834-4934-aa13-1066cd51d9a4\\\") }, $readPreference: { mode: \\\"secondaryPreferred\\\" } }. Unrecognized field 'allowDiskUse'.\", \"code\": 9, \"codeName\": \"FailedToParse\", \"$clusterTime\": {\"clusterTime\": {\"$timestamp\": {\"t\": 1711546108, \"i\": 1}}, \"signature\": {\"hash\": {\"$binary\": {\"base64\": \"AAAAAAAAAAAAAAAAAAAAAAAAAAA=\", \"subType\": \"00\"}}, \"keyId\": 0}}}",
  "metadata" : {
    "attemptNumber" : 0,
    "jobId" : 1,
    "from_trace_message" : true,
    "connector_command" : "read"
  },
  "timestamp" : 1711546110451
}, {
  "failureOrigin" : "source",
  "internalMessage" : "Source process exited with non-zero exit code 1",
  "externalMessage" : "Something went wrong within the source connector",
  "metadata" : {
    "attemptNumber" : 0,
    "jobId" : 1,
    "connector_command" : "read"
  },
  "stacktrace" : "io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 1\n\tat io.airbyte.workers.general.BufferedReplicationWorker.readFromSource(BufferedReplicationWorker.java:362)\n\tat io.airbyte.workers.general.BufferedReplicationWorker.lambda$runAsyncWithHeartbeatCheck$3(BufferedReplicationWorker.java:234)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n",
  "timestamp" : 1711546110517
} ]
2024-03-27 13:28:30 platform > 
2024-03-27 13:28:30 platform > ----- END REPLICATION -----
2024-03-27 13:28:30 platform > 
2024-03-27 13:28:30 platform > Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, totalCompleteFailureLimit=10, successivePartialFailureLimit=1000, totalPartialFailureLimit=10, successiveCompleteFailures=1, totalCompleteFailures=1, successivePartialFailures=0, totalPartialFailures=0)
 Backoff before next attempt: 10 seconds

Contribute

marcosmarxm commented 7 months ago

The issue was triaged to the team backlog for further discussion in the next sprint session. If you're facing the same issue please react with šŸ‘or leave a comment to help the team understand how many users are facing the issue.

apulache commented 7 months ago

Here more details aout the issue: It happened when trying to sync a three node replicaSet using MongoDB Community 4.2 and an external additional node MongoDB 4.2.

Is there an easyway to change mongodb drivers based on mongodb specific versions?

evantahler commented 7 months ago

@apulache - can you confirm that the allowDiskUse() method isn't present for your version of Mongo? If it is not available, we will update our docs to reflect the minimum version of mongo we support to be higher than 4.2.

priyamtiet commented 6 months ago

I am getting this while connecting to AWS DocumentDB. Is there any resolution or alternate connector available to read from AWS Document DB.