Using changefeedmaxpagesperbatch = 1 on streaming

Azure / azure-cosmosdb-spark

Apache Spark Connector for Azure Cosmos DB

MIT License

201 stars 120 forks source link

Using changefeedmaxpagesperbatch = 1 on streaming #460

Open ealsur opened 3 years ago

ealsur commented 3 years ago

When reading the change feed on a streaming scenario with changefeedmaxpagesperbatch set to 1 and with Start from Beginning, some of the records are missing from the stream batches.

Increasing the the batch size to 5 or other number greater than 1, fixes the issue.