search

Azure / azure-cosmosdb-spark

Apache Spark Connector for Azure Cosmos DB
MIT License
199 stars 119 forks source link

Using "WriteThroughputBudget" makes the spark job to drag writing records in cosmos container and job runs longer #447

Open txl013 opened 3 years ago

txl013 commented 3 years ago

I started to use, 3.6.1 version and "WriteThroughputBudget" is working. But i am experiencing a different issue, that the spark job is running very long and writing only records in 100 or even less per second at the end of the job(identified by querying the cosmos container manually with time interveal). Which is kind of drags the spark job to run very longer. Even though there are available RUs, the spark job is writing data verly slowly. This is not the case when the "WriteThroughputBudget" is not put in the configuration. Below is the Screen shot of how the RUs usage and request to cosmosdb drops and drags at the end of job.

image

aaronS7 commented 4 months ago

Were you able to find a solution to this?