search

Azure / azure-cosmosdb-spark

Apache Spark Connector for Azure Cosmos DB
MIT License
201 stars 120 forks source link

Spark Application does not exit in client mode because of spark.CosmosDBConnectionCache: throughput-refresh-timer #404

Open txl013 opened 4 years ago

txl013 commented 4 years ago

I am using the azure-cosmosdb-spark connector to write data into cosmosdb from cloudera spark cluster. The connectivity works well and the documents gets created in the cosmos db.

But when I am running the Spark application in client mode, the spark application does not stop even though the processing has finished and the spark session has ended.

The spark driver session in the client mode is kept active by the spark.CosmosDBConnectionCache: throughput-refresh-timer. The info logs related to spark.CosmosDBConnectionCache: throughput-refresh-timer timer refresh keeps on coming and Spark Application does not finish itself.

I tried using System.exit(0) and it works, the application completes sucessfully. But without this, it does not exits. Below are the logs which keep on printing: 20/09/01 19:18:45 INFO spark.CosmosDBConnectionCache: throughput-refresh-timer: ClientConfiguration#-631429096 has been reset - new ClientCacheEntry#1640518163(DocumentClient#702954435, ContainerMetadata#-561333454['n***************s']), previously ClientCacheEntry#1640518163(DocumentClient#702954435, ContainerMetadata#-561333454['n***************s']) 20/09/01 19:33:45 INFO spark.CosmosDBConnectionCache: throughput-refresh-timer: ClientConfiguration#-631429096 has been reset - new ClientCacheEntry#1640518163(DocumentClient#702954435, ContainerMetadata#-561333454['n***************s']), previously ClientCacheEntry#1640518163(DocumentClient#702954435, ContainerMetadata#-561333454['n***************s']) 20/09/01 19:48:45 INFO spark.CosmosDBConnectionCache: throughput-refresh-timer: ClientConfiguration#-631429096 has been reset - new ClientCacheEntry#1640518163(DocumentClient#702954435, ContainerMetadata#-561333454['n***************s']), previously ClientCacheEntry#1640518163(DocumentClient#702954435, ContainerMetadata#-561333454['n***************s']) 20/09/01 20:03:45 INFO spark.CosmosDBConnectionCache: throughput-refresh-timer: ClientConfiguration#-631429096 has been reset - new ClientCacheEntry#1640518163(DocumentClient#702954435, ContainerMetadata#-561333454['n***************s']), previously ClientCacheEntry#1640518163(DocumentClient#702954435, ContainerMetadata#-561333454['n***************s']) 20/09/01 20:18:45 INFO spark.CosmosDBConnectionCache: throughput-refresh-timer: ClientConfiguration#-631429096 has been reset - new ClientCacheEntry#1640518163(DocumentClient#702954435, ContainerMetadata#-561333454['n***************s']), previously ClientCacheEntry#1640518163(DocumentClient#702954435, ContainerMetadata#-561333454['n***************s']) 20/09/01 20:33:45 INFO spark.CosmosDBConnectionCache: throughput-refresh-timer: ClientConfiguration#-631429096 has been reset - new ClientCacheEntry#1640518163(DocumentClient#702954435, ContainerMetadata#-561333454['n***************s']), previously ClientCacheEntry#1640518163(DocumentClient#702954435, ContainerMetadata#-561333454['n***************s']) 20/09/01 20:48:45 INFO spark.CosmosDBConnectionCache: throughput-refresh-timer: ClientConfiguration#-631429096 has been reset - new ClientCacheEntry#1640518163(DocumentClient#702954435, ContainerMetadata#-561333454['n***************s']), previously ClientCacheEntry#1640518163(DocumentClient#702954435, ContainerMetadata#-561333454['n***************s'])