search

Azure / spark-cdm-connector

MIT License
76 stars 33 forks source link

Exception when writing an empty dataframe to cdm using Spark CDM connector #70

Closed absognety closed 3 years ago

absognety commented 3 years ago

My code for writing to cdm (connector version 0.18.2)

(df.write.format("com.microsoft.cdm")
                  .option("storage", storage_account_name)
                  .option("manifestPath", container_name + "/" + cdm_folder_name + "/default.manifest.cdm.json")
                  .option("entity", entity_name)
                  .option("appId", app_id)
                  .option("appKey", app_key)
                  .option("tenantId", tenant_id)
                  .mode('overwrite')
                  .option("columnHeaders", False)
                  .save())

This is the exception:

An error occurred while calling o1896.save. : java.util.concurrent.CompletionException: java.lang.Exception: PersistenceLayer | Could not convert 'default.manifest.cdm.json'. Reason 'null'. | loadDocumentFromPathAsync at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606) at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Caused by: java.lang.Exception: PersistenceLayer | Could not convert 'default.manifest.cdm.json'. Reason 'null'. | loadDocumentFromPathAsync at com.microsoft.cdm.utils.CDMCallback$.apply(CDMUtils.scala:74) at com.microsoft.commondatamodel.objectmodel.utilities.logger.Logger.log(Logger.java:136) at com.microsoft.commondatamodel.objectmodel.utilities.logger.Logger.error(Logger.java:109) at com.microsoft.commondatamodel.objectmodel.persistence.PersistenceLayer.lambda$loadDocumentFromPathAsync$0(PersistenceLayer.java:249) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ... 5 more

Let me know if you need anything else

absognety commented 3 years ago

We fixed this ourselves, default.manifest.cdm.json got corrupted somehow so, we deleted the data and re-run our pipeline which generated manifest file as expected, closing this.