search

Azure / azure-functions-cosmosdb-extension

Extensions to support CosmosDB functionality on Azure Functions
MIT License
6 stars 7 forks source link

CosmosDB and Databricks communication/integration failure #10

Open FredericoOSS opened 5 months ago

FredericoOSS commented 5 months ago

When we do the integration between CosmosDB reading from Databricks, using the config with inferschema enabled, the data that has a list within a list passes as null in the construction of the dataframe, or even if we have data within the list with the first iteration null, it replicates null for all other iterations. If we read directly from the JSON that is loaded into CosmosDB, we do not have the same problem. In the example of the images we have an iteration in the isOverParcelPayment field with null and the others filled in and when reading via Databricks it replicates null in all fields. How to solve this problem? Read_Dataframe

InferchemaEnabled notNULL notNULL_2 NULL

asos-martinsmith commented 4 months ago

This issue doesn't seem to be anything to do with the topic of this repository (which is a CosmosDB extension to the function app and has nothing to do with data bricks)