ag-ramachandran
commented
9 hours ago Hello @Sandeepvirkar
Will have a look, the connector itself does not have an issue with the count. With LA , I will try and do a replication though.
What version of the connector do you use ? I will start from there and have a look
We are trying to query log analtics tables using databricks. We have established connection between databricks and log analytics workspace using service principle. We are able to query the table with intake value 4000. But if we increase intake value more than 5000. It is failing with the error. We checked this with internal network team. there is no network issue. But we are not able to understand strange behavior for intake value more than 5000. We are using below code in databricks.
subscription = "" resource_group = "" log_analytics_workspace = "" client_id = "" client_secret = dbutils.secrets.get(scope="",key="") tenant_id = ""
df = spark.read.format("com.microsoft.kusto.spark.datasource") \ .option("kustoCluster", f"https://ade.loganalytics.io/subscriptions/{subscription}/resourceGroups/{resource_group}/providers/Microsoft.OperationalInsights/workspaces/{log_analytics_workspace}") \ .option("kustoDatabase", log_analytics_workspace) \ .option("kustoQuery", "AADManagedIdentitySignInLogs | take 4000") \ .option("kustoAadAppId", client_id) \ .option("kustoAadAppSecret", client_secret) \ .option("kustoAadAuthorityID", tenant_id) \ .option("KustoSourceOptions.KUSTO_READ_MODE", "ForceSingleMode") \ .load()
display(df)
Error message when take value is greater than 5000.
com.microsoft.azure.kusto.data.exceptions.DataServiceException: IOException when trying to retrieve cluster metadata:ingest-ade.loganalytics.io: Name or service not known