search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.22k stars 21.37k forks source link

Synapse workspace is not showing results - even if spark pool is running #119489

Closed V4A001 closed 1 month ago

V4A001 commented 7 months ago

From the documentation it is not clear how to debug properly the setup. To my idea I have setup everything. image

What is also a simple query to run to start the spark pool from running. I now have every hour a job running which ends into errors. For that reason I want to get more information out of the spark pool to see what is wrong. If the problem is there by the way, as the support teams from India are not of a great help till now.

Note; you have various teams across every technology stack, and maybe forget that on the other end somebody needs to setup every part individually to get it running. So help appreciated.

[Enter feedback here]

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

AjayBathini-MSFT commented 7 months ago

@V4A001 Thanks for your feedback! We will investigate and update as appropriate.

Naveenommi-MSFT commented 7 months ago

@Jejiang I am facing the same issue, while reproduce the issue. Could you please review it and add comments on this, update as appropriate.

image
Naveenommi-MSFT commented 7 months ago

@Jejiang Please review it.

Naveenommi-MSFT commented 7 months ago

@V4A001 Thank you for bringing this to our attention. I've delegated this to content author @Jejiang, who will review it and offer their insightful opinions.

nikkolas-irwin commented 6 months ago

Hello Microsoft,

Federal government customer here experiencing the same issue described by @V4A001. We have checked and rechecked our entries within our Apache Spark pools following Step 2 Option 1 as described here. Additionally, we made sure in our case, to add an additional parameter, as noted in the documentation for the spark.synapse.logAnalytics.uriSuffix with value ods.opinsights.azure.us.

We have an existing Log Analytics workspace and we've configured diagnostic settings for our instance of Synapse. After adding the four parameters mentioned above, we are able to see some new logging information using the Log Analytics workspace KQL query view, however, this data is not propagated to the Azure Synapse Spark Application workbook.

When viewing the Log Analytics workspace Logs tab KQL query window we can also see the SparkListenerEvent_CL, SparkLoggingEvent_CL, and SparkMetrics_CL objects, but again, this data is not displayed in the workbook that we create in Azure Monitor. The same three drop-down lists are greyed out. As shown in @V4A001's image. Anyhow, I hope this information is helpful and brings further attention to this issue. Thank you for your time and assistance.

andrehociko commented 5 months ago

Hello @Jejiang

After following the document, I still cannot see the custom logs in my LA workspace. Also, even after applying the new configuration to the Spark Pool, it just revert back to the default configuration after a while.

Is this guide still valid? It looks deprecated... Please, advise!

bandersmsft commented 4 months ago

Followed up with aticle author.

Jejiang commented 2 months ago

We followed the steps in the article, and everything seems to be working properly.

Jejiang commented 2 months ago

@nikkolas-irwin @andrehociko @V4A001 @Naveenommi-MSFT I am not able to reproduce the issues you have reported in this request. Can we please have a quick call to go over your issues. Feel free to ping or email me at: jejiang@microsoft.com

SnehaGunda commented 1 month ago

As there is no action item from the docs side, we will close this issue.

SnehaGunda commented 1 month ago

please-lcose

SnehaGunda commented 1 month ago

please-close

nikkolas-irwin commented 4 days ago

In coordination with Microsoft Support we were able to identify the issue with Synapse Spark Pool workbooks. These have to be enabled in under a Log Analytics workspace, whereas the Synapse SQL Pool workbooks can be enabled in Azure Monitor. Essentially it is a scoping issue. Hopefully this helps others who run into the same issue.