MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure

https://docs.microsoft.com/azure

Creative Commons Attribution 4.0 International

10.22k stars 21.37k forks source link

Synapse SparkPool: Java serialization or Kryo serialization #87651

Closed keen85 closed 3 months ago

keen85 commented 2 years ago

Feedback is concerning the subsection "Optimize data serialization"

Documentations states "Java serialization is the default". However, when running a Synapse Notebook on a 3.1 cluster, Spark UI says:

spark.serializer org.apache.spark.serializer.KryoSerializer

Is the documentation incorrect here? Should it better say that in Synapse, Kyro serialization is the default?

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 322322dc-ff98-c90d-e03c-ceedaa357c43
Version Independent ID: 29642731-3271-b2a9-0330-eded1c70b340
Content: Optimize Spark jobs for performance - Azure Synapse Analytics
Content Source: articles/synapse-analytics/spark/apache-spark-performance.md
Service: synapse-analytics
Sub-service: spark
GitHub Login: @JasonWHowell
Microsoft Alias: jasonh

SnehaGunda commented 2 years ago

reassign: @juluczni

Karishma-Tiwari-MSFT commented 2 years ago

Thanks for the feedback! I have assigned the issue to the content author to investigate further and update the document as appropriate.

bandersmsft commented 4 months ago

Thanks for your dedication to our documentation. Unfortunately, at this time we have been unable to review your issue in a timely manner and we sincerely apologize for the delayed response. We are closing this issue for now, but if you feel that it's still a concern, please respond and let us know. If you determine another possible update to our documentation, please don't hesitate to reach out again. #please-close

keen85 commented 4 months ago

@bandersmsft, Microsofts lack of dedication to its documentation is disappointing. 😞 It is still a concern, just as it was 2 years ago.... Please reopen

bandersmsft commented 4 months ago

please-open

@keen85 Gotcha. I'll work on getting engineering team assistance for this issue.

bandersmsft commented 4 months ago

Requested assistance from article author.

bandersmsft commented 3 months ago

@keen85 The team has updated the article to state the following:

In Synapse Spark (Runtime 3.1 or higher), Kryo data serialization is enabled by default Kryo data serialization.
Java serialization
Kryo serialization is the default. It's a newer format and can result in faster and more compact serialization than Java. Kryo requires that you register the classes in your program, and it doesn't yet support all Serializable types.

Sorry it took so long to address your issue. The article will get refreshed live tomorrow at 10:30 AM PST.

https://github.com/MicrosoftDocs/azure-docs-pr/pull/278733