Anaconda versioning in Synapse Spark pools

[VitalyMCT]

Can some details please be added around this important aspect?

As an example, Spark 3.3 (preview) pools currently include the pandas 1.2.3 library:

According to https://pypi.org/project/pandas/1.2.3/#history that version was released on March 2, 2021.

It's unclear to me when 1.2.3 was brought over into Anaconda, as I cannot find references to 1.2.3 on Anaconda's pages. However, it was probably sometime on or before Nov 17, 2021: https://docs.continuum.io/anaconda/reference/release-notes. An upgrade from 1.2.4 is referenced on that page for at least one target OS platform.

To continue with this example, the current Anaconda distribution includes a much newer version of pandas.

So the questions include:

• Which Anaconda versions are used for each Synapse Spark pool runtime version?
• Is this mapping frozen at the time a particular Synapse Spark runtime is released to GA?
• If not, how often are upgrades done and how to know when they happen?
• How to reliably and easily find out which Anaconda distribution is deployed on a given Azure Synapse Spark pool at a given time?

---

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

• ID: 315cbf88-9737-8e5d-12ab-d6988bbcfb84
• Version Independent ID: 8f8a1b52-c73d-9e35-a8f4-8825b6bbb4d2
• Content: Apache Spark version support - Azure Synapse Analytics
• Content Source: articles/synapse-analytics/spark/apache-spark-version-support.md
• Service: synapse-analytics
• Sub-service: spark
• GitHub Login: @eskot
• Microsoft Alias: eskot

[SaibabaBalapur-MSFT]

@VitalyMCT Thanks for your feedback! We will investigate and update as appropriate.

[RamanathanChinnappan-MSFT]

@VitalyMCT

I've delegated this to content author @MikeRys to review and share his valuable insights.

[MikeRys]

I am sorry but generally I am not really the right person for Synapse questions. In this case, all I can say is that each Spark version that we release in Synapse normally comes with a predefined set of built-in libraries where the chosen library version is fixed to that Spark version. The libraries and their versions are documented as follows:

1. The Spark versions are documented at https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-version-support.
2. This page links to more details about each version and their supported libraries. For example, Spark v3.3 is at https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-33-runtime.

Finally, there is documentation about library management and how to add your own libraries at https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-azure-portal-add-libraries.

[bandersmsft]

Thanks for your dedication to our documentation. Unfortunately, at this time we have been unable to review your issue in a timely manner and we sincerely apologize for the delayed response. We are closing this issue for now, but if you feel that it's still a concern, please respond and let us know. If you determine another possible update to our documentation, please don't hesitate to reach out again. #please-close