Sparklyr code is not working for Apache Spark version 3.3

[bernobfer]

When using a Spark Pool with version 3.3, the following piece of code provided by the documentation:

%%sparkr
spark_version = sparkR.version()
config = spark_config()
sc = spark_connect(master = 'yarn', version = spark_version, spark_home = '/opt/spark', config = config)

gives the following error:

Error: [1] 'Error: org.apache.spark.SparkException: The Spark SQL phase analysis failed with an internal error. Please, fill a bug report in, and provide the full stack trace.....

Basically any operation that includes the object sc, fails with the above error, with the exception of spark_connection_is_open(sc) which yields the value TRUE.

By adding the argument method='synapse', like this:

%%sparkr
spark_version = sparkR.version()
config = spark_config()
sc = spark_connect(master = 'yarn', version = spark_version, spark_home = '/opt/spark', config = config, method='synapse')

The error does not appear any longer. However this is not reflected on the documentation.

---

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

• ID: 94f90a2e-f64f-0cd9-30f4-4c0c059b9491
• Version Independent ID: 797f4bcf-fde9-7d05-5314-020dd9262d14
• Content: Use R for Apache Spark - Azure Synapse Analytics
• Content Source: articles/synapse-analytics/spark/apache-spark-r-language.md
• Service: synapse-analytics
• Sub-service: spark
• GitHub Login: @midesa
• Microsoft Alias: midesa

[AjayBathini-MSFT]

@bernobfer Thanks for your feedback! We will investigate and update as appropriate.

[RamanathanChinnappan-MSFT]

@bernobfer Thanks for reporting this! We have created a PR for this issue and the changes should go live soon.