I'm trying to use a remote spark cluster, which is defined within a Synapse Workspace environment - I'm able to send batch jobs to Synapse, but when trying to use a pyspark interactive window I get the following error for any command I try to send:
An error was encountered:
Invalid status code '500' from https://***.dev.azuresynapse.net/livyApi/versions/2019-11-01-preview/sparkPools/***/sessions/-1 with error payload: {"TraceId":"***","Message":"Something went wrong while processing your request. Please try again later."}
and when I try to access the link within the error I get the following message:
{"code":"BearerTokenNotFound","message":"BearerToken not found in request"}
edit: The guide linked above mentions that this extension doesn't support ms-python versions >= v2020.5.78807, I downgraded to v2020.4.76186 and now I get the following error while trying to initialize a Synapse Pyspark interactive window:
Error: Unable to write python.defaultInterpreterPath to Workspace Settings. This setting can be written only into User settings.
I'm trying to use a remote spark cluster, which is defined within a Synapse Workspace environment - I'm able to send batch jobs to Synapse, but when trying to use a pyspark interactive window I get the following error for any command I try to send:
An error was encountered:
Invalid status code '500' from https://***.dev.azuresynapse.net/livyApi/versions/2019-11-01-preview/sparkPools/***/sessions/-1 with error payload: {"TraceId":"***","Message":"Something went wrong while processing your request. Please try again later."}and when I try to access the link within the error I get the following message:
{"code":"BearerTokenNotFound","message":"BearerToken not found in request"}What might be the reason for the 500 error above?
I tried to use the following guide in order to support remote access to Synapse spark pools: Tutorial - Spark & Hive Tools for VSCode (Spark application) - Azure Synapse Analytics | Microsoft Docs
Some technical details:
edit: The guide linked above mentions that this extension doesn't support ms-python versions >= v2020.5.78807, I downgraded to v2020.4.76186 and now I get the following error while trying to initialize a Synapse Pyspark interactive window:
Error: Unable to write python.defaultInterpreterPath to Workspace Settings. This setting can be written only into User settings.