search

microsoft / SynapseVSCode

this is the repo of the Synapse VS Code extension for Microsoft Fabric
MIT License
29 stars 5 forks source link

Java heap space out of memory #30

Open Jassi95 opened 7 months ago

Jassi95 commented 7 months ago

When running a notebook from Vscode I get this error:

Py4JJavaError: An error occurred while calling o41.load. : java.util.concurrent.ExecutionException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

I have tried this, but it doesn't work: from pyspark.sql import SparkSession

spark = SparkSession.builder \ .appName('app') \ .config('spark.driver.memory', '4g') \ .getOrCreate()

When running the same code in Fabrics web UI there is no problems. This problem occurred suddenly, there were no problems last week.

PotatoLu666 commented 6 months ago

Hi @Jassi95 ,

Thanks for your feedback! The code cannot run due to insufficient memory space. Your config is work for local spark, not a remote spark cluster. Since the extension automatically creating the Spark session does not carry the manually set configurations, the code fails to run properly. I recommend writing your custom configurations into the environment artifact Spark properties. Here are the specific steps:

  1. Test the required memory size on the Fabric Portal.
  2. Write the memory configuration into the environment artifact.
  3. Associate the environment with your notebook.
  4. Run the notebook in VSCode.

Please test it and contract us if you have any problem.

Thanks!