Error in Notebook code

[bjarkimg]

Hi

I had to add this code in #Settings part of the Fabric Notebook code. We had 1 table with more than 100 columns (135) so we got error on that. check the error and found this solution to it.

spark.conf.set("spark.sql.analyzer.maxIterations", "300") # Default is 100, got error if more than 100 columns

Anyone had the same issue?

-Bjarki

[bjarkimg]

Error is here: Py4JJavaError Traceback (most recent call last) Cell In[17], line 72 70 df_deletes = df_new.filter(df_new['SystemCreatedAt-2000000001'].isNull()) 71 if ContainsCompany: ---> 72 df_new = df_new.join(df_deletes, ['$Company','systemId-2000000000'], 'leftanti') 73 else: 74 df_new = df_new.join(df_deletes, ['systemId-2000000000'], 'leftanti')

File /opt/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py:2346, in DataFrame.join(self, other, on, how) 2344 on = self._jseq([]) 2345 assert isinstance(how, str), "how should be a string" -> 2346 jdf = self._jdf.join(other._jdf, on, how) 2347 return DataFrame(jdf, self.sparkSession)

File ~/cluster-env/trident_env/lib/python3.10/site-packages/py4j/java_gateway.py:1322, in JavaMember.call(self, *args) 1316 command = proto.CALL_COMMAND_NAME +\ 1317 self.command_header +\ 1318 args_command +\ 1319 proto.END_COMMAND_PART 1321 answer = self.gateway_client.send_command(command) -> 1322 return_value = get_return_value( 1323 answer, self.gateway_client, self.target_id, self.name) 1325 for temp_arg in temp_args: 1326 if hasattr(temp_arg, "_detach"):

File /opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py:169, in capture_sql_exception..deco(*a, kw) 167 def deco(*a: Any, *kw: Any) -> Any: 168 try: --> 169 return f(a, kw) 170 except Py4JJavaError as e: 171 converted = convert_exception(e.java_exception)

File ~/cluster-env/trident_env/lib/python3.10/site-packages/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name) 324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client) 325 if answer[1] == REFERENCE_TYPE: --> 326 raise Py4JJavaError( 327 "An error occurred while calling {0}{1}{2}.\n". 328 format(target_id, ".", name), value) 329 else: 330 raise Py4JError( 331 "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n". 332 format(target_id, ".", name, value))

Py4JJavaError: An error occurred while calling o5249.join. : java.lang.RuntimeException: Max iterations (100) reached for batch Resolution, please set 'spark.sql.analyzer.maxIterations' to a larger value. at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:265) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:211) at scala.collection.immutable.List.foreach(List.scala:431) at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:211) at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:230) at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$execute$1(Analyzer.scala:226) at org.apache.spark.sql.catalyst.analysis.AnalysisContext$.withNewAnalysisContext(Analyzer.scala:175) at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:226) at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:190) at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:182) at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:93) at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:182) at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:211) at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:330) at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:210) at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:120) at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:120) at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$2(QueryExecution.scala:288) at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:642) at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:288) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827) at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:287) at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:120) at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:118) at org.apache.spark.sql.Dataset.join(Dataset.scala:1068) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:238) at java.base/java.lang.Thread.run(Thread.java:829)

[bjarkimg]

and ChatGPT solution :)

[Bertverbeek4PS]

Hi @bjarkimg never seen that issue. But moslty a table with 100 columns that is huge! You can create a pull request. But then the initial value I would set to 100.

[bjarkimg]

@Bertverbeek4PS will do.

And yes over 100 columns is allot, it was more of a test that i did, and yes tried 200 value, in my case but needed to go higher, but lets have default at 100.