search

dotnet / spark

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
https://dot.net/spark
MIT License
2.02k stars 312 forks source link

I am facing the following issue: The system cannot find the path specified but my pyspark opens up. #1165

Closed saumyatiwari03 closed 7 months ago

saumyatiwari03 commented 7 months ago 
          Hi @suhsteve, I am facing the similar issue but my pyspark opens up.

image

image

PATH= image

Spark-version: 3.5.0 Java-version: jdk-8u391 Python-version: python-3.9.11

Can someone please help me out?

Originally posted by @saumyatiwari03 in https://github.com/dotnet/spark/issues/557#issuecomment-1960224815

dbeavon commented 7 months ago

@saumyatiwari03 What is the problem you experience? Just the error message?

Is this related to .net programming, or are you a pyspark developer? it would help to have a bit more context.

Also I'd highly recommend using something like process monitor from sysinternals to give us more information about what file is missing. The reason it shows a message saying it cannot find a file, is because it is trying to find a file with a specific name, and those details would appear if you are running process monitor as administrator.

saumyatiwari03 commented 7 months ago

@dbeavon this issue is related to Pyspark. I tried running the process monitor and captured the events while initiating Pyspark.

image

dbeavon commented 7 months ago

Hi @saumyatiwari03 Sorry for the trouble.
This community is focused on .net language bindings for spark (c#) and not on general-purpose spark problems with python. It isn't a good place for normal pyspark Q&A.

I think you are on the right track with the use of "process monitor". You can see that it is making incremental searches to find the files.

Please note that there is lots of noise in process monitor. While you may be looking at something that can't be found, but it is possible that it isn't actually related to the error message. (There could be other things that cant be found as well).

Have you reviewed the installation instructions to see if there are PATH changes need in your environment? I would recommend making sure you have introduced all those changes as well. There are generally changes for both hadoop and spark.

Good luck. Please close the case, since it is not directly related to .net. It is something you could also ask on stackoverflow, perhaps.

saumyatiwari03 commented 7 months ago

Hi @dbeavon, thanks for your help. Yeah, I have certainly verified the environment variables. Thanks for your suggestion.