search

jupyter-incubator / sparkmagic

Jupyter magics and kernels for working with remote Spark clusters
Other
1.33k stars 447 forks source link

AttributeError: 'tuple' object has no attribute 'major' #599

Closed just-JL closed 4 years ago

just-JL commented 4 years ago

image

image hi, i am facing this question. When i execute this code in PySpark, The session state is idle and the executors of spark ui only has a driver. I am confused. Thanks in advance.

itamarst commented 4 years ago

The way sparkmagic's SQL works is it sends Python code to PySpark to execute. Not ideal, but that's how it happens now. And part of that code detects the Python version.

However... 'tuple' object has no attribute 'major' shouldn't happen in Python 2.7 or later, only in Python 2.6 or earlier. And PySpark 2.3.1 doesn't use Python 2.6... so it's a little confusing.

Can you do "import sys; print(sys.version_info)" in a normal Python Sparkmagic cell (on the server-side)?

just-JL commented 4 years ago

image it is (2, 6, 6, 'final', 0)

itamarst commented 4 years ago

Yeah, Python 2.6 is not supported (Python 2.7 was released in 2010).

Python 2 support in general is going to end sometime next year, so I strongly recommend starting the switch to Python 3.

sinosliver commented 1 year ago

The way sparkmagic's SQL works is it sends Python code to PySpark to execute. Not ideal, but that's how it happens now. And part of that code detects the Python version.

However... 'tuple' object has no attribute 'major' shouldn't happen in Python 2.7 or later, only in Python 2.6 or earlier. And PySpark 2.3.1 doesn't use Python 2.6... so it's a little confusing.

Can you do "import sys; print(sys.version_info)" in a normal Python Sparkmagic cell (on the server-side)?

It occured 'tuple' object has no attribute 'major' when I tried to execute datax.py which might depend on PySpark, or not, to complete the setting of dataX.
After I read your comment, I changed my python from 2.6 to 2.7. Then the problem had been solved.