Shima-Nao
commented
5 months ago @rickmcgeer @mhemmings-engage
June 14 (JST)
seems pyspark installed.
but my script exec failed.
sparktest.ipynb.zip
Py4J Error, something not enough?
Closed Shima-Nao closed 4 months ago
Shima-Nao
commented
5 months ago @rickmcgeer @mhemmings-engage
June 14 (JST)
seems pyspark installed.
but my script exec failed.
sparktest.ipynb.zip
Py4J Error, something not enough?
rickmcgeer
commented
4 months ago Pyspark is now installed in the Spark environment:
jovyan@jupyter-rick-2emcgeer-40engagelively-2ecom:~$ pip list
Package Version
------------------------------ -----------
aiofiles 22.1.0
aiohttp 3.8.4
aiosignal 1.3.1
aiosqlite 0.18.0
alembic 1.10.2
altair 4.2.2
anyio 3.6.2
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
arrow 1.2.3
asn1crypto 1.5.1
asttokens 2.2.1
async-generator 1.10
async-timeout 4.0.2
attrs 22.2.0
Babel 2.12.1
backcall 0.2.0
backports.functools-lru-cache 1.6.4
beautifulsoup4 4.11.2
bleach 6.0.0
blinker 1.5
bokeh 2.4.3
Bottleneck 1.3.7
brotlipy 0.7.0
cached-property 1.5.2
cachetools 5.3.0
certifi 2022.9.24
certipy 0.1.3
cffi 1.15.1
charset-normalizer 2.1.1
click 8.1.3
cloudpickle 2.2.1
colorama 0.4.6
comm 0.1.2
conda 23.1.0
conda-package-handling 2.0.2
conda_package_streaming 0.7.0
contourpy 1.0.7
cryptography 38.0.4
cycler 0.11.0
Cython 0.29.33
cytoolz 0.12.0
dask 2023.3.0
debugpy 1.6.6
decorator 5.1.1
defusedxml 0.7.1
dill 0.3.6
distributed 2023.3.0
entrypoints 0.4
et-xmlfile 1.1.0
executing 1.2.0
fastjsonschema 2.16.3
filelock 3.8.0
Flask 2.2.3
flit_core 3.8.0
fonttools 4.39.0
fqdn 1.5.1
frozenlist 1.3.3
fsspec 2023.3.0
galyleo 2022.12.19
gmpy2 2.1.2
google-api-core 2.11.0
google-auth 2.16.2
google-cloud-bigquery 3.6.0
google-cloud-core 2.3.2
google-crc32c 1.5.0
google-resumable-media 2.4.1
googleapis-common-protos 1.58.0
greenlet 2.0.2
grpcio 1.51.3
grpcio-status 1.51.3
gviz-api 1.10.0
h5py 3.8.0
HeapDict 1.0.1
idna 3.4
imagecodecs 2023.1.23
imageio 2.26.0
importlib-metadata 6.0.0
importlib-resources 5.12.0
ipykernel 6.21.3
ipympl 0.9.3
ipython 8.11.0
ipython-genutils 0.2.0
ipywidgets 8.0.4
isoduration 20.11.0
itsdangerous 2.1.2
jedi 0.18.2
Jinja2 3.1.2
joblib 1.2.0
json5 0.9.5
jsonpointer 2.3
jsonschema 4.17.3
jupyter_client 8.0.3
jupyter_core 5.2.0
jupyter-events 0.6.3
jupyter_server 2.4.0
jupyter_server_fileid 0.8.0
jupyter-server-proxy 3.2.2
jupyter_server_terminals 0.4.4
jupyter_server_ydoc 0.6.1
jupyter-telemetry 0.1.0
jupyter-vscode-server 0.0.0
jupyter-ydoc 0.2.2
jupyterhub 3.1.1
jupyterlab 3.6.1
jupyterlab-language-pack-ja-JP 0.0.1.dev0
jupyterlab-link-share 0.2.5
jupyterlab-pygments 0.2.2
jupyterlab_server 2.20.0
jupyterlab_universal_extension 0.1.0
jupyterlab-widgets 3.0.5
kiwisolver 1.4.4
libmambapy 1.3.1
Library 0.0.0
llvmlite 0.39.1
locket 1.0.0
lz4 4.3.2
Mako 1.2.4
mamba 1.3.1
MarkupSafe 2.1.2
matplotlib 3.7.1
matplotlib-inline 0.1.6
mistune 2.0.5
mpmath 1.3.0
msgpack 1.0.5
multidict 6.0.4
munkres 1.1.4
mysql-connector-python 8.0.32
nbclassic 0.5.3
nbclient 0.7.2
nbconvert 7.2.9
nbformat 5.7.3
nbgitpuller 1.1.1
nest-asyncio 1.5.6
networkx 3.0
notebook 6.5.3
notebook_shim 0.2.2
numba 0.56.4
numexpr 2.8.3
numpy 1.23.5
oauthlib 3.2.2
openpyxl 3.1.1
oscrypto 1.3.0
packaging 23.0
pamela 1.0.0
pandas 1.5.3
pandocfilters 1.5.0
parso 0.8.3
partd 1.3.0
patsy 0.5.3
pexpect 4.8.0
pickleshare 0.7.5
Pillow 9.4.0
pip 23.0.1
pkgutil_resolve_name 1.3.10
platformdirs 3.1.0
pluggy 1.0.0
pooch 1.7.0
prometheus-client 0.16.0
prompt-toolkit 3.0.38
proto-plus 1.22.2
protobuf 3.20.3
psutil 5.9.4
ptyprocess 0.7.0
pure-eval 0.2.2
py4j 0.10.9.7
pyarrow 11.0.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycosat 0.6.4
pycparser 2.21
pycryptodomex 3.16.0
pycurl 7.45.1
Pygments 2.14.0
PyJWT 2.6.0
pyOpenSSL 22.1.0
pyparsing 3.0.9
pyrsistent 0.19.3
PySocks 1.7.1
pyspark 3.5.1
python-dateutil 2.8.2
python-json-logger 2.0.7
pytz 2022.6
PyWavelets 1.4.1
PyYAML 6.0
pyzmq 25.0.0
requests 2.28.1
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rsa 4.9
ruamel.yaml 0.17.21
ruamel.yaml.clib 0.2.7
scikit-image 0.19.3
scikit-learn 1.2.2
scipy 1.10.1
seaborn 0.12.2
Send2Trash 1.8.0
setuptools 67.5.1
simpervisor 0.4
six 1.16.0
sniffio 1.3.0
snowflake-connector-python 2.8.3
sortedcontainers 2.4.0
soupsieve 2.3.2.post1
SQLAlchemy 2.0.5.post1
stack-data 0.6.2
statsmodels 0.13.5
sympy 1.11.1
tables 3.7.0
tblib 1.7.0
terminado 0.17.1
threadpoolctl 3.1.0
tifffile 2023.2.28
tinycss2 1.2.1
tomli 2.0.1
toolz 0.12.0
tornado 6.2
tqdm 4.65.0
traitlets 5.9.0
typing_extensions 4.4.0
unicodedata2 15.0.0
uri-template 1.2.0
urllib3 1.26.13
wcwidth 0.2.6
webcolors 1.12
webencodings 0.5.1
websocket-client 1.5.1
Werkzeug 2.2.3
wheel 0.38.4
widgetsnbextension 4.0.5
xlrd 2.0.1
y-py 0.5.9
yarl 1.8.2
ypy-websocket 0.8.2
zict 2.2.0
zipp 3.15.0
zstandard 0.19.0
@rickmcgeer @mhemmings-engage *I didn't know the best issue repository, so I'll post it here.
Are the following libraries and modules properly installed in the Spark environment? It looks like at least PySpark is not installed.