Can't install sparkmagic from pypi

[GiorgiKandelaki-de]

Describe the bug The installation error occurred when installed sparkmagic.

...
...
 • Installing cryptography (37.0.2)
 • Installing gssapi (1.7.3): Failed

  EnvCommandError

  Command ['~/.venv/bin/pip', 'install', '--no-deps', '~/.cache/pypoetry/artifacts/fe/e7/bb/9f1204d0e49af95bc7a0ada1f736a5c11f49bf9aba42382d749949fb45/gssapi-1.7.3.tar.gz'] errored with the following return code 1, and output: 
  Processing ~/.cache/pypoetry/artifacts/fe/e7/bb/9f1204d0e49af95bc7a0ada1f736a5c11f49bf9aba42382d749949fb45/gssapi-1.7.3.tar.gz
    Preparing metadata (setup.py): started
    Preparing metadata (setup.py): finished with status 'error'
    error: subprocess-exited-with-error

    × python setup.py egg_info did not run successfully.
    │ exit code: 1
    ╰─> [14 lines of output]
        /bin/sh: 1: krb5-config: not found
        In distributed package, building from C files...
        Traceback (most recent call last):
          File "<string>", line 2, in <module>
          File "<pip-setuptools-caller>", line 34, in <module>
          File "/tmp/pip-req-build-ldcjbt4l/setup.py", line 128, in <module>
            link_args = shlex.split(get_output(f"{kc} --libs gssapi"))
          File "/tmp/pip-req-build-ldcjbt4l/setup.py", line 41, in get_output
            res = subprocess.check_output(*args, shell=True, **kwargs)
          File "/usr/local/lib/python3.8/subprocess.py", line 415, in check_output
            return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
          File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
            raise CalledProcessError(retcode, process.args,
        subprocess.CalledProcessError: Command 'krb5-config --libs gssapi' returned non-zero exit status 127.
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: metadata-generation-failed

  × Encountered error while generating package metadata.
  ╰─> See above for output.

  note: This is an issue with the package mentioned above, not pip.
  hint: See above for details.`

and getting errors from the krb5 side

• Installing jupyter (1.0.0)
• Installing krb5 (0.3.0): Failed

  EnvCommandError

  Command ['/home/giorgi/development/spark-implementation/HARP-Spark-Implementation/harp-spark/.venv/bin/pip', 'install', '--no-deps', '/home/giorgi/.cache/pypoetry/artifacts/89/8e/9f/ce3783f19974a8adb3ea0df3913c40c39955f711fa3d571cb8ef9f82ea/krb5-0.3.0.tar.gz'] errored with the following return code 1, and output: 
  Processing /home/giorgi/.cache/pypoetry/artifacts/89/8e/9f/ce3783f19974a8adb3ea0df3913c40c39955f711fa3d571cb8ef9f82ea/krb5-0.3.0.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    error: subprocess-exited-with-error

    × Getting requirements to build wheel did not run successfully.
    │ exit code: 1
    ╰─> [25 lines of output]
        /bin/sh: 1: krb5-config: not found
        In distributed package, building from C files...
        Using krb5-config at 'krb5-config'
        Traceback (most recent call last):
          File "~/.venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
            main()
          File "~/.venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "~/.venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
            return hook(config_settings)
          File "/tmp/pip-build-env-3b4pm30u/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 177, in get_requires_for_build_wheel
            return self._get_build_requires(
          File "/tmp/pip-build-env-3b4pm30u/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 159, in _get_build_requires
            self.run_setup()
          File "/tmp/pip-build-env-3b4pm30u/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 174, in run_setup
            exec(compile(code, __file__, 'exec'), locals())
          File "setup.py", line 190, in <module>
            compile_args = shlex.split(run_command(f"{kc} --cflags krb5"))
          File "setup.py", line 47, in run_command
            stdout = subprocess.check_output(args, shell=True)
          File "/usr/local/lib/python3.8/subprocess.py", line 415, in check_output
            return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
          File "/usr/local/lib/python3.8/subprocess.py", line 516, in run
            raise CalledProcessError(retcode, process.args,
        subprocess.CalledProcessError: Command '('krb5-config --cflags krb5',)' returned non-zero exit status 127.
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.

To Reproduce I'm using:

python 3.8.10 Distributor ID: Debian Description: Debian GNU/Linux 10 (buster) Release: 10 Codename: buster

Versions:

• SparkMagic 0.20.0
• Livy 0.7.1
• Spark 3.1.1

[dpfrakes]

Try sudo apt install libkrb5-dev, then re-run pip install sparkmagic.

I was having the same issue on Ubuntu 20.04. I tried running the line that failed directly in zsh:

subprocess.CalledProcessError: Command '('krb5-config --cflags krb5',)' returned non-zero exit status 127.

Running that, I got:

$ krb5-config --cflags krb5
zsh: command not found: krb5-config

which led me to https://stackoverflow.com/a/50410443/2727510

[GiorgiKandelaki-de]

@dpfrakes Thanks, it helped.

[Y3NK]

Hello, it seems that I have a similar issue on my side:

Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /apps/python/venv_edw/lib/python3.9/site-packages (from packaging->ipykernel>=4.2.2->sparkmagic) (3.0.9)
Collecting gssapi>=1.6.0
  Downloading gssapi-1.8.2.tar.gz (94 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 94.3/94.3 kB 9.1 MB/s eta 0:00:00
  Running command pip subprocess to install build dependencies
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f8c4c0c43a0>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/cython/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f8c4c0c4550>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/cython/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f8c4c0c4ac0>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/cython/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f8c4c0c4c70>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/cython/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f8c4c0c4e20>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/cython/
  ERROR: Could not find a version that satisfies the requirement Cython<3.0.0,>=0.29.29 (from versions: none)
  ERROR: No matching distribution found for Cython<3.0.0,>=0.29.29
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /apps/python/venv_edw/bin/python /apps/python/venv_edw/lib/python3.9/site-packages/pip/__pip-runner__.py install --ignore-installed --no-user --prefix /tmp/pip-build-env-d4ieripm/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org -- 'Cython >= 0.29.29, < 3.0.0' 'setuptools >= 40.6.0'
  cwd: [inherit]
  Installing build dependencies ... error
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

My objective is to get sparkmagic in order to be able to connect jupyterlab to a Spark cluster (on YARN in cluster mode). I installed these packages:

Package krb5-server-1.18.2-22.el8_7.x86_64 is already installed.
Package krb5-libs-1.18.2-22.el8_7.x86_64 is already installed.
Package krb5-devel-1.18.2-22.el8_7.x86_64 is already installed.

But something is not passing through....

I am behind a corporate proxy, but firewall and exceptions should be alright (all others packages are fine).

I am on Red Hat 8

Regarding the previous link who should help, these files already exist: /usr/bin/krb5-config /usr/lib/libgssapi_krb5.so /usr/lib64/libgssapi_krb5.so

so I don't need to recreate symbolic link to them I think...

If I try to do a pip install --proxy=http://superproxy:8080 --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org gssapi --verbose It stick to

Collecting gssapi
  Using cached gssapi-1.8.2.tar.gz (94 kB)
  Running command pip subprocess to install build dependencies

for a long time then I got:

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fd7f103d2e0>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/cython/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fd7f103d490>: Failed to establish a new connection: [Errno 101] Network is unreachable')': /simple/cython/

Have you any idea ?

I am trying to use livy to send pyspark jobs to a Spark cluster in "deploy mode = cluster" directly through jupyterlab. (so without spark-submit...) I gave a try to Zeppelin but without success and if we can stick to jupyterlab, it's better (Zeppelin doesn't seems to have that much of mainteance and I don't saw a lot of feedbacks using it in production, even if the platform seems great...)

Many thanks for your help!

KR

[devstein]

Hi @Y3NK sorry to hear you are having issues. The error indicates it's likely a system dependencies error. It's hard to debug without having access to your exact environment. No recommendations other than verifying your python and pip installation and make sure your system packages are up-to-date.