search

qubole / rubix-admin

Admin scripts for Rubix
Apache License 2.0
5 stars 5 forks source link

Installing Rubix with Spark issue #25

Open amrfawzy-incorta opened 4 years ago

amrfawzy-incorta commented 4 years ago

We were trying to install rubix using the following installation guide

git clone https://github.com/qubole/rubix-admin.git
cd rubix-admin
virtualenv venv
source venv/bin/activate
python setup.py install

installed rubix-admin version is:

After successfully installing the rubix_admin, I tried to install the latest version of RubiX as following: rubix_admin installer install --cluster-type spark

We encounter the following issue in the o/p:

2020-08-06 16:03:12 (40.7 MB/s) - ‘/tmp/qubole-rubix-stable.noarch.rpm’ saved [13535563/13535563]

2020-08-06 16:03:12,552 - installer - INFO - Installing package /tmp/qubole-rubix-stable.noarch.rpm
warning: /tmp/rubix_rpms/qubole-rubix-stable.noarch.rpm: Header V4 RSA/SHA1 Signature, key ID fc7b37a2: NOKEY
    package qubole-rubix-0.3.1-1.noarch is already installed
Traceback (most recent call last):
  File "/home/incorta/rubix-admin/venv/bin/rubix_admin", line 11, in <module>
    load_entry_point('rubix-admin==0.4.0+7.g0419d0c.dirty', 'console_scripts', 'rubix_admin')()
  File "build/bdist.linux-x86_64/egg/rubix_admin/command_line.py", line 46, in main
  File "build/bdist.linux-x86_64/egg/rubix_admin/installer.py", line 33, in install_cmd
  File "build/bdist.linux-x86_64/egg/rubix_admin/installer.py", line 48, in install
  File "build/bdist.linux-x86_64/egg/rubix_admin/installer.py", line 89, in _rpm_install
  File "<decorator-gen-4>", line 2, in sudo
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/fabric-2.4.0-py2.7.egg/fabric/connection.py", line 30, in opens
    return method(self, *args, **kwargs)
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/fabric-2.4.0-py2.7.egg/fabric/connection.py", line 716, in sudo
    return self._sudo(self._remote_runner(), command, **kwargs)
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/invoke-1.2.0-py2.7.egg/invoke/context.py", line 212, in _sudo
    return runner.run(cmd_str, watchers=watchers, **kwargs)
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/invoke-1.2.0-py2.7.egg/invoke/runners.py", line 271, in run
    return self._run_body(command, **kwargs)
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/invoke-1.2.0-py2.7.egg/invoke/runners.py", line 404, in _run_body
    raise UnexpectedExit(result)
invoke.exceptions.UnexpectedExit: Encountered a bad command exit code!

Command: "sudo -S -p '[sudo] password: ' rpm -U --ignoreos /tmp/rubix_rpms/qubole-rubix-stable.noarch.rpm"

Exit code: 1

Stdout: already printed

Stderr: already printed

List of files in the installed qubole-rubix-stable.noarch.rpm are:

/etc/rubix
/etc/rubix/log4j.properties
/etc/rubix/log4j_lds.properties
/usr/lib/rubix/bin
/usr/lib/rubix/bin/configure.sh
/usr/lib/rubix/bin/rubix.service
/usr/lib/rubix/lib
/usr/lib/rubix/lib/rubix-bookkeeper-0.3.1.jar
/usr/lib/rubix/lib/rubix-core-0.3.1.jar
/usr/lib/rubix/lib/rubix-hadoop2-0.3.1.jar
/usr/lib/rubix/lib/rubix-presto-0.3.1.jar
/usr/lib/rubix/lib/rubix-rpm-0.3.1.jar
/usr/lib/rubix/lib/rubix-spi-0.3.1.jar
/var/log/rubix
/var/run/rubix

After searching we found another version of qubole-rubix package that has configure-spark.sh file which is not listed in the installed package.

Environment details:

Are there any recommended versions of Spark, Hadoop, Python other than the mentioned installed versions ?

Thanks in advance

shubhamtagra commented 4 years ago

I will upload the latest rpms and get back on this.

About the recommended versions, Spark3 hasn't been tested yet. Most of the users we have seen are on Spark2.4 with hadoop2 (hadoop3 should also work fine).

amrfawzy-incorta commented 4 years ago

Thanks for your prompt response.

I have updated the environment to be as following:

Still unfortunately getting the same error.

Looking forward to using the latest rpm.

shubhamtagra commented 4 years ago

I have updated the rpm, you can give it another shot.

amrfawzy-incorta commented 4 years ago

Thanks a lot for your response and cooperation.

I have tried the installation again using both ways From Source and From PIP and got the following output

From Source method:

when running $ python setup.py install I got the following errors

In file included from ext/_yaml.c:596:0:
ext/_yaml.h:2:18: fatal error: yaml.h: No such file or directory
 #include <yaml.h>

SyntaxError: invalid syntax

creating /home/incorta/rubix-admin/venv/lib/python2.7/site-packages/bcrypt-3.2.0-py2.7-linux-x86_64.egg
Extracting bcrypt-3.2.0-py2.7-linux-x86_64.egg to /home/incorta/rubix-admin/venv/lib/python2.7/site-packages
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/bcrypt-3.2.0-py2.7-linux-x86_64.egg/bcrypt/__init__.py", line 57
    def gensalt(rounds: int = 12, prefix: bytes = b"2b") -> bytes:
                      ^
SyntaxError: invalid syntax

And got the following errors when running $ rubix_admin -h

Traceback (most recent call last):
  File "/home/incorta/rubix-admin/venv/bin/rubix_admin", line 11, in <module>
    load_entry_point('rubix-admin==0.4.0+7.g0419d0c.dirty', 'console_scripts', 'rubix_admin')()
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "build/bdist.linux-x86_64/egg/rubix_admin/command_line.py", line 7, in <module>
  File "build/bdist.linux-x86_64/egg/rubix_admin/parsers.py", line 4, in <module>
  File "build/bdist.linux-x86_64/egg/rubix_admin/installer.py", line 4, in <module>
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/fabric-2.4.0-py2.7.egg/fabric/__init__.py", line 3, in <module>
    from .connection import Config, Connection
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/fabric-2.4.0-py2.7.egg/fabric/connection.py", line 16, in <module>
    from paramiko.agent import AgentRequestHandler
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/paramiko-2.7.1-py2.7.egg/paramiko/__init__.py", line 22, in <module>
    from paramiko.transport import SecurityOptions, Transport
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/paramiko-2.7.1-py2.7.egg/paramiko/transport.py", line 89, in <module>
    from paramiko.dsskey import DSSKey
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/paramiko-2.7.1-py2.7.egg/paramiko/dsskey.py", line 37, in <module>
    from paramiko.pkey import PKey
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/paramiko-2.7.1-py2.7.egg/paramiko/pkey.py", line 31, in <module>
    import bcrypt
  File "/home/incorta/rubix-admin/venv/lib/python2.7/site-packages/bcrypt-3.2.0-py2.7-linux-x86_64.egg/bcrypt/__init__.py", line 57
    def gensalt(rounds: int = 12, prefix: bytes = b"2b") -> bytes:
                      ^
SyntaxError: invalid syntax

Form PIP installation method:

$ pip install rubix-admin worked fine with no errors $ rubix_admin -h worked fine and generate ~/.radminrc file with the following content

hosts:
- localhost
remote_packages_path: /tmp/rubix_rpms

This is not the same as the configuration file in the documentation for the used rubix-admin version I didn't change the content of this file Do we need to make any changes in this configuration file for a stand alone spark installation on one host ?

When running $ rubix_admin installer install --cluster-type spark I got the following errors


{'remote_packages_path': '/tmp/rubix_rpms', 'hosts': ['localhost']}
2020-08-18 22:17:05,733 - installer - INFO - Installing packages None
[localhost] Executing task 'install'
2020-08-18 22:17:05,733 - installer - INFO - Installing Rubix from default location
[localhost] sudo: wget https://s3.amazonaws.com/public-qubole/rubix/rpms/qubole-rubix-stable.noarch.rpm -O /tmp/qubole-rubix-stable.noarch.rpm
Traceback (most recent call last):
  File "/home/incorta/.local/bin/rubix_admin", line 8, in <module>
    sys.exit(main())
  File "/home/incorta/.local/lib/python2.7/site-packages/rubix_admin/command_line.py", line 46, in main
    args.func(args)
  File "/home/incorta/.local/lib/python2.7/site-packages/rubix_admin/installer.py", line 31, in install_cmd
    return execute(cls.install, args, hosts=args.config["hosts"])
  File "/home/incorta/.local/lib/python2.7/site-packages/fabric/tasks.py", line 386, in execute
    multiprocessing
  File "/home/incorta/.local/lib/python2.7/site-packages/fabric/tasks.py", line 276, in _execute
    return task.run(*args, **kwargs)
  File "/home/incorta/.local/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
    return self.wrapped(*args, **kwargs)
  File "/home/incorta/.local/lib/python2.7/site-packages/rubix_admin/installer.py", line 35, in install
    cls._scp(args)
  File "/home/incorta/.local/lib/python2.7/site-packages/rubix_admin/installer.py", line 55, in _scp
    rpm_path = cls.get_rpm_path(args)
  File "/home/incorta/.local/lib/python2.7/site-packages/rubix_admin/installer.py", line 47, in get_rpm_path
    sudo('wget https://s3.amazonaws.com/public-qubole/rubix/rpms/' + rpm_file_name + ' -O /tmp/' + rpm_file_name)
  File "/home/incorta/.local/lib/python2.7/site-packages/fabric/network.py", line 683, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/home/incorta/.local/lib/python2.7/site-packages/fabric/operations.py", line 1148, in sudo
    capture_buffer_size=capture_buffer_size,
  File "/home/incorta/.local/lib/python2.7/site-packages/fabric/operations.py", line 930, in _run_command
    channel=default_channel(), command=wrapped_command, pty=pty,
  File "/home/incorta/.local/lib/python2.7/site-packages/fabric/state.py", line 424, in default_channel
    chan = _open_session()
  File "/home/incorta/.local/lib/python2.7/site-packages/fabric/state.py", line 416, in _open_session
    return connections[env.host_string].get_transport().open_session()
  File "/home/incorta/.local/lib/python2.7/site-packages/fabric/network.py", line 159, in __getitem__
    self.connect(key)
  File "/home/incorta/.local/lib/python2.7/site-packages/fabric/network.py", line 151, in connect
    user, host, port, cache=self, seek_gateway=seek_gateway)
  File "/home/incorta/.local/lib/python2.7/site-packages/fabric/network.py", line 434, in connect
    client.load_system_host_keys()
  File "/home/incorta/.local/lib/python2.7/site-packages/paramiko/client.py", line 104, in load_system_host_keys
    self._system_host_keys.load(filename)
  File "/home/incorta/.local/lib/python2.7/site-packages/paramiko/hostkeys.py", line 101, in load
    e = HostKeyEntry.from_line(line, lineno)
  File "/home/incorta/.local/lib/python2.7/site-packages/paramiko/hostkeys.py", line 364, in from_line
    key = ECDSAKey(data=decodebytes(key), validate_point=False)
  File "/home/incorta/.local/lib/python2.7/site-packages/paramiko/ecdsakey.py", line 163, in __init__
    key = ec.EllipticCurvePublicKey.from_encoded_point(
AttributeError: type object 'EllipticCurvePublicKey' has no attribute 'from_encoded_point'

I tried the previous scenario using both a regular user on the system and also using root user and got the same errors in both trials.

Thanks a lot for your support, your response is highly appreciated. Looking forward to hearing back from you.