Deploying hadoop/spark cluster fails following tutorial

[dn-ra]

Hi,

Trying to follow the instructions to deploy a hadoop cluster and I run into this error:

pip install -e .

DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Obtaining file:///home/dan/elasticluster
    ERROR: Command errored out with exit status 1:
     command: /home/dan/venv/bin/python2 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/dan/elasticluster/setup.py'"'"'; __file__='"'"'/home/dan/elasticluster/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-HxI4r4
         cwd: /home/dan/elasticluster/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/dan/elasticluster/setup.py", line 39, in <module>
        from ez_setup import use_setuptools
      File "ez_setup.py", line 30, in <module>
        from urllib.request import urlopen
    ImportError: No module named request
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

This seems to be an issue with using python2 to install a python3 script, so I recreated the virtual env with python3.

But after installing in a python3 env (successfully), launching the hadoop fails with the following error:

elasticluster start hadoop -n cluster
Traceback (most recent call last):
  File "/home/dan/venv/bin/elasticluster", line 33, in <module>
    sys.exit(load_entry_point('elasticluster', 'console_scripts', 'elasticluster')())
  File "/home/dan/venv/bin/elasticluster", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/home/dan/venv/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/home/dan/venv/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 855, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/home/dan/elasticluster/elasticluster/__main__.py", line 35, in <module>
    import cli.app
  File "/home/dan/venv/lib/python3.9/site-packages/cli/app.py", line 242
    except Exception, e:
                    ^
SyntaxError: invalid syntax

Which seems like another compatibility error, this time python3 interpreter reading python2 code.

[andybotting]

Hi @dn-ra

Yeah, that is definitely a Python 2/3 issue there.

It looks like many of the fixes on our fork are now upstream. Could you have a go at switching to the code from https://github.com/elasticluster/elasticluster instead?

Their docs suggest it should be working with Python 3

[dn-ra]

Thanks Andy. Just tried installing from the main elasticluster repo and had the same issue. This is with python 3.9.5

[andybotting]

Hi @dn-ra,

I've been looking Elasticluster on and off, and I think the code is in bad shape. There's a heap of open issues upstream, so I think the project itself might be close to unmaintained.

We'll have to work out internally whether we can continue to support it or not. How valuable is this project to you?

cheers, Andy

[dn-ra]

Hi Andy,

It's not an essential component of the work. I Understand if it's too much work to continue supporting.

Thanks for checking it out.

Dan