search

GoogleCloudDataproc / initialization-actions

Run in all nodes of your cluster before the cluster starts - lets you customize your cluster
https://cloud.google.com/dataproc/init-actions
Apache License 2.0
586 stars 513 forks source link

Hue install process failed w/ CloudSQL instance #395

Open sundhar010 opened 5 years ago

sundhar010 commented 5 years ago

I created an SQL instance for Hive metastore and I tried creating the Dataproc cluster using the following command. But, was getting an error at hue.sh.

gcloud dataproc clusters create hive-cluster \
    --scopes sql-admin \
    --image-version 1.3 \
    --initialization-actions gs://dataproc-initialization-actions/cloud-sql-proxy/cloud-sql-proxy.sh,gs://dataproc-initialization-actions/hue/hue.sh \
    --properties hive:hive.metastore.warehouse.dir=gs://$PROJECT-warehouse/datasets \
    --metadata "hive-metastore-instance=$PROJECT:$REGION:hive-metastore"

Found this in the log file, hope this would help:

traceback (most recent call last):
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/connectors/spark_shell.py", line 38, in <module>
    from spark.conf import LIVY_SERVER_SESSION_KIND
ImportError: No module named spark.conf
[10/Dec/2018 02:07:05 +0000] api          WARNING  Solr Search is not enabled
Traceback (most recent call last):
  File "/usr/lib/hue/build/env/bin/hue", line 12, in <module>
    load_entry_point('desktop==4.1.0', 'console_scripts', 'hue')()
  File "/usr/lib/hue/desktop/core/src/desktop/manage_entry.py", line 59, in entry
    execute_from_command_line(sys.argv)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/South-1.0.2-py2.7.egg/south/management/commands/migrate.py", line 111, in handle
    ignore_ghosts = ignore_ghosts,
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/South-1.0.2-py2.7.egg/south/migration/__init__.py", line 173, in migrate_app
    Migrations.calculate_dependencies()
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/South-1.0.2-py2.7.egg/south/migration/base.py", line 240, in calculate_dependencies
    migration.calculate_dependencies()
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/South-1.0.2-py2.7.egg/south/migration/base.py", line 374, in calculate_dependencies
    for migration in self._get_dependency_objects("depends_on"):
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/South-1.0.2-py2.7.egg/south/migration/base.py", line 354, in _get_dependency_objects
    for app, name in getattr(self.migration_class(), attrname, []):
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/South-1.0.2-py2.7.egg/south/migration/base.py", line 326, in migration_class
    return self.migration().Migration
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/South-1.0.2-py2.7.egg/south/utils/__init__.py", line 62, in method
    value = function(self)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/South-1.0.2-py2.7.egg/south/migration/base.py", line 315, in migration
    raise exceptions.UnknownMigration(self, sys.exc_info())
south.exceptions.UnknownMigration: Migration 'useradmin:0008_convert_documents' probably doesn't exist.
Traceback (most recent call last):
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/South-1.0.2-py2.7.egg/south/migration/base.py", line 313, in migration
    migration = __import__(full_name, {}, {}, ['Migration'])
  File "/usr/lib/hue/apps/useradmin/src/useradmin/migrations/0008_convert_documents.py", line 5, in <module>
    import desktop.management.commands.convert_documents
  File "/usr/lib/hue/desktop/core/src/desktop/management/commands/convert_documents.py", line 26, in <module>
    from desktop.converters import DocumentConverter
  File "/usr/lib/hue/desktop/core/src/desktop/converters.py", line 28, in <module>
    from notebook.api import _historify
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/api.py", line 28, in <module>
    from desktop.api2 import __paginate
  File "/usr/lib/hue/desktop/core/src/desktop/api2.py", line 41, in <module>
    from notebook.views import upgrade_session_properties
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/views.py", line 38, in <module>
    from notebook.decorators import check_editor_access_permission, check_document_access_permission, check_document_modify_permission
  File "/usr/lib/hue/desktop/libs/notebook/src/notebook/decorators.py", line 31, in <module>
    from dashboard.api import extract_solr_exception_message
  File "/usr/lib/hue/desktop/libs/dashboard/src/dashboard/api.py", line 30, in <module>
    from search.conf import SOLR_URL
ImportError: No module named search.conf
medb commented 5 years ago

May you try to install Hue without Cloud SQL Proxy to check if problem is in compatibility between these init actions or in Hue per se?

sundhar010 commented 5 years ago

@medb But I need the cluster to access hive metastore and for that, the SQL proxy is needed.

medb commented 5 years ago

I understand, this is not a solution, but it will help to pin down the problem.

sundhar010 commented 5 years ago

I tried installing only Hue and it worked. It got installed.

mikaylakonst commented 5 years ago

I was able to reproduce your issue.

Can you try swapping the order of the hue init action and the cloud sql proxy init action so that the hue init action comes first? That seemed to work for me

sundhar010 commented 5 years ago

@mikaylakonst I tried swapping the order of the init actions the cluster got deployed but got the following error if I try to access the hue web interface.

Traceback (most recent call last):
  File "/usr/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 1215, in communicate
    req.respond()
  File "/usr/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 576, in respond
    self._respond()
  File "/usr/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 588, in _respond
    response = self.wsgi_app(self.environ, self.start_response)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/handlers/wsgi.py", line 206, in __call__
    response = self.get_response(request)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/handlers/base.py", line 194, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/core/handlers/base.py", line 236, in handle_uncaught_exception
    return callback(request, **param_dict)
  File "/usr/lib/hue/desktop/core/src/desktop/views.py", line 391, in serve_500_error
    return render("500.mako", request, {'traceback': traceback.extract_tb(exc_info[2])})
  File "/usr/lib/hue/desktop/core/src/desktop/lib/django_util.py", line 230, in render
    **kwargs)
  File "/usr/lib/hue/desktop/core/src/desktop/lib/django_util.py", line 148, in _render_to_response
    return django_mako.render_to_response(template, *args, **kwargs)
  File "/usr/lib/hue/desktop/core/src/desktop/lib/django_mako.py", line 125, in render_to_response
    return HttpResponse(render_to_string(template_name, data_dictionary), **kwargs)
  File "/usr/lib/hue/desktop/core/src/desktop/lib/django_mako.py", line 114, in render_to_string_normal
    result = template.render(**data_dict)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Mako-0.8.1-py2.7.egg/mako/template.py", line 443, in render
    return runtime._render(self, self.callable_, args, data)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Mako-0.8.1-py2.7.egg/mako/runtime.py", line 786, in _render
    **_kwargs_for_callable(callable_, data))
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Mako-0.8.1-py2.7.egg/mako/runtime.py", line 818, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Mako-0.8.1-py2.7.egg/mako/runtime.py", line 844, in _exec_template
    callable_(context, *args, **kwargs)
  File "/tmp/tmpJHnMrT/desktop/500.mako.py", line 120, in render_body
    __M_writer(unicode( commonfooter(request, messages) ))
  File "/usr/lib/hue/desktop/core/src/desktop/views.py", line 515, in commonfooter
    hue_settings = Settings.get_settings()
  File "/usr/lib/hue/desktop/core/src/desktop/models.py", line 110, in get_settings
    settings, created = Settings.objects.get_or_create(id=1)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/manager.py", line 154, in get_or_create
    return self.get_queryset().get_or_create(**kwargs)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/query.py", line 376, in get_or_create
    return self.get(**lookup), False
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/query.py", line 304, in get
    num = len(clone)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/query.py", line 77, in __len__
    self._fetch_all()
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/query.py", line 857, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/query.py", line 220, in iterator
    for row in compiler.results_iter():
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/sql/compiler.py", line 713, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/models/sql/compiler.py", line 785, in execute_sql
    cursor = self.connection.cursor()
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/__init__.py", line 162, in cursor
    cursor = util.CursorWrapper(self._cursor(), self)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/__init__.py", line 132, in _cursor
    self.ensure_connection()
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/__init__.py", line 127, in ensure_connection
    self.connect()
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/__init__.py", line 127, in ensure_connection
    self.connect()
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/__init__.py", line 115, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.6.10-py2.7.egg/django/db/backends/mysql/base.py", line 435, in get_new_connection
    conn = Database.connect(**conn_params)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/MySQL_python-1.2.5-py2.7-linux-x86_64.egg/MySQLdb/connections.py", line 193, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
OperationalError: (1045, "Access denied for user 'hue'@'cloudsqlproxy~35.238.*.*' (using password: YES)")

Thanks

tomowang commented 5 years ago

It's caused by MySQL server. When use Cloud SQL Proxy, the MySQL will not be installed in the master node and hue.sh failed to init the DB user and tables.

To solve this, create database and DB user in Cloud SQL. You can refer to this code in hue.sh: https://github.com/GoogleCloudPlatform/dataproc-initialization-actions/blob/a341516f4912dc15d1baf3c0eb0b33728bcfe277/hue/hue.sh#L153-L170

daithi-walker commented 5 years ago

I've the same issue. I created the db, user and privs on the cloud sql instance but I dont see how that helps?

Also, when trying to restart the mysql service (on the dataproc server I presume?), I am getting an issue. No idea why...

sudo systemctl status mysql.service
● mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql; generated; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2019-02-06 18:56:12 UTC; 4s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 6690 ExecStart=/etc/init.d/mysql start (code=exited, status=1/FAILURE)

Feb 06 18:55:43 hive-cluster-m mysqld[6942]: 2019-02-06 18:55:43 140035900217536 [Note] Server socket created on IP: '0.0.0.0'.
Feb 06 18:55:43 hive-cluster-m mysqld[6942]: 2019-02-06 18:55:43 140035900217536 [ERROR] Can't start server: Bind on TCP/IP port. Got error: 98: Address already in use
Feb 06 18:55:43 hive-cluster-m mysqld[6942]: 2019-02-06 18:55:43 140035900217536 [ERROR] Do you already have another mysqld server running on port: 3306 ?
Feb 06 18:55:43 hive-cluster-m mysqld[6942]: 2019-02-06 18:55:43 140035900217536 [ERROR] Aborting
Feb 06 18:55:43 hive-cluster-m mysqld[6942]: 
Feb 06 18:56:12 hive-cluster-m mysql[6690]: Starting MariaDB database server: mysqld . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!
Feb 06 18:56:12 hive-cluster-m systemd[1]: mysql.service: Control process exited, code=exited status=1
Feb 06 18:56:12 hive-cluster-m systemd[1]: Failed to start LSB: Start and stop the mysql database server daemon.
Feb 06 18:56:12 hive-cluster-m systemd[1]: mysql.service: Unit entered failed state.
Feb 06 18:56:12 hive-cluster-m systemd[1]: mysql.service: Failed with result 'exit-code'.

And if I try to run the hue script with syncdb or migrate options, I get another error:

sudo /usr/lib/hue/build/env/bin/hue syncdb --noinput
Traceback (most recent call last):
  File "/usr/lib/hue/build/env/bin/hue", line 14, in <module>
    load_entry_point('desktop', 'console_scripts', 'hue')()
  File "/usr/lib/hue/desktop/core/src/desktop/manage_entry.py", line 216, in entry
    execute_from_command_line(sys.argv)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/core/management/__init__.py", line 307, in execute
    settings.INSTALLED_APPS
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/conf/__init__.py", line 56, in __getattr__
    self._setup(name)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/conf/__init__.py", line 41, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11-py2.7.egg/django/conf/__init__.py", line 110, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/usr/lib/hue/desktop/core/src/desktop/settings.py", line 271, in <module>
    conf.initialize(_desktop_conf_modules, _config_dir)
  File "/usr/lib/hue/desktop/core/src/desktop/lib/conf.py", line 598, in initialize
    conf_data = load_confs(_configs_from_dir(config_dir))
  File "/usr/lib/hue/desktop/core/src/desktop/lib/conf.py", line 530, in load_confs
    for in_conf in conf_source:
  File "/usr/lib/hue/desktop/core/src/desktop/lib/conf.py", line 511, in _configs_from_dir
    conf = ConfigObj(os.path.join(conf_dir, filename))
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/configobj-5.0.6-py2.7.egg/configobj.py", line 1229, in __init__
    self._load(infile, configspec)
  File "/usr/lib/hue/build/env/lib/python2.7/site-packages/configobj-5.0.6-py2.7.egg/configobj.py", line 1318, in _load
    raise error
configobj.DuplicateError: Duplicate keyword name at line 1906.

Any ideas?

faddiekh commented 5 years ago

SOLUTION

CREATE DATABASE hue;     (REQUIRED)

1- CREATE USER 'hue'@'localhost' IDENTIFIED BY '<password>'; 
1.1- GRANT ALL PRIVILEGES ON hue.* TO 'hue'@'localhost';
                                     OR
2-CREATE USER 'hue'@'cloudsqlproxy~35.238.*.*' IDENTIFIED BY '<password>';
2.1- GRANT ALL PRIVILEGES ON hue.* TO 'hue'@'cloudsqlproxy~35.238.*.*';

Then go to /etc/hue/conf and edit file hue.ini, in that file there are four parameters should be like following:

host=127.0.0.1
port=3306
engine=mysql
user=hue
password=<your_hue_password_in_mysql_instance>

Kill the process which is running on 0.0.0.0:8888 by following command

ps -f -u hue | grep -i runspawningserver

Give permissions to hue user to create log file in /usr/lib/hue/build/env/bin/logs: (NOT REQUIRED)

sudo chown hue:hue /usr/lib/hue/build/env/bin/logs

Then go to /usr/lib/hue/build/env/bin/ directory and run following commands

sudo ./hue syncdb --noinput
sudo ./hue migrate

If the migration is completed successfully then start hue server by following command:

sudo ./hue runcpserver &

Now if you'll go to <yourclustername-m>:8888, you'll see the Hue UI.

hugomsm commented 4 years ago

Hello, for anyone having this issue, I've managed to correct this issue and connect Hue with Cloud SQL Proxy. DON'T use the Hue init script on init action having Cloud SQL Proxy, it will fail. You need to manually edit the hue.sh script to change the following lines:

https://github.com/GoogleCloudDataproc/initialization-actions/blob/a341516f4912dc15d1baf3c0eb0b33728bcfe277/hue/hue.sh#L153-L157

What I've did:

Changed from: 

mysql -u root -p root-password -e " \
    CREATE DATABASE hue; \
    CREATE USER 'hue'@'localhost' IDENTIFIED BY '${hue_password}'; \
    GRANT ALL PRIVILEGES ON hue.* TO 'hue'@'localhost';" \
    || err "Unable to create database"

To:

mysql -u root -pYOURCLOUDSQLROOTPASSWORDHERE -e " \
    CREATE DATABASE hue; \
    CREATE USER 'hue'@'%' IDENTIFIED BY '${hue_password}'; \
    GRANT ALL PRIVILEGES ON hue.* TO 'hue'@'%';" \
    || err "Unable to create database"

Then, I manually copied the script to the Dataproc's Master Node, ran the script with sudo bash hue.sh inside the Master node of the Dataproc cluster. Then, the installation will proceed normally.

If you already have ran the command before, getting the error, recreate the cluster without the init script. It leaves some mess and if you try to correct the hue.sh file and run again, it will fail (I was getting these errors, and after a clean Dataproc installation, it worked well with the changes that I've mentioned).