search

apache / superset

Apache Superset is a Data Visualization and Data Exploration Platform
https://superset.apache.org/
Apache License 2.0
62.3k stars 13.68k forks source link

Unable to start superset with reporting via docker-compose #22080

Closed satish-mittal closed 1 year ago

satish-mittal commented 1 year ago

A clear and concise description of what the bug is.

How to reproduce the bug

I am trying to run superset with reporting enabled via SMTP. For this purpose, I have been following the instructions on https://superset.apache.org/docs/installation/alerts-reports and start superset via docker-compose. However, I am constantly running into various errors. Any help is greatly appreciated.

Here are the steps that I followed:

  1. I checked out 1.5.2 (git checkout tags/1.5.2)
  2. Created a new dir (superset/newdir)
  3. Created the following Custom Dockerfile to include chrome driver as per the link: 
    
FROM apache/superset:1.5.2

USER root

RUN apt-get update && \ wget -q https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb && \ apt-get install -y --no-install-recommends ./google-chrome-stable_current_amd64.deb && \ rm -f google-chrome-stable_current_amd64.deb

RUN export CHROMEDRIVER_VERSION=$(curl --silent https://chromedriver.storage.googleapis.com/LATEST_RELEASE_88) && \ wget -q https://chromedriver.storage.googleapis.com/${CHROMEDRIVER_VERSION}/chromedriver_linux64.zip && \ unzip chromedriver_linux64.zip -d /usr/bin && \ chmod 755 /usr/bin/chromedriver && \ rm -f chromedriver_linux64.zip

RUN pip install --no-cache gevent psycopg2 redis

USER superset


4. Built docker image: (`docker build . -t superset-1.5.2-extended`)
5. Created the docker-compose.yaml file inside the new directory:

version: '3.6' services: redis: image: redis:6.0.9-buster restart: on-failure volumes:

The sample docker-compose.yaml provided in the above link does not work. It throws the following errors: a) error decoding 'Command': invalid command line string. I had to modify the command section and convert it into array of strings. b) When that is fixed, it throws the error: external volume "db" not found. I had to modify db section to external: false.

  1. Created superset_config.py file in superset/newdir/config subdirectory: 
    
from celery.schedules import crontab

FEATURE_FLAGS = { "ALERT_REPORTS": True }

REDIS_HOST = "redis-superset" REDIS_PORT = "6379"

DATABASE_DB = "superset" DATABASE_HOST = "postgres" DATABASE_PORT = "5432" DATABASE_USER = "superset" DATABASE_PASSWORD = "superset" DATABASE_DIALECT = "postgresql"

POSTGRES_DB = "superset" POSTGRES_USER = "superset" POSTGRES_PASSWORD = "superset"

The SQLAlchemy connection string.

SQLALCHEMY_DATABASE_URI = "%s://%s:%s@%s:%s/%s" % ( DATABASE_DIALECT, DATABASE_USER, DATABASE_PASSWORD, DATABASE_HOST, DATABASE_PORT, DATABASE_DB, )

class CeleryConfig: broker_url = 'redis://%s:%s/0' % (REDIS_HOST, REDIS_PORT) imports = ('superset.sql_lab', "superset.tasks", "superset.tasks.thumbnails", ) result_backend = 'redis://%s:%s/0' % (REDIS_HOST, REDIS_PORT) worker_prefetch_multiplier = 10 task_acks_late = True task_annotations = { 'sql_lab.get_sql_results': { 'rate_limit': '100/s', }, 'email_reports.send': { 'rate_limit': '1/s', 'time_limit': 600, 'soft_time_limit': 600, 'ignore_result': True, }, } beat_schedule = { 'reports.scheduler': { 'task': 'reports.scheduler', 'schedule': crontab(minute='', hour=''), }, 'reports.prune_log': { 'task': 'reports.prune_log', 'schedule': crontab(minute=0, hour=0), }, } CELERY_CONFIG = CeleryConfig

SCREENSHOT_LOCATE_WAIT = 100 SCREENSHOT_LOAD_WAIT = 600

Slack configuration

SLACK_API_TOKEN = "xoxb-"

Email configuration

SMTP_HOST = "XX" #change to your host SMTP_STARTTLS = True SMTP_SSL_SERVER_AUTH = False # If your using an SMTP server with a valid certificate SMTP_SSL = False SMTP_USER = "XX" SMTP_PORT = 587 # your port eg. 587 SMTP_PASSWORD = "XX" SMTP_MAIL_FROM = "no-reply@example.com"

WebDriver configuration

If you use Firefox, you can stick with default values

If you use Chrome, then add the following WEBDRIVER_TYPE and WEBDRIVER_OPTION_ARGS

WEBDRIVER_TYPE = "chrome" WEBDRIVER_OPTION_ARGS = [ "--force-device-scale-factor=2.0", "--high-dpi-support=2.0", "--headless", "--disable-gpu", "--disable-dev-shm-usage", "--no-sandbox", "--disable-setuid-sandbox", "--disable-extensions", ]

This is for internal use, you can keep http

WEBDRIVER_BASEURL="http://superset:8088"

This is the link sent to the recipient, change to your domain eg. https://superset.mydomain.com

WEBDRIVER_BASEURL_USER_FRIENDLY="http://localhost:8088"

7. Started the image using `docker-compose up`. However it keeps on throwing the error:

newdir-worker-1 | sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file newdir-worker-1 | (Background on this error at: http://sqlalche.me/e/13/e3q8) newdir-worker-1 | [2022-11-09 14:21:22,414: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection... newdir-worker-1 | Traceback (most recent call last): newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect newdir-worker-1 | return fn() newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 364, in connect newdir-worker-1 | return _ConnectionFairy._checkout(self) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout newdir-worker-1 | fairy = _ConnectionRecord.checkout(pool) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 495, in checkout newdir-worker-1 | rec = pool._do_get() newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 241, in _do_get newdir-worker-1 | return self._create_connection() newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection newdir-worker-1 | return _ConnectionRecord(self) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 440, in init newdir-worker-1 | self.connect(first_connect_check=True) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 661, in connect newdir-worker-1 | pool.logger.debug("Error on connect(): %s", e) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in exit newdir-worker-1 | compat.raise( newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise newdir-worker-1 | raise exception newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 656, in connect newdir-worker-1 | connection = pool._invoke_creator(self) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect newdir-worker-1 | return dialect.connect(*cargs, *cparams) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 508, in connect newdir-worker-1 | return self.dbapi.connect(cargs, cparams) newdir-worker-1 | sqlite3.OperationalError: unable to open database file newdir-worker-1 | newdir-worker-1 | The above exception was the direct cause of the following exception: newdir-worker-1 | newdir-worker-1 | Traceback (most recent call last): newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 318, in start newdir-worker-1 | blueprint.start(self) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/celery/bootsteps.py", line 119, in start newdir-worker-1 | step.start(parent) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/tasks.py", line 40, in start newdir-worker-1 | c.task_consumer = c.app.amqp.TaskConsumer( newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/celery/app/amqp.py", line 301, in TaskConsumer newdir-worker-1 | return self.Consumer( newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 386, in init newdir-worker-1 | self.revive(self.channel) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 408, in revive newdir-worker-1 | self.declare() newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 421, in declare newdir-worker-1 | queue.declare() newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/kombu/entity.py", line 611, in declare newdir-worker-1 | self._create_queue(nowait=nowait, channel=channel) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/kombu/entity.py", line 620, in _create_queue newdir-worker-1 | self.queue_declare(nowait=nowait, passive=False, channel=channel) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/kombu/entity.py", line 648, in queue_declare newdir-worker-1 | ret = channel.queue_declare( newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 531, in queue_declare newdir-worker-1 | self._new_queue(queue, kwargs) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/kombu/transport/sqlalchemy/init.py", line 101, in _new_queue newdir-worker-1 | self._get_or_create(queue) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/kombu/transport/sqlalchemy/init.py", line 80, in _get_or_create newdir-worker-1 | obj = self.session.query(self.queue_cls) \ newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/kombu/transport/sqlalchemy/init.py", line 75, in session newdir-worker-1 | _, Session = self._open() newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/kombu/transport/sqlalchemy/init__.py", line 67, in _open newdir-worker-1 | metadata.create_all(engine) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/schema.py", line 4664, in create_all newdir-worker-1 | bind._run_visitor( newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2094, in _run_visitor newdir-worker-1 | with self._optional_conn_ctx_manager(connection) as conn: newdir-worker-1 | File "/usr/local/lib/python3.8/contextlib.py", line 113, in enter newdir-worker-1 | return next(self.gen) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2086, in _optional_conn_ctx_manager newdir-worker-1 | with self._contextual_connect() as conn: newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect newdir-worker-1 | self._wrap_pool_connect(self.pool.connect, None), newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2339, in _wrap_pool_connect newdir-worker-1 | Connection._handle_dbapi_exception_noconnection( newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1583, in _handle_dbapi_exceptionnoconnection newdir-worker-1 | util.raise( newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_ newdir-worker-1 | raise exception newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect newdir-worker-1 | return fn() newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 364, in connect newdir-worker-1 | return _ConnectionFairy._checkout(self) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout newdir-worker-1 | fairy = _ConnectionRecord.checkout(pool) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 495, in checkout newdir-worker-1 | rec = pool._do_get() newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 241, in _do_get newdir-worker-1 | return self._create_connection() newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection newdir-worker-1 | return _ConnectionRecord(self) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 440, in init newdir-worker-1 | self.connect(first_connect_check=True) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 661, in connect newdir-worker-1 | pool.logger.debug("Error on connect(): %s", e) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 68, in exit newdir-worker-1 | compat.raise( newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise newdir-worker-1 | raise exception newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 656, in __connect newdir-worker-1 | connection = pool._invoke_creator(self) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect newdir-worker-1 | return dialect.connect(*cargs, *cparams) newdir-worker-1 | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 508, in connect newdir-worker-1 | return self.dbapi.connect(cargs, **cparams)



I have been trying to play around with various values for `SQLALCHEMY_DATABASE_URI` ("postgresql+psycopg2://superset:superset@postgres:5432/superset", "postgresql://superset:superset@postgres:5432/superset" etc etc ) but nothing works.

### Expected results

Superset should start without any errors.

### Actual results

Internal server error shows up in UI.

#### Screenshots

If applicable, add screenshots to help explain your problem.

### Environment

(please complete the following information):

- browser type and version:
- superset version: `superset version`
- python version: `python --version`
- node.js version: `node -v`
- any feature flags active:

### Checklist

Make sure to follow these steps before submitting your issue - thank you!

- [ ] I have checked the superset logs for python stacktraces and included it here as text if there are any.
- [ ] I have reproduced the issue with at least the latest released version of superset.
- [ ] I have checked the issue tracker for the same issue and I haven't found one similar.

### Additional context

Add any other context about the problem here.
sfirke commented 1 year ago

This is less of a bug report (which is what these issues are best for) and more of a request for debugging help. You might try the Superset Slack chat which is better suited to debugging advice. The general tip I'd give here is to try docker-compose with the image 2.0.0-dev - that will have Firefox installed with a headless browser, so you wouldn't need to build your own images.

BowenXiao1999 commented 1 year ago

This is less of a bug report (which is what these issues are best for) and more of a request for debugging help. You might try the Superset Slack chat which is better suited to debugging advice. The general tip I'd give here is to try docker-compose with the image 2.0.0-dev - that will have Firefox installed with a headless browser, so you wouldn't need to build your own images.

Can you again update the link? It's expired. Thanks a lot!

sfirke commented 1 year ago

Which link? The Slack invitation link on the main README still seems live.

BowenXiao1999 commented 1 year ago

Which link? The Slack invitation link on the main README still seems live.

Not for me... This won't work: https://join.slack.com/t/apache-superset/shared_invite/zt-16jvzmoi8-sI7jKWp~xc2zYRe~NqiY9Q. I also see people have same problem in discussion and some issue comments.

https://github.com/apache/superset/discussions/22076

satish-mittal commented 1 year ago

Yes the Slack invitation link doesn't work for me as well.

sfirke commented 1 year ago

Thanks for pointing that out. Try this? https://join.slack.com/t/apache-superset/shared_invite/zt-1jn0xwmea-Ej44Zt3uVjBRowKitGMb_Q If that works we should get it changed in the README. When I made this new link I set it to expire "Never", but it flashed a message about "you can share it with up to 400 people" so maybe it can get used up.

rusackas commented 1 year ago

Superset Slack

^ this is the best link to use for Slack. It's basically an alias for an invitation link. Those all expire from time to time, but hopefully we'll do a good job of refreshing it :)

Anyway, sounds like the conversation did or should move to Slack, so I'll close this out :) Thanks all!