apache / superset

Apache Superset is a Data Visualization and Data Exploration Platform
https://superset.apache.org/
Apache License 2.0
62.81k stars 13.88k forks source link

Feature Email Reports [Errno 111] Connection refused> #10173

Closed thematheusgomes closed 4 years ago

thematheusgomes commented 4 years ago

Hello guys,

I am trying to activate the email reporting feature, but I am caught in an error that is happening in my local environment.

When I try to send a test email I get the following error:

superset-worker_1  | [2020-06-26 14:18:21,977: INFO/MainProcess] Received task: email_reports.send[424af3f6-dfc4-4854-896e-325929e597d8]  
superset-worker_1  | [2020-06-26 14:18:21,977: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f27f16ea400> (args:('email_reports.send', '424af3f6-dfc4-4854-896e-325929e597d8', {'lang': 'py', 'task': 'email_reports.send', 'id': '424af3f6-dfc4-4854-896e-325929e597d8', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [120, 150], 'root_id': '424af3f6-dfc4-4854-896e-325929e597d8', 'parent_id': None, 'argsrepr': "('slice', 1)", 'kwargsrepr': "{'recipients': 'thematheusgomes@gmail.com'}", 'origin': 'gen10@f44409800e15', 'reply_to': '63fed0c8-a976-384b-8b8e-691eabe81119', 'correlation_id': '424af3f6-dfc4-4854-896e-325929e597d8', 'hostname': 'celery@6bb9f2776617', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}, 'args': ['slice', 1], 'kwargs': {'recipients': 'thematheusgomes@gmail.com'}}, b'[["slice", 1], {"recipients": "thematheusgomes@gmail.com"}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
superset-worker_1  | [2020-06-26 14:18:21,978: DEBUG/MainProcess] Task accepted: email_reports.send[424af3f6-dfc4-4854-896e-325929e597d8] pid:21
superset_1         | 172.20.0.1 - - [26/Jun/2020 14:18:21] "POST /sliceemailscheduleview/edit/1 HTTP/1.1" 302 -
superset_1         | INFO:werkzeug:172.20.0.1 - - [26/Jun/2020 14:18:21] "POST /sliceemailscheduleview/edit/1 HTTP/1.1" 302 -
superset-worker_1  | [2020-06-26 14:18:21,994: ERROR/ForkPoolWorker-1] Task email_reports.send[424af3f6-dfc4-4854-896e-325929e597d8] raised unexpected: URLError(ConnectionRefusedError(111, 'Connection refused'),)
superset-worker_1  | Traceback (most recent call last):
superset-worker_1  |   File "/usr/local/lib/python3.6/urllib/request.py", line 1318, in do_open
superset-worker_1  |     encode_chunked=req.has_header('Transfer-encoding'))
superset-worker_1  |   File "/usr/local/lib/python3.6/http/client.py", line 1254, in request
superset-worker_1  |     self._send_request(method, url, body, headers, encode_chunked)
superset-worker_1  |   File "/usr/local/lib/python3.6/http/client.py", line 1300, in _send_request
superset-worker_1  |     self.endheaders(body, encode_chunked=encode_chunked)
superset-worker_1  |   File "/usr/local/lib/python3.6/http/client.py", line 1249, in endheaders
superset-worker_1  |     self._send_output(message_body, encode_chunked=encode_chunked)
superset-worker_1  |   File "/usr/local/lib/python3.6/http/client.py", line 1036, in _send_output
superset-worker_1  |     self.send(msg)
superset-worker_1  |   File "/usr/local/lib/python3.6/http/client.py", line 974, in send
superset-worker_1  |     self.connect()
superset-worker_1  |   File "/usr/local/lib/python3.6/http/client.py", line 946, in connect
superset-worker_1  |     (self.host,self.port), self.timeout, self.source_address)
superset-worker_1  |   File "/usr/local/lib/python3.6/socket.py", line 724, in create_connection
superset-worker_1  |     raise err
superset-worker_1  |   File "/usr/local/lib/python3.6/socket.py", line 713, in create_connection
superset-worker_1  |     sock.connect(sa)
superset-worker_1  | ConnectionRefusedError: [Errno 111] Connection refused
superset-worker_1  | 
superset-worker_1  | During handling of the above exception, another exception occurred:
superset-worker_1  | 
superset-worker_1  | Traceback (most recent call last):
superset-worker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task
superset-worker_1  |     R = retval = fun(*args, **kwargs)
superset-worker_1  |   File "/app/superset/app.py", line 114, in __call__
superset-worker_1  |     return task_base.__call__(self, *args, **kwargs)
superset-worker_1  |   File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 650, in __protected_call__
superset-worker_1  |     return self.run(*args, **kwargs)
superset-worker_1  |   File "/app/superset/tasks/schedules.py", line 374, in schedule_email_report
superset-worker_1  |     deliver_slice(schedule)
superset-worker_1  |   File "/app/superset/tasks/schedules.py", line 335, in deliver_slice
superset-worker_1  |     email = _get_slice_data(schedule)
superset-worker_1  |   File "/app/superset/tasks/schedules.py", line 262, in _get_slice_data
superset-worker_1  |     response = opener.open(slice_url)
superset-worker_1  |   File "/usr/local/lib/python3.6/urllib/request.py", line 526, in open
superset-worker_1  |     response = self._open(req, data)
superset-worker_1  |   File "/usr/local/lib/python3.6/urllib/request.py", line 544, in _open
superset-worker_1  |     '_open', req)
superset-worker_1  |   File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
superset-worker_1  |     result = func(*args)
superset-worker_1  |   File "/usr/local/lib/python3.6/urllib/request.py", line 1346, in http_open
superset-worker_1  |     return self.do_open(http.client.HTTPConnection, req)
superset-worker_1  |   File "/usr/local/lib/python3.6/urllib/request.py", line 1320, in do_open
superset-worker_1  |     raise URLError(err)
superset-worker_1  | urllib.error.URLError: <urlopen error [Errno 111] Connection refused>

I thought it was a problem with the data on the smtp server, but I did some local tests and the credentials are correct, so I imagine it is some blocking on my local network, some communication failure with the bank, I don't know I need your help .

Here are the settings I'm using:

superset_config.py

MYSQL_USER = get_env_variable("MYSQL_USER")
MYSQL_PASSWORD = get_env_variable("MYSQL_PASSWORD")
MYSQL_HOST = get_env_variable("MYSQL_HOST")
MYSQL_PORT = get_env_variable("MYSQL_PORT")
MYSQL_DATABASE = get_env_variable("MYSQL_DATABASE")

CACHE_CONFIG = {
    'CACHE_TYPE': 'redis',
    'CACHE_DEFAULT_TIMEOUT': 300,
    'CACHE_KEY_PREFIX': 'superset_',
    'CACHE_REDIS_HOST': 'redis',
    'CACHE_REDIS_PORT': 6379,
    'CACHE_REDIS_DB': 1,
    'CACHE_REDIS_URL': 'redis://redis:6379/1'}
SQLALCHEMY_DATABASE_URI = 'mysql://%s:%s@%s:%s/%s' % (
    MYSQL_USER,
    MYSQL_PASSWORD,
    MYSQL_HOST,
    MYSQL_PORT,
    MYSQL_DATABASE
)
SQLALCHEMY_TRACK_MODIFICATIONS = True
SECRET_KEY = 'xxxxxxxxxxxxxxxxx'

REDIS_HOST = get_env_variable("REDIS_HOST")
REDIS_PORT = get_env_variable("REDIS_PORT")

RESULTS_BACKEND = FileSystemCache('/app/superset_home/sqllab')

# Extras
ENABLE_PROXY_FIX = True

class CeleryConfig(object):
    BROKER_URL = 'redis://redis:6379/0'
    CELERY_IMPORTS = (
        'superset.sql_lab',
        'superset.tasks',
    )
    CELERY_RESULT_BACKEND = 'redis://redis:6379/0'
    CELERYD_LOG_LEVEL = 'DEBUG'
    CELERYD_PREFETCH_MULTIPLIER = 10
    CELERY_ACKS_LATE = True
    CELERY_ANNOTATIONS = {
        'sql_lab.get_sql_results': {
            'rate_limit': '100/s',
        },
        'email_reports.send': {
            'rate_limit': '1/s',
            'time_limit': 120,
            'soft_time_limit': 150,
            'ignore_result': True,
        },
    }
    CELERYBEAT_SCHEDULE = {
        'email_reports.schedule_hourly': {
            'task': 'email_reports.schedule_hourly',
            'schedule': crontab(minute=1, hour='*'),
        },
    }

CELERY_CONFIG = CeleryConfig

# Email Reports

EMAIL_NOTIFICATIONS = True
SMTP_HOST = "email-smtp.us-east-1.amazonaws.com"
SMTP_STARTTLS = True
SMTP_SSL = True
SMTP_USER = "xxxxxxxxxxxxxxxxxxxxxxxx"
SMTP_PORT = 25
SMTP_PASSWORD = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
SMTP_MAIL_FROM = "xxxxxxxxxx@xxxxxxxxx.com"

docker-compose.yml

x-superset-build: &superset-build
  args:
    NPM_BUILD_CMD: build-dev
  context: ./
  dockerfile: Dockerfile
  target: dev
x-superset-depends-on: &superset-depends-on
  - mysql
  - redis
x-superset-volumes: &superset-volumes
  # /app/pythonpath_docker will be appended to the PYTHONPATH in the final container
  - ./docker/docker-init.sh:/app/docker-init.sh
  - ./docker/pythonpath_dev:/app/pythonpath
  - ./superset:/app/superset
  - ./superset-frontend:/app/superset-frontend
  - superset_home:/app/superset_home

version: "3.7"
services:
  redis:
    image: redis:3.2
    restart: unless-stopped
    ports:
      - "127.0.0.1:6379:6379"
    volumes:
      - redis:/data

  mysql:
    command: --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
    env_file: docker/.env
    image: mysql:5
    restart: unless-stopped
    ports:
      - "127.0.0.1:3306:3306"
    volumes:
      - mysql:/var/lib/mysql

  superset:
    build: *superset-build
    command: ["flask", "run", "-p", "8088", "--with-threads", "--reload", "--debugger", "--host=0.0.0.0"]
    env_file: docker/.env
    restart: unless-stopped
    ports:
      - 8088:8088
    depends_on: *superset-depends-on
    volumes: *superset-volumes

  superset-init:
    build: *superset-build
    command: ["/app/docker-init.sh"]
    env_file: docker/.env
    depends_on: *superset-depends-on
    volumes: *superset-volumes

  superset-worker:
    user: root
    build: *superset-build
    command: ["celery", "worker", "--app=superset.tasks.celery_app:app", "--pool=prefork", "-Ofair", "-c", "4", "--loglevel=DEBUG", "--max-tasks-per-child=128"]
    env_file: docker/.env
    restart: unless-stopped
    ports:
      - 25:25
    depends_on: *superset-depends-on
    volumes: *superset-volumes

  superset-beat:
    build: *superset-build
    user: root # Just a Test
    command: ["celery", "beat", "--app=superset.tasks.celery_app:app", "--pidfile=", "--loglevel=DEBUG", "-f", "/app/celery_beat.log"]
    env_file: docker/.env
    restart: unless-stopped
    depends_on: *superset-depends-on
    volumes: *superset-volumes

volumes:
  superset_home:
    external: false
  mysql:
    external: false
  redis:
    external: false

Please, if anyone can help me I would be grateful.

issue-label-bot[bot] commented 4 years ago

Issue-Label Bot is automatically applying the label #bug to this issue, with a confidence of 0.77. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

EwertonDCSilv commented 4 years ago

Obtive o mesmo erro, estou na a versão 0.36.0 !

deangoku commented 4 years ago

same problem here, any solution?

EwertonDCSilv commented 4 years ago

Consegui resolver aqui, mudamos muitas coisas, inclusiva reinstalando o apache-superset. Um grande problema do projeto é apresentar erro extremamente genéricos, que faz o usuário dar um muitas de voltas para realizar a configuração correta...

1 - remover e instalar a versão 0.36.0 2 - executar o comando superset db upgrad (caso mantenha a base de dados) 3 - executar o comando superset init 4 - Instalar o redis ( pip install redis) 5 - Definir no arquivo de configuração do superset as configurações do Celery (pode copiar da documentação) 6 - Instalar e definir o webdrive, por padrão a configuração do superset trás o geckodriver. 7 - Definir nas configurações do superset o local dos arquivos de logger do webdrive. 8 - Iniciar o Celery worker e o Celery beat com arquivos de logger devidamente configurados. Os loggers vão te ajudar. 9 - você pode utilizar o Celery flower para monitorar as tarefas, mas não é necessário para o funcionamento.

EwertonDCSilv commented 4 years ago

Consegui resolver aqui, mudamos muitas coisas, inclusiva reinstalando o apache-superset. Um grande problema do projeto é apresentar erro extremamente genéricos, que faz o usuário dar um muitas de voltas para realizar a configuração correta...

1 - remover e instalar a versão 0.36.0 2 - executar o comando superset upgradedb (caso mantenha a base de dados) 3 - executar o comando superset init 4 - Instalar o redis ( pip install redis) 5 - Definir no arquivo de configuração do superset as configurações do Celery (pode copiar da documentação) 6 - Instalar e definir o webdrive, por padrão a configuração do superset trás o geckodriver. 7 - Definir nas configurações do superset o local dos arquivos de logger do webdrive. 8 - Iniciar o Celery worker e o Celery beat com arquivos de logger devidamente configurados. Os loggers vão te ajudar. 9 - você pode utilizar o Celery flower para monitorar as tarefas, mas não é necessário para o funcionamento.

Fiz essas coisas e no fim funcionou, não sei ao certo a causa da falhar, mas no fim funcionou. Imagino que a falha pode ser causada por erros na configuração do webdrive ou falta de permissão para os arquivos de logger.

thematheusgomes commented 4 years ago

Concordo, você poderia me passar suas configurações ? o superset_config.py, o Dockerfile e o docker-compose Eu tentei seguir os passos que você passou, mas o mesmo erro persiste

EwertonDCSilv commented 4 years ago

Realizei a instalação via pip, mas acredito que não seja tão complicado criar as configs para docker

EwertonDCSilv commented 4 years ago

O superset_config.py:

from cachelib.redis import RedisCache
from celery.schedules import crontab

SUPERSET_WEBSERVER_TIMEOUT = 120
SQLLAB_TIMEOUT = 60

ENABLE_TIME_ROTATE = True
ENABLE_SCHEDULED_EMAIL_REPORTS = True
EMAIL_NOTIFICATIONS = True
EMAIL_REPORTS_USER = "user-admin"

SMTP_HOST = "smtp.gmail.com"
SMTP_STARTTLS = True
SMTP_SSL = True
SMTP_USER = "email@gmail.com"
SMTP_PORT = 587
SMTP_PASSWORD = "token_pass"
SMTP_MAIL_FROM = "email@gmail.com"

WEBDRIVER_BASEURL = "http://HOST:8088/"
WEBDRIVER_CONFIGURATION = {
    "service_log_path": '/home/user/log/geckodriver.log',
    "timeout": 60
}

BABEL_DEFAULT_LOCALE = "pt_BR"

class CeleryConfig(object):
    BROKER_URL = 'redis://localhost:6379/0'
    CELERY_IMPORTS = (
        'superset.sql_lab',
        'superset.tasks',
    )
    CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
    CELERYD_LOG_LEVEL = 'DEBUG'
    CELERYD_PREFETCH_MULTIPLIER = 10
    CELERY_ACKS_LATE = True
    CELERY_ANNOTATIONS = {
        'sql_lab.get_sql_results': {
            'rate_limit': '100/s',
        },
        'email_reports.send': {
            'rate_limit': '1/s',
            'time_limit': 120,
            'soft_time_limit': 150,
            'ignore_result': True,
        },
    }
    CELERYBEAT_SCHEDULE = {
        'email_reports.schedule_hourly': {
            'task': 'email_reports.schedule_hourly',
            'schedule': crontab(minute=1, hour='*'),
        }
    }

CELERY_CONFIG = CeleryConfig

RESULTS_BACKEND = RedisCache(
    host='localhost', port=6379, key_prefix='superset_results')

CACHE_CONFIG = {
    'CACHE_TYPE': 'redis',
    'CACHE_DEFAULT_TIMEOUT': 60 * 60 * 24, # 1 day default (in secs)
    'CACHE_KEY_PREFIX': 'superset_results',
    'CACHE_REDIS_URL': 'redis://localhost:6379/0',
}
stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. For admin, please label this issue .pinned to prevent stale bot from closing the issue.

sfirke commented 11 months ago

Someone in Slack who was facing this same error in December 2023 reports:

the problem was in my original config i had result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/1" whereas it should've been: result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"

Talin24 commented 2 months ago

Facing the same issue. Please help me to resolve it.

EwertonDCSilv commented 1 month ago

Facing the same issue. Please help me to resolve it.

Você configurou o celery para o superset ?