redis connection refused

[hamsunwoo]

에러메세지

[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO - Traceback (most recent call last):
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -   File "/home/***/.local/lib/python3.12/site-packages/redis/connection.py", line 357, in connect
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -     sock = self.retry.call_with_retry(
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -   File "/home/***/.local/lib/python3.12/site-packages/redis/retry.py", line 62, in call_with_retry
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -     return do()
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -            ^^^^
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -   File "/home/***/.local/lib/python3.12/site-packages/redis/connection.py", line 358, in <lambda>
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -     lambda: self._connect(), lambda error: self.disconnect(error)
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -             ^^^^^^^^^^^^^^^
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -   File "/home/***/.local/lib/python3.12/site-packages/redis/connection.py", line 730, in _connect
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -     raise err
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -   File "/home/***/.local/lib/python3.12/site-packages/redis/connection.py", line 718, in _connect
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -     sock.connect(socket_address)
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO - ConnectionRefusedError: [Errno 111] Connection refused
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO - 
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO - During handling of the above exception, another exception occurred:
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO - 
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO - Traceback (most recent call last):
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -   File "/tmp/venv-call3ljc3h7r/script.py", line 53, in <module>
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -     res = get_link(*arg_dict["args"], **arg_dict["kwargs"])
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -   File "/tmp/venv-call3ljc3h7r/script.py", line 21, in get_link
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -     last_id = get_last_id_from_redis()
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -               ^^^^^^^^^^^^^^^^^^^^^^^^
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -   File "/tmp/venvl13he1zq/lib/python3.12/site-packages/crawling/utils.py", line 12, in get_last_id_from_redis
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -     last_id = r.get('last_processed_id')
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -               ^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -   File "/home/***/.local/lib/python3.12/site-packages/redis/commands/core.py", line 1822, in get
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -     return self.execute_command("GET", name, keys=[name])
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -   File "/home/***/.local/lib/python3.12/site-packages/redis/client.py", line 559, in execute_command
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -     return self._execute_command(*args, **options)
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -   File "/home/***/.local/lib/python3.12/site-packages/redis/client.py", line 565, in _execute_command
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -     conn = self.connection or pool.get_connection(command_name, **options)
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -   File "/home/***/.local/lib/python3.12/site-packages/redis/connection.py", line 1422, in get_connection
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -     connection.connect()
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -   File "/home/***/.local/lib/python3.12/site-packages/redis/connection.py", line 363, in connect
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO -     raise ConnectionError(self._error_message(e))
[2024-11-25, 03:02:21 UTC] {process_utils.py:194} INFO - redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.
[2024-11-25, 03:02:21 UTC] {taskinstance.py:3311} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 767, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 733, in _execute_callable
    return ExecutionCallableRunner(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/operator_helpers.py", line 252, in run
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 417, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/operators/python.py", line 505, in execute
    return super().execute(context=serializable_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 417, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/operators/python.py", line 238, in execute
    return_value = self.execute_callable()
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/operators/python.py", line 870, in execute_callable
    result = self._execute_python_callable_in_subprocess(python_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/operators/python.py", line 588, in _execute_python_callable_in_subprocess
    raise AirflowException(error_msg) from None
airflow.exceptions.AirflowException: Process returned non-zero exit status 1.
Error 111 connecting to localhost:6379. Connection refused.
[2024-11-25, 03:02:21 UTC] {taskinstance.py:1225} INFO - Marking task as UP_FOR_RETRY. dag_id=yes24_to_MongoDB, task_id=get_link, run_id=scheduled__2024-11-24T00:00:00+00:00, execution_date=20241124T000000, start_date=20241125T030206, end_date=20241125T030221
[2024-11-25, 03:02:21 UTC] {taskinstance.py:340} ▼ Post task execution logs
[2024-11-25, 03:02:21 UTC] {standard_task_runner.py:124} ERROR - Failed to execute job 176 for task get_link (Process returned non-zero exit status 1.
Error 111 connecting to localhost:6379. Connection refused.; 7100)
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/task/task_runner/standard_task_runner.py", line 117, in _start_by_fork
    ret = args.func(args, dag=self.dag)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/cli_config.py", line 49, in command
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/cli.py", line 116, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/task_command.py", line 483, in task_run
    task_return_code = _run_task_by_selected_method(args, _dag, ti)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/task_command.py", line 256, in _run_task_by_selected_method
    return _run_raw_task(args, ti)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/task_command.py", line 341, in _run_raw_task
    return ti._run_raw_task(
           ^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py", line 97, in wrapper
    return func(*args, session=session, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 3005, in _run_raw_task
    return _run_raw_task(
           ^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 273, in _run_raw_task
    TaskInstance._execute_task_with_callbacks(
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 3159, in _execute_task_with_callbacks
    result = self._execute_task(context, task_orig)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 3183, in _execute_task
    return _execute_task(self, context, task_orig)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 767, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 733, in _execute_callable
    return ExecutionCallableRunner(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/operator_helpers.py", line 252, in run
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 417, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/operators/python.py", line 505, in execute
    return super().execute(context=serializable_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 417, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/operators/python.py", line 238, in execute
    return_value = self.execute_callable()
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/operators/python.py", line 870, in execute_callable
    result = self._execute_python_callable_in_subprocess(python_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/operators/python.py", line 588, in _execute_python_callable_in_subprocess
    raise AirflowException(error_msg) from None
airflow.exceptions.AirflowException: Process returned non-zero exit status 1.
Error 111 connecting to localhost:6379. Connection refused.

[hamsunwoo]

원인

connect_to_redis 함수에서는 host가 로컬로 되어 있는데 docker-compose.yaml에서는 container_name을 redis 로 명시하였기 때문에 host를 'redis'로 명시해줘야 서로 연결할 수 있다.

변경 전

import redis

def connect_to_redis():
    return redis.StrictRedis(
        host='localhost',  
        port=6379,
        decode_responses=True
    )

변경 후

import redis

def connect_to_redis():
    return redis.StrictRedis(
        host='redis',  
        port=6379,
        decode_responses=True
    )