search

tubearchivist / tubearchivist

Your self hosted YouTube media server
https://www.tubearchivist.com
GNU General Public License v3.0
5.14k stars 227 forks source link

[Installation]: Timeout Connecting to Redis Server #480

Closed Th3M1CR0WAV3 closed 1 year ago

Th3M1CR0WAV3 commented 1 year ago

Installation instructions

Operating System

Synology

Your installation or setup question

I have followed the wiki install instructions for Synology.

Tube Archivist ES and Redis both appear to be installed correctly and run, when I run the Tube Archivist container it is unable to connect to Redis server and the container restarts.

Any help/advice on getting the container to run would be really appreciated.

Relevant log output

Traceback (most recent call last):

  File "/app/manage.py", line 23, in <module>

    main()

  File "/app/manage.py", line 19, in main

    execute_from_command_line(sys.argv)

  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line

    utility.execute()

  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 382, in execute

    settings.INSTALLED_APPS

  File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 102, in __getattr__

    self._setup(name)

  File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 89, in _setup

    self._wrapped = Settings(settings_module)

  File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 217, in __init__

    mod = importlib.import_module(self.SETTINGS_MODULE)

  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 883, in exec_module

  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed

  File "/app/config/settings.py", line 20, in <module>

    from home.src.ta.config import AppConfig

  File "/app/home/__init__.py", line 3, in <module>

    from .tasks import app as celery_app

  File "/app/home/tasks.py", line 12, in <module>

    from home.src.download.queue import PendingList

  File "/app/home/src/download/queue.py", line 10, in <module>

    from home.src.download.subscriptions import (

  File "/app/home/src/download/subscriptions.py", line 8, in <module>

    from home.src.download.thumbnails import ThumbManager

  File "/app/home/src/download/thumbnails.py", line 21, in <module>

    class ThumbManagerBase:

  File "/app/home/src/download/thumbnails.py", line 24, in ThumbManagerBase

    CONFIG = AppConfig().config

  File "/app/home/src/ta/config.py", line 23, in __init__

    self.config = self.get_config()

  File "/app/home/src/ta/config.py", line 28, in get_config

    config = self.get_config_redis()

  File "/app/home/src/ta/config.py", line 70, in get_config_redis

    config = RedisArchivist().get_message("config")

  File "/app/home/src/ta/ta_redis.py", line 59, in get_message

    reply = self.conn.execute_command("JSON.GET", self.NAME_SPACE + key)

  File "/root/.local/lib/python3.10/site-packages/redis/client.py", line 1255, in execute_command

    conn = self.connection or pool.get_connection(command_name, **options)

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 1442, in get_connection

    connection.connect()

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 704, in connect

    raise ConnectionError(self._error_message(e))

redis.exceptions.ConnectionError: Error -2 connecting to http://192.168.50.250:6379. Name or service not known.

Traceback (most recent call last):

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 698, in connect

    sock = self.retry.call_with_retry(

  File "/root/.local/lib/python3.10/site-packages/redis/retry.py", line 51, in call_with_retry

    raise error

  File "/root/.local/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry

    return do()

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 699, in <lambda>

    lambda: self._connect(), lambda error: self.disconnect(error)

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 987, in _connect

    raise err

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 975, in _connect

    sock.connect(socket_address)

TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/app/manage.py", line 23, in <module>

    main()

  File "/app/manage.py", line 19, in main

    execute_from_command_line(sys.argv)

  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line

    utility.execute()

  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 382, in execute

    settings.INSTALLED_APPS

  File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 102, in __getattr__

    self._setup(name)

  File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 89, in _setup

    self._wrapped = Settings(settings_module)

  File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 217, in __init__

    mod = importlib.import_module(self.SETTINGS_MODULE)

  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 883, in exec_module

  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed

  File "/app/config/settings.py", line 20, in <module>

    from home.src.ta.config import AppConfig

  File "/app/home/__init__.py", line 3, in <module>

    from .tasks import app as celery_app

  File "/app/home/tasks.py", line 12, in <module>

    from home.src.download.queue import PendingList

  File "/app/home/src/download/queue.py", line 10, in <module>

    from home.src.download.subscriptions import (

  File "/app/home/src/download/subscriptions.py", line 8, in <module>

    from home.src.download.thumbnails import ThumbManager

  File "/app/home/src/download/thumbnails.py", line 21, in <module>

    class ThumbManagerBase:

  File "/app/home/src/download/thumbnails.py", line 24, in ThumbManagerBase

    CONFIG = AppConfig().config

  File "/app/home/src/ta/config.py", line 23, in __init__

    self.config = self.get_config()

  File "/app/home/src/ta/config.py", line 28, in get_config

    config = self.get_config_redis()

  File "/app/home/src/ta/config.py", line 70, in get_config_redis

    config = RedisArchivist().get_message("config")

  File "/app/home/src/ta/ta_redis.py", line 59, in get_message

    reply = self.conn.execute_command("JSON.GET", self.NAME_SPACE + key)

  File "/root/.local/lib/python3.10/site-packages/redis/client.py", line 1255, in execute_command

    conn = self.connection or pool.get_connection(command_name, **options)

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 1442, in get_connection

    connection.connect()

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 702, in connect

    raise TimeoutError("Timeout connecting to server")

redis.exceptions.TimeoutError: Timeout connecting to server
Th3M1CR0WAV3 commented 1 year ago

Here's the Redis log:

9:C 26 May 2023 15:21:30.849 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo

9:C 26 May 2023 15:21:30.849 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=9, just started

9:C 26 May 2023 15:21:30.849 # Configuration loaded

9:M 26 May 2023 15:21:30.850 * monotonic clock: POSIX clock_gettime

            _._                                                  

       _.-``__ ''-._                                             

  _.-``    `.  `_.  ''-._           Redis 6.2.12 (00000000/0) 64 bit

.-`` .-.\/ ., ''-._

( ' , .-|, ) Running in standalone mode

|-._-...-__...-.``-._|' _.-'| Port: 6379

| -._. / .-' | PID: 9

-._-. `-./ .-' _.-'

|-._-._ `-._.-' .-'_.-'|

| -._-. .-'_.-' | https://redis.io

-._-._`-._.-'.-' _.-'

|-._-._ `-._.-' .-'_.-'|

| -._-. .-'_.-' |

-._-._`-._.-'.-' _.-' 

  `-._    `-.__.-'    _.-'                                       

      `-._        _.-'                                           

          `-.__.-'

9:M 26 May 2023 15:21:30.850 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.

9:M 26 May 2023 15:21:30.850 # Server initialized

9:M 26 May 2023 15:21:30.850 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

9:M 26 May 2023 15:21:30.962 * Redis version found by RedisSearch : 6.2.12 - oss

9:M 26 May 2023 15:21:30.962 * RediSearch version 2.6.9 (Git=HEAD-b205a1d5)

9:M 26 May 2023 15:21:30.962 * Low level api version 1 initialized successfully

9:M 26 May 2023 15:21:30.962 * concurrent writes: OFF, gc: ON, prefix min length: 2, prefix max expansions: 200, query timeout (ms): 500, timeout policy: return, cursor read size: 1000, cursor max idle (ms): 300000, max doctable size: 1000000, max number of search results: 10000, search pool size: 20, index pool size: 8,

9:M 26 May 2023 15:21:30.962 * Initialized thread pool!

9:M 26 May 2023 15:21:30.962 * Enabled diskless replication

9:M 26 May 2023 15:21:30.962 * Enabled role change notification

9:M 26 May 2023 15:21:30.962 * Module 'search' loaded from /opt/redis-stack/lib/redisearch.so

9:M 26 May 2023 15:21:31.073 * Enabled role change notification

9:M 26 May 2023 15:21:31.073 * Starting up RedisGraph version 2.10.10.

9:M 26 May 2023 15:21:31.074 * Thread pool created, using 8 threads.

9:M 26 May 2023 15:21:31.074 * Maximum number of OpenMP threads set to 8

9:M 26 May 2023 15:21:31.074 * Module 'graph' loaded from /opt/redis-stack/lib/redisgraph.so

9:M 26 May 2023 15:21:31.076 * RedisTimeSeries version 10810, git_sha=78d8254a4ed499e4e24566ab76c9275130c8786b

9:M 26 May 2023 15:21:31.076 * Redis version found by RedisTimeSeries : 6.2.12 - oss

9:M 26 May 2023 15:21:31.076 * loaded default CHUNK_SIZE_BYTES policy: 4096

9:M 26 May 2023 15:21:31.076 * loaded server DUPLICATE_POLICY: block

9:M 26 May 2023 15:21:31.076 * Setting default series ENCODING to: compressed

9:M 26 May 2023 15:21:31.076 * Detected redis oss

9:M 26 May 2023 15:21:31.076 * Enabled diskless replication

9:M 26 May 2023 15:21:31.076 * Module 'timeseries' loaded from /opt/redis-stack/lib/redistimeseries.so

9:M 26 May 2023 15:21:31.092 * version: 20407 git sha: a4412b3 branch: HEAD

9:M 26 May 2023 15:21:31.092 * Exported RedisJSON_V1 API

9:M 26 May 2023 15:21:31.092 * Exported RedisJSON_V2 API

9:M 26 May 2023 15:21:31.092 * Exported RedisJSON_V3 API

9:M 26 May 2023 15:21:31.092 * Enabled diskless replication

9:M 26 May 2023 15:21:31.092 * Created new data type 'ReJSON-RL'

9:M 26 May 2023 15:21:31.092 * Module 'ReJSON' loaded from /opt/redis-stack/lib/rejson.so

9:M 26 May 2023 15:21:31.092 * Acquired RedisJSON_V3 API

9:M 26 May 2023 15:21:31.092 * Acquired RedisJSON_V1 API

9:M 26 May 2023 15:21:31.093 * RedisBloom version 2.4.5 (Git=unknown)

9:M 26 May 2023 15:21:31.094 * Module 'bf' loaded from /opt/redis-stack/lib/redisbloom.so

9:M 26 May 2023 15:21:31.094 * Ready to accept connections

lamusmaser commented 1 year ago

This typically points to a configuration issue on the TA container or a networking issue that doesn't allow the containers to communicate to each other.

Can you provide the following details: What is the host IP address? What is the configured REDIS_HOST variable for the TA container? What is the the container name for the redis container? Did you supply an alias during the linking configuration on the TA container? Do you have a port forward enabled under the Port Settings for the Redis container?

That should get us a good starting point to work from.

Th3M1CR0WAV3 commented 1 year ago

Hi @lamusmaser thanks for following up on this. Details below, please let me know if you need anything else:

What is the host IP address? synology.local What is the configured REDIS_HOST variable for the TA container? tubearchivist-redis What is the the container name for the redis container? tubearchivist-redis Did you supply an alias during the linking configuration on the TA container? tubearchivist-es and tubearchivist-redis for each respective containers Do you have a port forward enabled under the Port Settings for the Redis container? 6379:6379

lamusmaser commented 1 year ago

If that is how it is configured, then it should be working. There is a line that is present in the logs that is leading me to ask those specific questions:

redis.exceptions.ConnectionError: Error -2 connecting to http://192.168.50.250:6379. Name or service not known.

This is during the attempt to connect, where I would normally expect it to easily connect to the Redis container and make that attempt over http://tubearchivist-redis:6379.

Can you upload or link to a full set of the logs so that we can review it in context? There might be a preceding action that we are missing.

Th3M1CR0WAV3 commented 1 year ago

Hi so that was me testing to see if using the local IP address would work instead of referencing linked container, the error was because of the inclusion of http:// in the environment variable. Under both 192.168.50.250 and tubearchivist-redis variables I'm getting the same redis.exceptions.TimeoutError: Timeout connecting to server error.

lamusmaser commented 1 year ago

OK, understood. With the REDIS_HOST set to tubearchivist-redis, please restart the TA container, let it generate the associated logs, then please upload or provide a reference link to those logs so we can review what is happening. After I review the logs, I might have some additional troubleshooting we can do.

Th3M1CR0WAV3 commented 1 year ago 
Traceback (most recent call last):

  File "/app/manage.py", line 23, in <module>

    main()

  File "/app/manage.py", line 19, in main

    execute_from_command_line(sys.argv)

  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line

    utility.execute()

  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 382, in execute

    settings.INSTALLED_APPS

  File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 102, in __getattr__

    self._setup(name)

  File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 89, in _setup

    self._wrapped = Settings(settings_module)

  File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 217, in __init__

    mod = importlib.import_module(self.SETTINGS_MODULE)

  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 883, in exec_module

  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed

  File "/app/config/settings.py", line 20, in <module>

    from home.src.ta.config import AppConfig

  File "/app/home/__init__.py", line 3, in <module>

    from .tasks import app as celery_app

  File "/app/home/tasks.py", line 12, in <module>

    from home.src.download.queue import PendingList

  File "/app/home/src/download/queue.py", line 10, in <module>

    from home.src.download.subscriptions import (

  File "/app/home/src/download/subscriptions.py", line 8, in <module>

    from home.src.download.thumbnails import ThumbManager

  File "/app/home/src/download/thumbnails.py", line 21, in <module>

    class ThumbManagerBase:

  File "/app/home/src/download/thumbnails.py", line 24, in ThumbManagerBase

    CONFIG = AppConfig().config

  File "/app/home/src/ta/config.py", line 23, in __init__

    self.config = self.get_config()

  File "/app/home/src/ta/config.py", line 28, in get_config

    config = self.get_config_redis()

  File "/app/home/src/ta/config.py", line 70, in get_config_redis

    config = RedisArchivist().get_message("config")

  File "/app/home/src/ta/ta_redis.py", line 59, in get_message

    reply = self.conn.execute_command("JSON.GET", self.NAME_SPACE + key)

  File "/root/.local/lib/python3.10/site-packages/redis/client.py", line 1255, in execute_command

    conn = self.connection or pool.get_connection(command_name, **options)

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 1442, in get_connection

    connection.connect()

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 702, in connect

    raise TimeoutError("Timeout connecting to server")

redis.exceptions.TimeoutError: Timeout connecting to server

Traceback (most recent call last):

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 698, in connect

    sock = self.retry.call_with_retry(

  File "/root/.local/lib/python3.10/site-packages/redis/retry.py", line 51, in call_with_retry

    raise error

  File "/root/.local/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry

    return do()

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 699, in <lambda>

    lambda: self._connect(), lambda error: self.disconnect(error)

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 987, in _connect

    raise err

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 975, in _connect

    sock.connect(socket_address)

TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/app/manage.py", line 23, in <module>

    main()

  File "/app/manage.py", line 19, in main

    execute_from_command_line(sys.argv)

  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line

    utility.execute()

  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 382, in execute

    settings.INSTALLED_APPS

  File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 102, in __getattr__

    self._setup(name)

  File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 89, in _setup

    self._wrapped = Settings(settings_module)

  File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 217, in __init__

    mod = importlib.import_module(self.SETTINGS_MODULE)

  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "<frozen importlib._bootstrap_external>", line 883, in exec_module

  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed

  File "/app/config/settings.py", line 20, in <module>

    from home.src.ta.config import AppConfig

  File "/app/home/__init__.py", line 3, in <module>

    from .tasks import app as celery_app

  File "/app/home/tasks.py", line 12, in <module>

    from home.src.download.queue import PendingList

  File "/app/home/src/download/queue.py", line 10, in <module>

    from home.src.download.subscriptions import (

  File "/app/home/src/download/subscriptions.py", line 8, in <module>

    from home.src.download.thumbnails import ThumbManager

  File "/app/home/src/download/thumbnails.py", line 21, in <module>

    class ThumbManagerBase:

  File "/app/home/src/download/thumbnails.py", line 24, in ThumbManagerBase

    CONFIG = AppConfig().config

  File "/app/home/src/ta/config.py", line 23, in __init__

    self.config = self.get_config()

  File "/app/home/src/ta/config.py", line 28, in get_config

    config = self.get_config_redis()

  File "/app/home/src/ta/config.py", line 70, in get_config_redis

    config = RedisArchivist().get_message("config")

  File "/app/home/src/ta/ta_redis.py", line 59, in get_message

    reply = self.conn.execute_command("JSON.GET", self.NAME_SPACE + key)

  File "/root/.local/lib/python3.10/site-packages/redis/client.py", line 1255, in execute_command

    conn = self.connection or pool.get_connection(command_name, **options)

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 1442, in get_connection

    connection.connect()

  File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 702, in connect

    raise TimeoutError("Timeout connecting to server")

redis.exceptions.TimeoutError: Timeout connecting to server
lamusmaser commented 1 year ago

I need the whole log set, not just the associated error. We need to be able to review the surrounding actions, as well. This is looking like it is actually establishing an initial connection, but then having an issue connecting to something else while it is setting this up.

Can you also confirm your TA container's network setting? I would expect it to be bridge and want to confirm that configuration.

Th3M1CR0WAV3 commented 1 year ago

TA container's network is bridge. I'm not sure how to obtain further detail for logs sorry, please advise.

lamusmaser commented 1 year ago

Where are you getting this error from? You should be able to pull the whole logs from that same method, typically.

Th3M1CR0WAV3 commented 1 year ago

Portainer and Docker directly but they're only providing the above level of detail.

lamusmaser commented 1 year ago

Ah, that's important information. Portainer is not really capable of running Tubearchivist natively, changes have to be made to allow the connections to work. This is because Portainer is a rootless container operator, so certain implicit connections won't exist. These aren't the general instructions provided within the wiki.

Th3M1CR0WAV3 commented 1 year ago

Something to note, both tubearchivist-es and tubearchivist-redis containers are running however when I run the tubearchivist container it states:

This container depends on tubearchivist-es, tubearchivist-redis. We've started them for you.

Th3M1CR0WAV3 commented 1 year ago

So I set up the container just via Docker, not using Portainer, and I'm running the container via Docker. Not sure if that changes anything.

lamusmaser commented 1 year ago

That would change it back to normal instruction. If that is the message you are seeing, then we should be getting it to work.

So, let's start with the following:

  1. Go to the container for TA and click the Details button.
  2. Go to the Terminal tab.
  3. Click the Create button. This will generate a bash session. Click on the bash session.
  4. Run the following command:
    1. (printf "PING\r\nQUIT\r\n";) | curl -v telnet://tubearchivist-redis:6379
  5. This should give you an output like the following: 
    root@bb6c54d3fcb4:/app# (printf "PING\r\nQUIT\r\n";) | curl -v telnet://tubearchivist-redis:6379                                                                                                       
*   Trying 172.18.0.5:6379...                                                                                                                                                                               
* Connected to tubearchivist-redis (172.18.0.5) port 6379 (#0)                                                                                                                                         
+PONG                                                                                                                                                                                                       
+OK                                                                                                                                                                                                         
* Closing connection 0

If this is not the output that you get, then there is an issue with them connecting to each other. We'll have to look at the configurations closer and determine where it is going wrong. If this is similar to what you get, then we need to determine what else is happening. To do that, we'll need the full log stack.

To do so, please follow the below:

  1. Go to the container for TA and click the Details button.
  2. Go to the Log tab.
  3. Go to today's date. Then click on the Export button's dropdown menu. Select CSV from the options and let it download.
  4. Upload the log either to here directly or another text upload area and provide the link.

We'll look into it further after those are provided.

Th3M1CR0WAV3 commented 1 year ago

Thanks for all your assistance with this mate, really appreciated. The container crashes and restarts after a couple of minutes:

 root@tubearchivist:/app# (printf "PING\r\nQUIT\r\n";) | curl -v telnet://tubearchivist-redis:6379                                                                                   
*   Trying 172.17.0.5:6379...

Followed by:

Docker container tubearchivist stopped unexpectedly. Go to Docker for more information.
Th3M1CR0WAV3 commented 1 year ago

Docker inspect results for containers (pastebin):

tubearchivist

tubearchivist-es

tubearchivist-redis

lamusmaser commented 1 year ago

The inspects gave me the details we needed.

As part of your TA container, we have this section:

"NetworkSettings": {
        "Bridge": "",
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "HairpinMode": false,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "MacAddress": "",
        "Networks": {
            "bridge": {
                "Aliases": null,
                "DriverOpts": null,
                "EndpointID": "",
                "Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAddress": "",
                "IPAMConfig": null,
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "Links": null,
                "MacAddress": "",
                "NetworkID": "6865a427ab9af95af7e3e2295cfd73e798c512a18a8af71d74890bd49487887b"
            }
        },
        "Ports": {
        },
        "SandboxID": "9414507afc1421e99146abdac6b73e5460366fab6b3fa97f0db5456628c69e0d",
        "SandboxKey": "/var/run/docker/netns/9414507afc14",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null
    },

and on a healthy container, we have this section:

"NetworkSettings": {
        "Bridge": "",
        "EndpointID": "56678f387a6924923b7d86213ee9c6f6743f2f65e8a9a36a4b161a9ecd102caf",
        "Gateway": "172.17.0.1",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "HairpinMode": false,
        "IPAddress": "172.17.0.6",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "MacAddress": "02:42:ac:11:00:06",
        "Networks": {
            "bridge": {
                "Aliases": null,
                "DriverOpts": null,
                "EndpointID": "56678f387a6924923b7d86213ee9c6f6743f2f65e8a9a36a4b161a9ecd102caf",
                "Gateway": "172.17.0.1",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAddress": "172.17.0.6",
                "IPAMConfig": null,
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "Links": null,
                "MacAddress": "02:42:ac:11:00:06",
                "NetworkID": "ce57a60dba91c81ffcad4c1585a57bad50b23c0c35c85633483fc24aa6b50b49"
            }
        },
        "Ports": {
            "8000/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "8050"
                }
            ]
        },
        "SandboxID": "991bd996fc186fa72d11d1f96cf35fd39cc0efc47869f43052dbc6be7637c910",
        "SandboxKey": "/var/run/docker/netns/991bd996fc18",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null
    },

Because the container is in an exited status, we are unable to confirm if it is part of the same network as the other two, therefore allowing it access. As a quick test, to see if this is an issue with it not being available during an exited state or if this is because it doesn't have one anyway, I force-quit my running container and confirmed that it does remove the networking details during its stopped state.

So, for the next stage of troubleshooting, I will want to see that configuration during an alive state. I recognize it is dying quickly, but I will need it to be started, then have a fresh inspect pulled so we can review if it is in the same network as the other two containers and therefore able to access them.

Additionally, it looks like we are still needing the TA container's logs, so if we can get a set of those complete logs, that would be helpful in reviewing this particular issue.

Th3M1CR0WAV3 commented 1 year ago

Hi I'm having issues getting further detail in the logs. Could you please confirm the appropriate steps (or environmental variables needing to be added) to provide these?

Tubearchivist inspect repaste

Tubearchivist logs

Tubearchivist logs 2

lamusmaser commented 1 year ago

The method for pulling the CSV output from Synology was provided earlier, as seen here:

...To do that, we'll need the full log stack.

To do so, please follow the below:

  1. Go to the container for TA and click the Details button.
  2. Go to the Log tab.
  3. Go to today's date. Then click on the Export button's dropdown menu. Select CSV from the options and let it download.
  4. Upload the log either to here directly or another text upload area and provide the link.

Alternatively to that, you can redirect from the CLI and output a log file there: docker logs tubearchivist >> tubearchivist_docker.log

That will give you a file that you can also upload.

Th3M1CR0WAV3 commented 1 year ago

tubearchivist.csv

lamusmaser commented 1 year ago

Let's blow them away and rebuild, just in case this is an issue of a stuck file somewhere that is causing problems. I'm not seeing anything in the configuration provided and the testing that was done that should be limiting that communication sequence, but there it is not allowing the TA container to connect appropriately. It gets as far as talking to the internal DNS, because it is properly referencing the correct IP address, but either the Gateway or something else is not letting that connection through.

To do so, let's go through the following steps: Shut down all three containers. Remove/delete all three containers. Delete the contents of the es, redis, cache, and media folders. If you have downloaded items in media, feel free to copy those out of there first. Rebuild, following the instructions that you were following previously.

If we see this issue again after that, then we'll see what we can do to dump all of the logs for review and go from there.

Th3M1CR0WAV3 commented 1 year ago

Did a complete reinstall but still having the same issues. Logs and inspects are attached. tubearchivist-redis inspect.txt tubearchivist-es inspect.txt tubearchivist inspect.txt tubearchivist-redis.csv tubearchivist.csv tubearchivist-es.csv

lamusmaser commented 1 year ago

From the ES container, can you test the following: (printf "PING\r\nQUIT\r\n";) | curl -v telnet://tubearchivist-redis:6379

I want to see if this is happening in all directions or only from the TA container.

Just as a heads up, the configs look fine. If the logs are really erroring out for TA when they are occurring, it is quite literally on the first step, which is why it was causing confusion about why they look so different.

I guess to cover all bases, can you provide the Synology model, current SynologyOS version, current Docker version, and if you happen to have Docker Compose installed?

Th3M1CR0WAV3 commented 1 year ago 
elasticsearch@tubearchivist-es:~$ (printf "PING\r\nQUIT\r\n";) | curl -v telnet://tubearchivist-redis:6379                                                                          
* Could not resolve host: tubearchivist-redis                                                                                                                                       
* Closing connection 0                                                                                                                                                              
curl: (6) Could not resolve host: tubearchivist-redis
Th3M1CR0WAV3 commented 1 year ago

Synology DS1821+ DSM 7.1.1-42962 Update 5 Docker Version: 20.10.3-1308 docker-compose version 1.28.5, build 24fb474e

lamusmaser commented 1 year ago

OK, let's go the compose route and see if that helps things out with stabilization - you seem to have enough technical experience that this can probably be an easier way to maintain them long-term anyway. Go ahead and shut down and remove those containers. You don't need to delete anything, but we're removing them so we can reuse their names.

Utilize this Synology-specific docker-compose.yml, replacing areas that have appropriate comments (I have updated those variables that I have from your previously provided inspects):

version: '3.3'

services:
  tubearchivist-ws:
    container_name: tubearchivist
    restart: always
    image: bbilly1/tubearchivist:latest
    ports:
      - 8050:8000
    volumes:
      - /volume1/docker/tubearchivist/media:/youtube
      - /volume1/docker/tubearchivist/cache:/cache
    environment:
      - ES_URL=http://tubearchivist-es:9200
      - REDIS_HOST=tubearchivist-redis
      - HOST_UID=1000
      - HOST_GID=0
      - "TA_HOST=synology.local"
      - TA_USERNAME=tubearchivist
      - "TA_PASSWORD=verysecret"
      - "ELASTIC_PASSWORD=verysecret"
      - TZ=Australia/Sydney
    depends_on:
      - tubearchivist-es
      - tubearchivist-redis
  tubearchivist-redis:
    image: redis/redis-stack-server:latest
    container_name: tubearchivist-redis
    restart: always
    environment:
      - "UID=1000"
      - "GID=0"
    volumes:
      - /volume1/docker/tubearchivist/redis:/data
    depends_on:
      - tubearchivist-es
  tubearchivist-es:
    image: bbilly1/tubearchivist-es
    container_name: tubearchivist-es
    restart: always
    environment:
      - "discovery.type=single-node"
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "UID=1000"
      - "GID=0"
      - "xpack.security.enabled=true"
      - "ELASTIC_PASSWORD=verysecret"
      - "path.repo=/usr/share/elasticsearch/data/snapshot"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - /volume1/docker/tubearchivist/es:/usr/share/elasticsearch/data

This docker-compose.yml will go in the /volume1/docker/tubearchivist directory. To both confirm that it works and ensure that it is pulling the latest versions of the images, use:

docker-compose pull

Once that is completed, we start up the containers:

docker-compose up -d

After the images are up, then you can review the logs and see if they are working. Fortunately, images that are created via Docker Compose are still accessible in the Synology Docker application, so you can still review the logs in that method. Alternatively, if you want to view logs for all the products in time order, you can always run:

docker-compose logs

OR to request logs for an individual container:

docker logs <container-name>
Th3M1CR0WAV3 commented 1 year ago 
Attaching to tubearchivist, tubearchivist-redis, tubearchivist-es
tubearchivist-es       | Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:23.604Z", "log.level": "WARN", "message":"unable to install syscall filter: ", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.JNANatives","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster","error.type":"java.lang.UnsupportedOperationException","error.message":"seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed","error.stack_trace":"java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:331)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:622)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:251)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:102)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:271)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:161)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:66)\n"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:24.611Z", "log.level": "INFO", "message":"version[8.7.0], pid[180], build[docker/09520b59b6bc1057340b55750186466ea715e30e/2023-03-27T16:31:09.816451435Z], OS[Linux/4.4.180+/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/19.0.2/19.0.2+7-44]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:24.613Z", "log.level": "INFO", "message":"JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:24.613Z", "log.level": "INFO", "message":"JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-14360707750801046178, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:27.535Z", "log.level": "INFO", "message":"Package versions: jackson-core=2.13.4, jackson-databind=2.13.4-2, jackson-dataformat-xml=2.13.4, jackson-datatype-jsr310=2.13.4, azure-core=1.34.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"com.azure.core.implementation.jackson.JacksonVersion","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.523Z", "log.level": "INFO", "message":"loaded module [aggregations]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.523Z", "log.level": "INFO", "message":"loaded module [analysis-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.524Z", "log.level": "INFO", "message":"loaded module [apm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.524Z", "log.level": "INFO", "message":"loaded module [blob-cache]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.524Z", "log.level": "INFO", "message":"loaded module [constant-keyword]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.524Z", "log.level": "INFO", "message":"loaded module [data-streams]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.524Z", "log.level": "INFO", "message":"loaded module [frozen-indices]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.525Z", "log.level": "INFO", "message":"loaded module [ingest-attachment]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.525Z", "log.level": "INFO", "message":"loaded module [ingest-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.525Z", "log.level": "INFO", "message":"loaded module [ingest-geoip]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.525Z", "log.level": "INFO", "message":"loaded module [ingest-user-agent]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.525Z", "log.level": "INFO", "message":"loaded module [kibana]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.526Z", "log.level": "INFO", "message":"loaded module [lang-expression]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.526Z", "log.level": "INFO", "message":"loaded module [lang-mustache]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.526Z", "log.level": "INFO", "message":"loaded module [lang-painless]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.526Z", "log.level": "INFO", "message":"loaded module [legacy-geo]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.526Z", "log.level": "INFO", "message":"loaded module [mapper-extras]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.527Z", "log.level": "INFO", "message":"loaded module [mapper-version]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.527Z", "log.level": "INFO", "message":"loaded module [old-lucene-versions]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.527Z", "log.level": "INFO", "message":"loaded module [parent-join]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.527Z", "log.level": "INFO", "message":"loaded module [percolator]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.527Z", "log.level": "INFO", "message":"loaded module [rank-eval]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.527Z", "log.level": "INFO", "message":"loaded module [reindex]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.528Z", "log.level": "INFO", "message":"loaded module [repositories-metering-api]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.528Z", "log.level": "INFO", "message":"loaded module [repository-azure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.528Z", "log.level": "INFO", "message":"loaded module [repository-gcs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.528Z", "log.level": "INFO", "message":"loaded module [repository-s3]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.528Z", "log.level": "INFO", "message":"loaded module [repository-url]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.528Z", "log.level": "INFO", "message":"loaded module [runtime-fields-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.529Z", "log.level": "INFO", "message":"loaded module [search-business-rules]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-redis    | 9:C 01 Jun 2023 05:27:18.440 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
tubearchivist-redis    | 9:C 01 Jun 2023 05:27:18.440 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=9, just started
tubearchivist-redis    | 9:C 01 Jun 2023 05:27:18.440 # Configuration loaded
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.441 * monotonic clock: POSIX clock_gettime
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.441 * Running mode=standalone, port=6379.
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.441 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.441 # Server initialized
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.441 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * <search> Redis version found by RedisSearch : 6.2.12 - oss
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * <search> RediSearch version 2.6.9 (Git=HEAD-b205a1d5)
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * <search> Low level api version 1 initialized successfully
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * <search> concurrent writes: OFF, gc: ON, prefix min length: 2, prefix max expansions: 200, query timeout (ms): 500, timeout policy: return, cursor read size: 1000, cursor max idle (ms): 300000, max doctable size: 1000000, max number of search results:  10000, search pool size: 20, index pool size: 8, 
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * <search> Initialized thread pool!
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * <search> Enabled diskless replication
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * <search> Enabled role change notification
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * Module 'search' loaded from /opt/redis-stack/lib/redisearch.so
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.711 * <graph> Enabled role change notification
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.711 * <graph> Starting up RedisGraph version 2.10.10.
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.713 * <graph> Thread pool created, using 8 threads.
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.713 * <graph> Maximum number of OpenMP threads set to 8
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.713 * Module 'graph' loaded from /opt/redis-stack/lib/redisgraph.so
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * <timeseries> RedisTimeSeries version 10810, git_sha=78d8254a4ed499e4e24566ab76c9275130c8786b
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * <timeseries> Redis version found by RedisTimeSeries : 6.2.12 - oss
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * <timeseries> loaded default CHUNK_SIZE_BYTES policy: 4096
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * <timeseries> loaded server DUPLICATE_POLICY: block
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * <timeseries> Setting default series ENCODING to: compressed
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * <timeseries> Detected redis oss
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * <timeseries> Enabled diskless replication
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * Module 'timeseries' loaded from /opt/redis-stack/lib/redistimeseries.so
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> version: 20407 git sha: a4412b3 branch: HEAD
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> Exported RedisJSON_V1 API
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> Exported RedisJSON_V2 API
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> Exported RedisJSON_V3 API
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> Enabled diskless replication
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> Created new data type 'ReJSON-RL'
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * Module 'ReJSON' loaded from /opt/redis-stack/lib/rejson.so
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <search> Acquired RedisJSON_V3 API
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <graph> Acquired RedisJSON_V1 API
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.752 * <bf> RedisBloom version 2.4.5 (Git=unknown)
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.752 * Module 'bf' loaded from /opt/redis-stack/lib/redisbloom.so
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.752 * Ready to accept connections
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.529Z", "log.level": "INFO", "message":"loaded module [searchable-snapshots]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.529Z", "log.level": "INFO", "message":"loaded module [snapshot-based-recoveries]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.529Z", "log.level": "INFO", "message":"loaded module [snapshot-repo-test-kit]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.529Z", "log.level": "INFO", "message":"loaded module [spatial]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.529Z", "log.level": "INFO", "message":"loaded module [transform]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.530Z", "log.level": "INFO", "message":"loaded module [transport-netty4]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.530Z", "log.level": "INFO", "message":"loaded module [unsigned-long]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.530Z", "log.level": "INFO", "message":"loaded module [vector-tile]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.530Z", "log.level": "INFO", "message":"loaded module [wildcard]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.530Z", "log.level": "INFO", "message":"loaded module [x-pack-aggregate-metric]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.530Z", "log.level": "INFO", "message":"loaded module [x-pack-analytics]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.531Z", "log.level": "INFO", "message":"loaded module [x-pack-async]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.531Z", "log.level": "INFO", "message":"loaded module [x-pack-async-search]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.531Z", "log.level": "INFO", "message":"loaded module [x-pack-autoscaling]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.531Z", "log.level": "INFO", "message":"loaded module [x-pack-ccr]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.531Z", "log.level": "INFO", "message":"loaded module [x-pack-core]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.531Z", "log.level": "INFO", "message":"loaded module [x-pack-deprecation]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.532Z", "log.level": "INFO", "message":"loaded module [x-pack-enrich]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.532Z", "log.level": "INFO", "message":"loaded module [x-pack-eql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.532Z", "log.level": "INFO", "message":"loaded module [x-pack-fleet]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.532Z", "log.level": "INFO", "message":"loaded module [x-pack-graph]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.532Z", "log.level": "INFO", "message":"loaded module [x-pack-identity-provider]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.532Z", "log.level": "INFO", "message":"loaded module [x-pack-ilm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.533Z", "log.level": "INFO", "message":"loaded module [x-pack-logstash]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.533Z", "log.level": "INFO", "message":"loaded module [x-pack-ml]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.533Z", "log.level": "INFO", "message":"loaded module [x-pack-monitoring]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.533Z", "log.level": "INFO", "message":"loaded module [x-pack-profiling]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.533Z", "log.level": "INFO", "message":"loaded module [x-pack-ql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.533Z", "log.level": "INFO", "message":"loaded module [x-pack-rollup]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.534Z", "log.level": "INFO", "message":"loaded module [x-pack-security]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.534Z", "log.level": "INFO", "message":"loaded module [x-pack-shutdown]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.534Z", "log.level": "INFO", "message":"loaded module [x-pack-sql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.534Z", "log.level": "INFO", "message":"loaded module [x-pack-stack]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.534Z", "log.level": "INFO", "message":"loaded module [x-pack-text-structure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.534Z", "log.level": "INFO", "message":"loaded module [x-pack-voting-only-node]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.535Z", "log.level": "INFO", "message":"loaded module [x-pack-watcher]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.535Z", "log.level": "INFO", "message":"loaded module [x-pack-write-load-forecaster]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.535Z", "log.level": "INFO", "message":"no plugins loaded", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:33.766Z", "log.level": "WARN", "message":"Jun 01, 2023 5:27:33 AM org.apache.lucene.store.MemorySegmentIndexInputProvider <init>", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"stderr","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:33.778Z", "log.level": "INFO", "message":"using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/cachedev_0)]], net usable_space [15.2tb], net total_space [62.8tb], types [btrfs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:33.778Z", "log.level": "INFO", "message":"heap size [512mb], compressed ordinary object pointers [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:33.935Z", "log.level": "INFO", "message":"node name [175730da42c0], node ID [ZSqgorS4R-GItb-xYY9R2w], cluster name [docker-cluster], roles [transform, data_content, data_warm, master, remote_cluster_client, data, data_cold, ingest, data_frozen, ml, data_hot]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:36.850Z", "log.level": "INFO", "message":"[controller/211] [Main.cc@123] controller (64 bit): Version 8.7.0 (Build e4e1c23721e58c) Copyright (c) 2023 Elasticsearch BV", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"ml-cpp-log-tail-thread","log.logger":"org.elasticsearch.xpack.ml.process.logging.CppLogMessageHandler","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:37.151Z", "log.level": "INFO", "message":"Security is enabled", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.security.Security","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:37.694Z", "log.level": "INFO", "message":"parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.security.authz.store.FileRolesStore","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:38.623Z", "log.level": "INFO", "message":"Profiling is enabled", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.profiler.ProfilingPlugin","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:39.603Z", "log.level": "INFO", "message":"creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=512mb}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.transport.netty4.NettyAllocator","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:39.639Z", "log.level": "INFO", "message":"using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:39.690Z", "log.level": "INFO", "message":"using discovery type [single-node] and seed hosts providers [settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.discovery.DiscoveryModule","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:41.473Z", "log.level": "INFO", "message":"initialized", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:41.473Z", "log.level": "INFO", "message":"starting ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:41.735Z", "log.level": "INFO", "message":"persistent cache index loaded", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:41.736Z", "log.level": "INFO", "message":"deprecation component started", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingComponent","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:41.864Z", "log.level": "INFO", "message":"publish_address {172.18.0.2:9300}, bound_addresses {0.0.0.0:9300}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.transport.TransportService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:43.058Z", "log.level": "WARN", "message":"max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.BootstrapChecks","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:43.059Z", "log.level": "WARN", "message":"system call filters failed to install; check the logs and fix your configuration", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.BootstrapChecks","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:43.059Z", "log.level": "WARN", "message":"Transport SSL must be enabled if security is enabled. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.BootstrapChecks","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:43.060Z", "log.level": "INFO", "message":"this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to [175730da42c0]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.cluster.coordination.ClusterBootstrapService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:43.071Z", "log.level": "INFO", "message":"setting initial configuration to VotingConfiguration{ZSqgorS4R-GItb-xYY9R2w}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.cluster.coordination.Coordinator","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:44.063Z", "log.level": "INFO", "message":"elected-as-master ([1] nodes joined)[_FINISH_ELECTION_, {175730da42c0}{ZSqgorS4R-GItb-xYY9R2w}{4MbypjmkTyeRlyGpkFsrgQ}{175730da42c0}{172.18.0.2}{172.18.0.2:9300}{cdfhilmrstw}{8.7.0} completing election], term: 1, version: 1, delta: master node changed {previous [], current [{175730da42c0}{ZSqgorS4R-GItb-xYY9R2w}{4MbypjmkTyeRlyGpkFsrgQ}{175730da42c0}{172.18.0.2}{172.18.0.2:9300}{cdfhilmrstw}{8.7.0}]}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:45.262Z", "log.level": "INFO", "message":"cluster UUID set to [qwVY1DaCTC2LnZ88CorpUQ]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.CoordinationState","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:45.780Z", "log.level": "INFO", "message":"master node changed {previous [], current [{175730da42c0}{ZSqgorS4R-GItb-xYY9R2w}{4MbypjmkTyeRlyGpkFsrgQ}{175730da42c0}{172.18.0.2}{172.18.0.2:9300}{cdfhilmrstw}{8.7.0}]}, term: 1, version: 1, reason: Publication{term=1, version=1}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.ClusterApplierService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:45.858Z", "log.level": "INFO", "message":"starting file settings watcher ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.reservedstate.service.FileSettingsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:45.863Z", "log.level": "INFO", "message":"file settings service up and running [tid=55]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[file-settings-watcher]","log.logger":"org.elasticsearch.reservedstate.service.FileSettingsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:45.868Z", "log.level": "INFO", "message":"publish_address {172.18.0.2:9200}, bound_addresses {0.0.0.0:9200}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.http.AbstractHttpServerTransport","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:45.869Z", "log.level": "INFO", "message":"node-join: [{175730da42c0}{ZSqgorS4R-GItb-xYY9R2w}{4MbypjmkTyeRlyGpkFsrgQ}{175730da42c0}{172.18.0.2}{172.18.0.2:9300}{cdfhilmrstw}{8.7.0}] with reason [completing election]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.coordination.NodeJoinExecutor","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:45.869Z", "log.level": "INFO", "message":"started {175730da42c0}{ZSqgorS4R-GItb-xYY9R2w}{4MbypjmkTyeRlyGpkFsrgQ}{175730da42c0}{172.18.0.2}{172.18.0.2:9300}{cdfhilmrstw}{8.7.0}{xpack.installed=true, ml.machine_memory=20946878464, ml.allocated_processors=8, ml.max_jvm_size=536870912, ml.allocated_processors_double=8.0}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.159Z", "log.level": "INFO", "message":"recovered [0] indices into cluster_state", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.gateway.GatewayService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.373Z", "log.level": "INFO", "message":"adding index template [.ml-stats] for index patterns [.ml-stats-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.381Z", "log.level": "INFO", "message":"adding index template [.ml-state] for index patterns [.ml-state*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.411Z", "log.level": "INFO", "message":"adding index template [.monitoring-logstash-mb] for index patterns [.monitoring-logstash-8-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.436Z", "log.level": "INFO", "message":"adding index template [.monitoring-kibana-mb] for index patterns [.monitoring-kibana-8-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.449Z", "log.level": "INFO", "message":"adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.461Z", "log.level": "INFO", "message":"adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.479Z", "log.level": "INFO", "message":"adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.499Z", "log.level": "INFO", "message":"adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.520Z", "log.level": "INFO", "message":"adding template [.monitoring-es] for index patterns [.monitoring-es-7-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.531Z", "log.level": "INFO", "message":"adding index template [.ml-notifications-000002] for index patterns [.ml-notifications-000002]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.556Z", "log.level": "INFO", "message":"adding index template [.ml-anomalies-] for index patterns [.ml-anomalies-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.577Z", "log.level": "INFO", "message":"adding index template [.monitoring-ent-search-mb] for index patterns [.monitoring-ent-search-8-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.622Z", "log.level": "INFO", "message":"adding index template [.monitoring-es-mb] for index patterns [.monitoring-es-8-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.652Z", "log.level": "INFO", "message":"adding index template [.monitoring-beats-mb] for index patterns [.monitoring-beats-8-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.660Z", "log.level": "INFO", "message":"adding component template [logs-mappings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.665Z", "log.level": "INFO", "message":"adding component template [logs-settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.669Z", "log.level": "INFO", "message":"adding component template [metrics-settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.673Z", "log.level": "INFO", "message":"adding component template [synthetics-settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.685Z", "log.level": "INFO", "message":"adding component template [data-streams-mappings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.691Z", "log.level": "INFO", "message":"adding component template [metrics-mappings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.696Z", "log.level": "INFO", "message":"adding component template [synthetics-mappings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.712Z", "log.level": "INFO", "message":"adding index template [.watch-history-16] for index patterns [.watcher-history-16*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.722Z", "log.level": "INFO", "message":"adding index template [ilm-history] for index patterns [ilm-history-5*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.728Z", "log.level": "INFO", "message":"adding component template [.deprecation-indexing-mappings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.732Z", "log.level": "INFO", "message":"adding component template [.deprecation-indexing-settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.740Z", "log.level": "INFO", "message":"adding index template [.slm-history] for index patterns [.slm-history-5*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.747Z", "log.level": "INFO", "message":"adding index template [.fleet-file-data] for index patterns [.fleet-file-data-*-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.754Z", "log.level": "INFO", "message":"adding index template [.fleet-files] for index patterns [.fleet-files-*-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:47.422Z", "log.level": "INFO", "message":"adding index template [metrics] for index patterns [metrics-*-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:47.432Z", "log.level": "INFO", "message":"adding index template [synthetics] for index patterns [synthetics-*-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:47.440Z", "log.level": "INFO", "message":"adding index template [.deprecation-indexing-template] for index patterns [.logs-deprecation.*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:47.449Z", "log.level": "INFO", "message":"adding index template [logs] for index patterns [logs-*-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:47.852Z", "log.level": "INFO", "message":"adding index lifecycle policy [.monitoring-8-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:48.280Z", "log.level": "INFO", "message":"adding index lifecycle policy [ml-size-based-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:48.652Z", "log.level": "INFO", "message":"adding index lifecycle policy [logs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:49.394Z", "log.level": "INFO", "message":"adding index lifecycle policy [synthetics]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:49.760Z", "log.level": "INFO", "message":"adding index lifecycle policy [metrics]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:50.244Z", "log.level": "INFO", "message":"adding index lifecycle policy [30-days-default]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:50.617Z", "log.level": "INFO", "message":"adding index lifecycle policy [7-days-default]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:51.245Z", "log.level": "INFO", "message":"adding index lifecycle policy [365-days-default]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:51.653Z", "log.level": "INFO", "message":"adding index lifecycle policy [90-days-default]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:52.011Z", "log.level": "INFO", "message":"adding index lifecycle policy [180-days-default]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:52.410Z", "log.level": "INFO", "message":"adding index lifecycle policy [watch-history-ilm-policy-16]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:52.860Z", "log.level": "INFO", "message":"adding index lifecycle policy [ilm-history-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:53.282Z", "log.level": "INFO", "message":"adding index lifecycle policy [slm-history-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:53.625Z", "log.level": "INFO", "message":"adding index lifecycle policy [.deprecation-indexing-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:53.992Z", "log.level": "INFO", "message":"adding index lifecycle policy [.fleet-files-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:55.092Z", "log.level": "INFO", "message":"adding index lifecycle policy [.fleet-file-data-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:55.433Z", "log.level": "INFO", "message":"adding index lifecycle policy [.fleet-actions-results-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:56.815Z", "log.level": "INFO", "message":"Node [{175730da42c0}{ZSqgorS4R-GItb-xYY9R2w}] is selected as the current health node.", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][management][T#1]","log.logger":"org.elasticsearch.health.node.selection.HealthNodeTaskExecutor","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:57.694Z", "log.level": "INFO", "message":"license [730ecfd3-9f9d-4b1a-9c05-0e8c6539f246] mode [basic] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.LicenseService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:57.696Z", "log.level": "INFO", "message":"license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
root@Shep-Media:/volume1/docker/tubearchivist# docker-compose logs
Attaching to tubearchivist, tubearchivist-redis, tubearchivist-es
tubearchivist          | Traceback (most recent call last):
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 698, in connect
tubearchivist          |     sock = self.retry.call_with_retry(
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/retry.py", line 51, in call_with_retry
tubearchivist          |     raise error
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
tubearchivist          |     return do()
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 699, in <lambda>
tubearchivist          |     lambda: self._connect(), lambda error: self.disconnect(error)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 987, in _connect
tubearchivist          |     raise err
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 975, in _connect
tubearchivist          |     sock.connect(socket_address)
tubearchivist          | TimeoutError: [Errno 110] Connection timed out
tubearchivist          | 
tubearchivist          | During handling of the above exception, another exception occurred:
tubearchivist          | 
tubearchivist          | Traceback (most recent call last):
tubearchivist          |   File "/app/manage.py", line 23, in <module>
tubearchivist          |     main()
tubearchivist          |   File "/app/manage.py", line 19, in main
tubearchivist          |     execute_from_command_line(sys.argv)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
tubearchivist          |     utility.execute()
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 382, in execute
tubearchivist          |     settings.INSTALLED_APPS
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 102, in __getattr__
tubearchivist          |     self._setup(name)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 89, in _setup
tubearchivist          |     self._wrapped = Settings(settings_module)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 217, in __init__
tubearchivist          |     mod = importlib.import_module(self.SETTINGS_MODULE)
tubearchivist          |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
tubearchivist          |     return _bootstrap._gcd_import(name[level:], package, level)
tubearchivist          |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
tubearchivist          |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
tubearchivist          |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
tubearchivist          |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
tubearchivist          |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
tubearchivist          |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
tubearchivist          |   File "/app/config/settings.py", line 20, in <module>
tubearchivist          |     from home.src.ta.config import AppConfig
tubearchivist          |   File "/app/home/__init__.py", line 3, in <module>
tubearchivist          |     from .tasks import app as celery_app
tubearchivist          |   File "/app/home/tasks.py", line 12, in <module>
tubearchivist          |     from home.src.download.queue import PendingList
tubearchivist          |   File "/app/home/src/download/queue.py", line 10, in <module>
tubearchivist          |     from home.src.download.subscriptions import (
tubearchivist          |   File "/app/home/src/download/subscriptions.py", line 8, in <module>
tubearchivist          |     from home.src.download.thumbnails import ThumbManager
tubearchivist          |   File "/app/home/src/download/thumbnails.py", line 21, in <module>
tubearchivist          |     class ThumbManagerBase:
tubearchivist          |   File "/app/home/src/download/thumbnails.py", line 24, in ThumbManagerBase
tubearchivist          |     CONFIG = AppConfig().config
tubearchivist          |   File "/app/home/src/ta/config.py", line 23, in __init__
tubearchivist          |     self.config = self.get_config()
tubearchivist          |   File "/app/home/src/ta/config.py", line 28, in get_config
tubearchivist          |     config = self.get_config_redis()
tubearchivist          |   File "/app/home/src/ta/config.py", line 70, in get_config_redis
tubearchivist          |     config = RedisArchivist().get_message("config")
tubearchivist          |   File "/app/home/src/ta/ta_redis.py", line 59, in get_message
tubearchivist          |     reply = self.conn.execute_command("JSON.GET", self.NAME_SPACE + key)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/client.py", line 1255, in execute_command
tubearchivist          |     conn = self.connection or pool.get_connection(command_name, **options)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 1442, in get_connection
tubearchivist          |     connection.connect()
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 702, in connect
tubearchivist          |     raise TimeoutError("Timeout connecting to server")
tubearchivist          | redis.exceptions.TimeoutError: Timeout connecting to server
tubearchivist          | Traceback (most recent call last):
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 698, in connect
tubearchivist          |     sock = self.retry.call_with_retry(
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/retry.py", line 51, in call_with_retry
tubearchivist          |     raise error
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
tubearchivist          |     return do()
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 699, in <lambda>
tubearchivist          |     lambda: self._connect(), lambda error: self.disconnect(error)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 987, in _connect
tubearchivist          |     raise err
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 975, in _connect
tubearchivist          |     sock.connect(socket_address)
tubearchivist          | TimeoutError: [Errno 110] Connection timed out
tubearchivist          | 
tubearchivist          | During handling of the above exception, another exception occurred:
tubearchivist          | 
tubearchivist          | Traceback (most recent call last):
tubearchivist          |   File "/app/manage.py", line 23, in <module>
tubearchivist          |     main()
tubearchivist          |   File "/app/manage.py", line 19, in main
tubearchivist          |     execute_from_command_line(sys.argv)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
tubearchivist          |     utility.execute()
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 382, in execute
tubearchivist          |     settings.INSTALLED_APPS
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 102, in __getattr__
tubearchivist          |     self._setup(name)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 89, in _setup
tubearchivist          |     self._wrapped = Settings(settings_module)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 217, in __init__
tubearchivist          |     mod = importlib.import_module(self.SETTINGS_MODULE)
tubearchivist          |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
tubearchivist          |     return _bootstrap._gcd_import(name[level:], package, level)
tubearchivist          |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
tubearchivist          |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
tubearchivist          |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
tubearchivist          |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
tubearchivist          |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
tubearchivist          |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
tubearchivist          |   File "/app/config/settings.py", line 20, in <module>
tubearchivist-es       | Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:23.604Z", "log.level": "WARN", "message":"unable to install syscall filter: ", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.JNANatives","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster","error.type":"java.lang.UnsupportedOperationException","error.message":"seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed","error.stack_trace":"java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:331)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:622)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:251)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:102)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.bootstrap.Elasticsearch.initializeNatives(Elasticsearch.java:271)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase2(Elasticsearch.java:161)\n\tat org.elasticsearch.server@8.7.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:66)\n"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:24.611Z", "log.level": "INFO", "message":"version[8.7.0], pid[180], build[docker/09520b59b6bc1057340b55750186466ea715e30e/2023-03-27T16:31:09.816451435Z], OS[Linux/4.4.180+/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/19.0.2/19.0.2+7-44]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:24.613Z", "log.level": "INFO", "message":"JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:24.613Z", "log.level": "INFO", "message":"JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Des.cgroups.hierarchy.override=/, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-14360707750801046178, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -Xms512m, -Xmx512m, -XX:MaxDirectMemorySize=268435456, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=docker, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:27.535Z", "log.level": "INFO", "message":"Package versions: jackson-core=2.13.4, jackson-databind=2.13.4-2, jackson-dataformat-xml=2.13.4, jackson-datatype-jsr310=2.13.4, azure-core=1.34.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"com.azure.core.implementation.jackson.JacksonVersion","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.523Z", "log.level": "INFO", "message":"loaded module [aggregations]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.523Z", "log.level": "INFO", "message":"loaded module [analysis-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.524Z", "log.level": "INFO", "message":"loaded module [apm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.524Z", "log.level": "INFO", "message":"loaded module [blob-cache]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.524Z", "log.level": "INFO", "message":"loaded module [constant-keyword]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.524Z", "log.level": "INFO", "message":"loaded module [data-streams]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.524Z", "log.level": "INFO", "message":"loaded module [frozen-indices]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.525Z", "log.level": "INFO", "message":"loaded module [ingest-attachment]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.525Z", "log.level": "INFO", "message":"loaded module [ingest-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.525Z", "log.level": "INFO", "message":"loaded module [ingest-geoip]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.525Z", "log.level": "INFO", "message":"loaded module [ingest-user-agent]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.525Z", "log.level": "INFO", "message":"loaded module [kibana]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.526Z", "log.level": "INFO", "message":"loaded module [lang-expression]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.526Z", "log.level": "INFO", "message":"loaded module [lang-mustache]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.526Z", "log.level": "INFO", "message":"loaded module [lang-painless]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.526Z", "log.level": "INFO", "message":"loaded module [legacy-geo]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.526Z", "log.level": "INFO", "message":"loaded module [mapper-extras]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.527Z", "log.level": "INFO", "message":"loaded module [mapper-version]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.527Z", "log.level": "INFO", "message":"loaded module [old-lucene-versions]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.527Z", "log.level": "INFO", "message":"loaded module [parent-join]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.527Z", "log.level": "INFO", "message":"loaded module [percolator]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.527Z", "log.level": "INFO", "message":"loaded module [rank-eval]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.527Z", "log.level": "INFO", "message":"loaded module [reindex]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.528Z", "log.level": "INFO", "message":"loaded module [repositories-metering-api]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.528Z", "log.level": "INFO", "message":"loaded module [repository-azure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.528Z", "log.level": "INFO", "message":"loaded module [repository-gcs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.528Z", "log.level": "INFO", "message":"loaded module [repository-s3]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.528Z", "log.level": "INFO", "message":"loaded module [repository-url]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.528Z", "log.level": "INFO", "message":"loaded module [runtime-fields-common]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.529Z", "log.level": "INFO", "message":"loaded module [search-business-rules]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist          |     from home.src.ta.config import AppConfig
tubearchivist          |   File "/app/home/__init__.py", line 3, in <module>
tubearchivist          |     from .tasks import app as celery_app
tubearchivist          |   File "/app/home/tasks.py", line 12, in <module>
tubearchivist          |     from home.src.download.queue import PendingList
tubearchivist          |   File "/app/home/src/download/queue.py", line 10, in <module>
tubearchivist          |     from home.src.download.subscriptions import (
tubearchivist          |   File "/app/home/src/download/subscriptions.py", line 8, in <module>
tubearchivist          |     from home.src.download.thumbnails import ThumbManager
tubearchivist          |   File "/app/home/src/download/thumbnails.py", line 21, in <module>
tubearchivist          |     class ThumbManagerBase:
tubearchivist          |   File "/app/home/src/download/thumbnails.py", line 24, in ThumbManagerBase
tubearchivist          |     CONFIG = AppConfig().config
tubearchivist          |   File "/app/home/src/ta/config.py", line 23, in __init__
tubearchivist          |     self.config = self.get_config()
tubearchivist          |   File "/app/home/src/ta/config.py", line 28, in get_config
tubearchivist          |     config = self.get_config_redis()
tubearchivist          |   File "/app/home/src/ta/config.py", line 70, in get_config_redis
tubearchivist          |     config = RedisArchivist().get_message("config")
tubearchivist          |   File "/app/home/src/ta/ta_redis.py", line 59, in get_message
tubearchivist          |     reply = self.conn.execute_command("JSON.GET", self.NAME_SPACE + key)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/client.py", line 1255, in execute_command
tubearchivist          |     conn = self.connection or pool.get_connection(command_name, **options)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 1442, in get_connection
tubearchivist          |     connection.connect()
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 702, in connect
tubearchivist          |     raise TimeoutError("Timeout connecting to server")
tubearchivist          | redis.exceptions.TimeoutError: Timeout connecting to server
tubearchivist          | Traceback (most recent call last):
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 698, in connect
tubearchivist          |     sock = self.retry.call_with_retry(
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/retry.py", line 51, in call_with_retry
tubearchivist          |     raise error
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
tubearchivist          |     return do()
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 699, in <lambda>
tubearchivist          |     lambda: self._connect(), lambda error: self.disconnect(error)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 987, in _connect
tubearchivist          |     raise err
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 975, in _connect
tubearchivist          |     sock.connect(socket_address)
tubearchivist          | TimeoutError: [Errno 110] Connection timed out
tubearchivist          | 
tubearchivist          | During handling of the above exception, another exception occurred:
tubearchivist          | 
tubearchivist          | Traceback (most recent call last):
tubearchivist          |   File "/app/manage.py", line 23, in <module>
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.529Z", "log.level": "INFO", "message":"loaded module [searchable-snapshots]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.529Z", "log.level": "INFO", "message":"loaded module [snapshot-based-recoveries]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.529Z", "log.level": "INFO", "message":"loaded module [snapshot-repo-test-kit]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.529Z", "log.level": "INFO", "message":"loaded module [spatial]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.529Z", "log.level": "INFO", "message":"loaded module [transform]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.530Z", "log.level": "INFO", "message":"loaded module [transport-netty4]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.530Z", "log.level": "INFO", "message":"loaded module [unsigned-long]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.530Z", "log.level": "INFO", "message":"loaded module [vector-tile]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.530Z", "log.level": "INFO", "message":"loaded module [wildcard]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.530Z", "log.level": "INFO", "message":"loaded module [x-pack-aggregate-metric]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.530Z", "log.level": "INFO", "message":"loaded module [x-pack-analytics]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.531Z", "log.level": "INFO", "message":"loaded module [x-pack-async]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.531Z", "log.level": "INFO", "message":"loaded module [x-pack-async-search]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.531Z", "log.level": "INFO", "message":"loaded module [x-pack-autoscaling]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.531Z", "log.level": "INFO", "message":"loaded module [x-pack-ccr]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.531Z", "log.level": "INFO", "message":"loaded module [x-pack-core]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.531Z", "log.level": "INFO", "message":"loaded module [x-pack-deprecation]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.532Z", "log.level": "INFO", "message":"loaded module [x-pack-enrich]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.532Z", "log.level": "INFO", "message":"loaded module [x-pack-eql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.532Z", "log.level": "INFO", "message":"loaded module [x-pack-fleet]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.532Z", "log.level": "INFO", "message":"loaded module [x-pack-graph]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist          |     main()
tubearchivist          |   File "/app/manage.py", line 19, in main
tubearchivist          |     execute_from_command_line(sys.argv)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
tubearchivist          |     utility.execute()
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 382, in execute
tubearchivist          |     settings.INSTALLED_APPS
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 102, in __getattr__
tubearchivist          |     self._setup(name)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 89, in _setup
tubearchivist          |     self._wrapped = Settings(settings_module)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 217, in __init__
tubearchivist          |     mod = importlib.import_module(self.SETTINGS_MODULE)
tubearchivist          |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
tubearchivist          |     return _bootstrap._gcd_import(name[level:], package, level)
tubearchivist          |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
tubearchivist          |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
tubearchivist          |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
tubearchivist          |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
tubearchivist          |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
tubearchivist          |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
tubearchivist          |   File "/app/config/settings.py", line 20, in <module>
tubearchivist          |     from home.src.ta.config import AppConfig
tubearchivist          |   File "/app/home/__init__.py", line 3, in <module>
tubearchivist          |     from .tasks import app as celery_app
tubearchivist          |   File "/app/home/tasks.py", line 12, in <module>
tubearchivist          |     from home.src.download.queue import PendingList
tubearchivist          |   File "/app/home/src/download/queue.py", line 10, in <module>
tubearchivist          |     from home.src.download.subscriptions import (
tubearchivist          |   File "/app/home/src/download/subscriptions.py", line 8, in <module>
tubearchivist          |     from home.src.download.thumbnails import ThumbManager
tubearchivist          |   File "/app/home/src/download/thumbnails.py", line 21, in <module>
tubearchivist          |     class ThumbManagerBase:
tubearchivist          |   File "/app/home/src/download/thumbnails.py", line 24, in ThumbManagerBase
tubearchivist          |     CONFIG = AppConfig().config
tubearchivist          |   File "/app/home/src/ta/config.py", line 23, in __init__
tubearchivist          |     self.config = self.get_config()
tubearchivist          |   File "/app/home/src/ta/config.py", line 28, in get_config
tubearchivist          |     config = self.get_config_redis()
tubearchivist          |   File "/app/home/src/ta/config.py", line 70, in get_config_redis
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.532Z", "log.level": "INFO", "message":"loaded module [x-pack-identity-provider]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.532Z", "log.level": "INFO", "message":"loaded module [x-pack-ilm]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.533Z", "log.level": "INFO", "message":"loaded module [x-pack-logstash]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.533Z", "log.level": "INFO", "message":"loaded module [x-pack-ml]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.533Z", "log.level": "INFO", "message":"loaded module [x-pack-monitoring]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.533Z", "log.level": "INFO", "message":"loaded module [x-pack-profiling]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.533Z", "log.level": "INFO", "message":"loaded module [x-pack-ql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.533Z", "log.level": "INFO", "message":"loaded module [x-pack-rollup]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.534Z", "log.level": "INFO", "message":"loaded module [x-pack-security]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.534Z", "log.level": "INFO", "message":"loaded module [x-pack-shutdown]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.534Z", "log.level": "INFO", "message":"loaded module [x-pack-sql]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.534Z", "log.level": "INFO", "message":"loaded module [x-pack-stack]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.534Z", "log.level": "INFO", "message":"loaded module [x-pack-text-structure]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.534Z", "log.level": "INFO", "message":"loaded module [x-pack-voting-only-node]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.535Z", "log.level": "INFO", "message":"loaded module [x-pack-watcher]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.535Z", "log.level": "INFO", "message":"loaded module [x-pack-write-load-forecaster]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:29.535Z", "log.level": "INFO", "message":"no plugins loaded", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.plugins.PluginsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:33.766Z", "log.level": "WARN", "message":"Jun 01, 2023 5:27:33 AM org.apache.lucene.store.MemorySegmentIndexInputProvider <init>", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"stderr","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:33.778Z", "log.level": "INFO", "message":"using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/cachedev_0)]], net usable_space [15.2tb], net total_space [62.8tb], types [btrfs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:33.778Z", "log.level": "INFO", "message":"heap size [512mb], compressed ordinary object pointers [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.env.NodeEnvironment","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:33.935Z", "log.level": "INFO", "message":"node name [175730da42c0], node ID [ZSqgorS4R-GItb-xYY9R2w], cluster name [docker-cluster], roles [transform, data_content, data_warm, master, remote_cluster_client, data, data_cold, ingest, data_frozen, ml, data_hot]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist          |     config = RedisArchivist().get_message("config")
tubearchivist          |   File "/app/home/src/ta/ta_redis.py", line 59, in get_message
tubearchivist          |     reply = self.conn.execute_command("JSON.GET", self.NAME_SPACE + key)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/client.py", line 1255, in execute_command
tubearchivist          |     conn = self.connection or pool.get_connection(command_name, **options)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 1442, in get_connection
tubearchivist          |     connection.connect()
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 702, in connect
tubearchivist          |     raise TimeoutError("Timeout connecting to server")
tubearchivist          | redis.exceptions.TimeoutError: Timeout connecting to server
tubearchivist          | Traceback (most recent call last):
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 698, in connect
tubearchivist          |     sock = self.retry.call_with_retry(
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/retry.py", line 51, in call_with_retry
tubearchivist          |     raise error
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
tubearchivist          |     return do()
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 699, in <lambda>
tubearchivist          |     lambda: self._connect(), lambda error: self.disconnect(error)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 987, in _connect
tubearchivist          |     raise err
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 975, in _connect
tubearchivist          |     sock.connect(socket_address)
tubearchivist          | TimeoutError: [Errno 110] Connection timed out
tubearchivist          | 
tubearchivist          | During handling of the above exception, another exception occurred:
tubearchivist          | 
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:36.850Z", "log.level": "INFO", "message":"[controller/211] [Main.cc@123] controller (64 bit): Version 8.7.0 (Build e4e1c23721e58c) Copyright (c) 2023 Elasticsearch BV", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"ml-cpp-log-tail-thread","log.logger":"org.elasticsearch.xpack.ml.process.logging.CppLogMessageHandler","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:37.151Z", "log.level": "INFO", "message":"Security is enabled", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.security.Security","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:37.694Z", "log.level": "INFO", "message":"parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.security.authz.store.FileRolesStore","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:38.623Z", "log.level": "INFO", "message":"Profiling is enabled", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.profiler.ProfilingPlugin","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:39.603Z", "log.level": "INFO", "message":"creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=512mb}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.transport.netty4.NettyAllocator","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:39.639Z", "log.level": "INFO", "message":"using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.indices.recovery.RecoverySettings","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:39.690Z", "log.level": "INFO", "message":"using discovery type [single-node] and seed hosts providers [settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.discovery.DiscoveryModule","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:41.473Z", "log.level": "INFO", "message":"initialized", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:41.473Z", "log.level": "INFO", "message":"starting ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:41.735Z", "log.level": "INFO", "message":"persistent cache index loaded", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.searchablesnapshots.cache.full.PersistentCache","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:41.736Z", "log.level": "INFO", "message":"deprecation component started", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.xpack.deprecation.logging.DeprecationIndexingComponent","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:41.864Z", "log.level": "INFO", "message":"publish_address {172.18.0.2:9300}, bound_addresses {0.0.0.0:9300}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.transport.TransportService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:43.058Z", "log.level": "WARN", "message":"max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.BootstrapChecks","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:43.059Z", "log.level": "WARN", "message":"system call filters failed to install; check the logs and fix your configuration", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.BootstrapChecks","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:43.059Z", "log.level": "WARN", "message":"Transport SSL must be enabled if security is enabled. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.BootstrapChecks","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:43.060Z", "log.level": "INFO", "message":"this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to [175730da42c0]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.cluster.coordination.ClusterBootstrapService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:43.071Z", "log.level": "INFO", "message":"setting initial configuration to VotingConfiguration{ZSqgorS4R-GItb-xYY9R2w}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.cluster.coordination.Coordinator","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:44.063Z", "log.level": "INFO", "message":"elected-as-master ([1] nodes joined)[_FINISH_ELECTION_, {175730da42c0}{ZSqgorS4R-GItb-xYY9R2w}{4MbypjmkTyeRlyGpkFsrgQ}{175730da42c0}{172.18.0.2}{172.18.0.2:9300}{cdfhilmrstw}{8.7.0} completing election], term: 1, version: 1, delta: master node changed {previous [], current [{175730da42c0}{ZSqgorS4R-GItb-xYY9R2w}{4MbypjmkTyeRlyGpkFsrgQ}{175730da42c0}{172.18.0.2}{172.18.0.2:9300}{cdfhilmrstw}{8.7.0}]}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist          | Traceback (most recent call last):
tubearchivist          |   File "/app/manage.py", line 23, in <module>
tubearchivist          |     main()
tubearchivist          |   File "/app/manage.py", line 19, in main
tubearchivist          |     execute_from_command_line(sys.argv)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
tubearchivist          |     utility.execute()
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 382, in execute
tubearchivist          |     settings.INSTALLED_APPS
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 102, in __getattr__
tubearchivist          |     self._setup(name)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 89, in _setup
tubearchivist          |     self._wrapped = Settings(settings_module)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/django/conf/__init__.py", line 217, in __init__
tubearchivist          |     mod = importlib.import_module(self.SETTINGS_MODULE)
tubearchivist          |   File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
tubearchivist          |     return _bootstrap._gcd_import(name[level:], package, level)
tubearchivist          |   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
tubearchivist          |   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
tubearchivist          |   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
tubearchivist          |   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
tubearchivist          |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
tubearchivist          |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
tubearchivist          |   File "/app/config/settings.py", line 20, in <module>
tubearchivist          |     from home.src.ta.config import AppConfig
tubearchivist          |   File "/app/home/__init__.py", line 3, in <module>
tubearchivist          |     from .tasks import app as celery_app
tubearchivist          |   File "/app/home/tasks.py", line 12, in <module>
tubearchivist          |     from home.src.download.queue import PendingList
tubearchivist          |   File "/app/home/src/download/queue.py", line 10, in <module>
tubearchivist          |     from home.src.download.subscriptions import (
tubearchivist          |   File "/app/home/src/download/subscriptions.py", line 8, in <module>
tubearchivist          |     from home.src.download.thumbnails import ThumbManager
tubearchivist          |   File "/app/home/src/download/thumbnails.py", line 21, in <module>
tubearchivist          |     class ThumbManagerBase:
tubearchivist          |   File "/app/home/src/download/thumbnails.py", line 24, in ThumbManagerBase
tubearchivist          |     CONFIG = AppConfig().config
tubearchivist          |   File "/app/home/src/ta/config.py", line 23, in __init__
tubearchivist          |     self.config = self.get_config()
tubearchivist          |   File "/app/home/src/ta/config.py", line 28, in get_config
tubearchivist          |     config = self.get_config_redis()
tubearchivist          |   File "/app/home/src/ta/config.py", line 70, in get_config_redis
tubearchivist          |     config = RedisArchivist().get_message("config")
tubearchivist          |   File "/app/home/src/ta/ta_redis.py", line 59, in get_message
tubearchivist          |     reply = self.conn.execute_command("JSON.GET", self.NAME_SPACE + key)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/client.py", line 1255, in execute_command
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:45.262Z", "log.level": "INFO", "message":"cluster UUID set to [qwVY1DaCTC2LnZ88CorpUQ]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.CoordinationState","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:45.780Z", "log.level": "INFO", "message":"master node changed {previous [], current [{175730da42c0}{ZSqgorS4R-GItb-xYY9R2w}{4MbypjmkTyeRlyGpkFsrgQ}{175730da42c0}{172.18.0.2}{172.18.0.2:9300}{cdfhilmrstw}{8.7.0}]}, term: 1, version: 1, reason: Publication{term=1, version=1}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.service.ClusterApplierService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:45.858Z", "log.level": "INFO", "message":"starting file settings watcher ...", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.reservedstate.service.FileSettingsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:45.863Z", "log.level": "INFO", "message":"file settings service up and running [tid=55]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[file-settings-watcher]","log.logger":"org.elasticsearch.reservedstate.service.FileSettingsService","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:45.868Z", "log.level": "INFO", "message":"publish_address {172.18.0.2:9200}, bound_addresses {0.0.0.0:9200}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.http.AbstractHttpServerTransport","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:45.869Z", "log.level": "INFO", "message":"node-join: [{175730da42c0}{ZSqgorS4R-GItb-xYY9R2w}{4MbypjmkTyeRlyGpkFsrgQ}{175730da42c0}{172.18.0.2}{172.18.0.2:9300}{cdfhilmrstw}{8.7.0}] with reason [completing election]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.coordination.NodeJoinExecutor","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:45.869Z", "log.level": "INFO", "message":"started {175730da42c0}{ZSqgorS4R-GItb-xYY9R2w}{4MbypjmkTyeRlyGpkFsrgQ}{175730da42c0}{172.18.0.2}{172.18.0.2:9300}{cdfhilmrstw}{8.7.0}{xpack.installed=true, ml.machine_memory=20946878464, ml.allocated_processors=8, ml.max_jvm_size=536870912, ml.allocated_processors_double=8.0}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.159Z", "log.level": "INFO", "message":"recovered [0] indices into cluster_state", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.gateway.GatewayService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.373Z", "log.level": "INFO", "message":"adding index template [.ml-stats] for index patterns [.ml-stats-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.381Z", "log.level": "INFO", "message":"adding index template [.ml-state] for index patterns [.ml-state*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.411Z", "log.level": "INFO", "message":"adding index template [.monitoring-logstash-mb] for index patterns [.monitoring-logstash-8-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.436Z", "log.level": "INFO", "message":"adding index template [.monitoring-kibana-mb] for index patterns [.monitoring-kibana-8-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.449Z", "log.level": "INFO", "message":"adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-7-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.461Z", "log.level": "INFO", "message":"adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-7-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist          |     conn = self.connection or pool.get_connection(command_name, **options)
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 1442, in get_connection
tubearchivist          |     connection.connect()
tubearchivist          |   File "/root/.local/lib/python3.10/site-packages/redis/connection.py", line 702, in connect
tubearchivist          |     raise TimeoutError("Timeout connecting to server")
tubearchivist          | redis.exceptions.TimeoutError: Timeout connecting to server
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.479Z", "log.level": "INFO", "message":"adding template [.monitoring-alerts-7] for index patterns [.monitoring-alerts-7]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.499Z", "log.level": "INFO", "message":"adding template [.monitoring-beats] for index patterns [.monitoring-beats-7-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.520Z", "log.level": "INFO", "message":"adding template [.monitoring-es] for index patterns [.monitoring-es-7-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.531Z", "log.level": "INFO", "message":"adding index template [.ml-notifications-000002] for index patterns [.ml-notifications-000002]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.556Z", "log.level": "INFO", "message":"adding index template [.ml-anomalies-] for index patterns [.ml-anomalies-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.577Z", "log.level": "INFO", "message":"adding index template [.monitoring-ent-search-mb] for index patterns [.monitoring-ent-search-8-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.622Z", "log.level": "INFO", "message":"adding index template [.monitoring-es-mb] for index patterns [.monitoring-es-8-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.652Z", "log.level": "INFO", "message":"adding index template [.monitoring-beats-mb] for index patterns [.monitoring-beats-8-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.660Z", "log.level": "INFO", "message":"adding component template [logs-mappings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.665Z", "log.level": "INFO", "message":"adding component template [logs-settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.669Z", "log.level": "INFO", "message":"adding component template [metrics-settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.673Z", "log.level": "INFO", "message":"adding component template [synthetics-settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.685Z", "log.level": "INFO", "message":"adding component template [data-streams-mappings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.691Z", "log.level": "INFO", "message":"adding component template [metrics-mappings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.696Z", "log.level": "INFO", "message":"adding component template [synthetics-mappings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.712Z", "log.level": "INFO", "message":"adding index template [.watch-history-16] for index patterns [.watcher-history-16*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.722Z", "log.level": "INFO", "message":"adding index template [ilm-history] for index patterns [ilm-history-5*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.728Z", "log.level": "INFO", "message":"adding component template [.deprecation-indexing-mappings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.732Z", "log.level": "INFO", "message":"adding component template [.deprecation-indexing-settings]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.740Z", "log.level": "INFO", "message":"adding index template [.slm-history] for index patterns [.slm-history-5*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.747Z", "log.level": "INFO", "message":"adding index template [.fleet-file-data] for index patterns [.fleet-file-data-*-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:46.754Z", "log.level": "INFO", "message":"adding index template [.fleet-files] for index patterns [.fleet-files-*-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:47.422Z", "log.level": "INFO", "message":"adding index template [metrics] for index patterns [metrics-*-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:47.432Z", "log.level": "INFO", "message":"adding index template [synthetics] for index patterns [synthetics-*-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:47.440Z", "log.level": "INFO", "message":"adding index template [.deprecation-indexing-template] for index patterns [.logs-deprecation.*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:47.449Z", "log.level": "INFO", "message":"adding index template [logs] for index patterns [logs-*-*]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.metadata.MetadataIndexTemplateService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:47.852Z", "log.level": "INFO", "message":"adding index lifecycle policy [.monitoring-8-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-redis    | 9:C 01 Jun 2023 05:27:18.440 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
tubearchivist-redis    | 9:C 01 Jun 2023 05:27:18.440 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=9, just started
tubearchivist-redis    | 9:C 01 Jun 2023 05:27:18.440 # Configuration loaded
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.441 * monotonic clock: POSIX clock_gettime
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.441 * Running mode=standalone, port=6379.
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.441 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.441 # Server initialized
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.441 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * <search> Redis version found by RedisSearch : 6.2.12 - oss
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * <search> RediSearch version 2.6.9 (Git=HEAD-b205a1d5)
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * <search> Low level api version 1 initialized successfully
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * <search> concurrent writes: OFF, gc: ON, prefix min length: 2, prefix max expansions: 200, query timeout (ms): 500, timeout policy: return, cursor read size: 1000, cursor max idle (ms): 300000, max doctable size: 1000000, max number of search results:  10000, search pool size: 20, index pool size: 8, 
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * <search> Initialized thread pool!
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * <search> Enabled diskless replication
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * <search> Enabled role change notification
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.484 * Module 'search' loaded from /opt/redis-stack/lib/redisearch.so
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.711 * <graph> Enabled role change notification
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.711 * <graph> Starting up RedisGraph version 2.10.10.
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.713 * <graph> Thread pool created, using 8 threads.
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.713 * <graph> Maximum number of OpenMP threads set to 8
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.713 * Module 'graph' loaded from /opt/redis-stack/lib/redisgraph.so
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * <timeseries> RedisTimeSeries version 10810, git_sha=78d8254a4ed499e4e24566ab76c9275130c8786b
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * <timeseries> Redis version found by RedisTimeSeries : 6.2.12 - oss
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * <timeseries> loaded default CHUNK_SIZE_BYTES policy: 4096
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * <timeseries> loaded server DUPLICATE_POLICY: block
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * <timeseries> Setting default series ENCODING to: compressed
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * <timeseries> Detected redis oss
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * <timeseries> Enabled diskless replication
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.716 * Module 'timeseries' loaded from /opt/redis-stack/lib/redistimeseries.so
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> version: 20407 git sha: a4412b3 branch: HEAD
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> Exported RedisJSON_V1 API
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:48.280Z", "log.level": "INFO", "message":"adding index lifecycle policy [ml-size-based-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:48.652Z", "log.level": "INFO", "message":"adding index lifecycle policy [logs]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:49.394Z", "log.level": "INFO", "message":"adding index lifecycle policy [synthetics]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:49.760Z", "log.level": "INFO", "message":"adding index lifecycle policy [metrics]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:50.244Z", "log.level": "INFO", "message":"adding index lifecycle policy [30-days-default]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:50.617Z", "log.level": "INFO", "message":"adding index lifecycle policy [7-days-default]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:51.245Z", "log.level": "INFO", "message":"adding index lifecycle policy [365-days-default]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:51.653Z", "log.level": "INFO", "message":"adding index lifecycle policy [90-days-default]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:52.011Z", "log.level": "INFO", "message":"adding index lifecycle policy [180-days-default]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:52.410Z", "log.level": "INFO", "message":"adding index lifecycle policy [watch-history-ilm-policy-16]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:52.860Z", "log.level": "INFO", "message":"adding index lifecycle policy [ilm-history-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:53.282Z", "log.level": "INFO", "message":"adding index lifecycle policy [slm-history-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:53.625Z", "log.level": "INFO", "message":"adding index lifecycle policy [.deprecation-indexing-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:53.992Z", "log.level": "INFO", "message":"adding index lifecycle policy [.fleet-files-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:55.092Z", "log.level": "INFO", "message":"adding index lifecycle policy [.fleet-file-data-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:55.433Z", "log.level": "INFO", "message":"adding index lifecycle policy [.fleet-actions-results-ilm-policy]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.action.TransportPutLifecycleAction","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:56.815Z", "log.level": "INFO", "message":"Node [{175730da42c0}{ZSqgorS4R-GItb-xYY9R2w}] is selected as the current health node.", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][management][T#1]","log.logger":"org.elasticsearch.health.node.selection.HealthNodeTaskExecutor","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:57.694Z", "log.level": "INFO", "message":"license [730ecfd3-9f9d-4b1a-9c05-0e8c6539f246] mode [basic] - valid", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.license.LicenseService","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-es       | {"@timestamp":"2023-06-01T05:27:57.696Z", "log.level": "INFO", "message":"license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[175730da42c0][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.security.authc.Realms","elasticsearch.cluster.uuid":"qwVY1DaCTC2LnZ88CorpUQ","elasticsearch.node.id":"ZSqgorS4R-GItb-xYY9R2w","elasticsearch.node.name":"175730da42c0","elasticsearch.cluster.name":"docker-cluster"}
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> Exported RedisJSON_V2 API
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> Exported RedisJSON_V3 API
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> Enabled diskless replication
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> Created new data type 'ReJSON-RL'
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * Module 'ReJSON' loaded from /opt/redis-stack/lib/rejson.so
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <search> Acquired RedisJSON_V3 API
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.750 * <graph> Acquired RedisJSON_V1 API
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.752 * <bf> RedisBloom version 2.4.5 (Git=unknown)
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.752 * Module 'bf' loaded from /opt/redis-stack/lib/redisbloom.so
tubearchivist-redis    | 9:M 01 Jun 2023 05:27:18.752 * Ready to accept connections
lamusmaser commented 1 year ago

Can I get just the redis logs from that? See earlier comment for details.

I'm suspecting that redis is restarting on us and that's what is killing the connection.

Th3M1CR0WAV3 commented 1 year ago

Sorry it's still not working. I'm trying to think what else could be contributing to this error - qould my PiHole have any impact?

Th3M1CR0WAV3 commented 1 year ago

tubearchivist-redis(1).csv

9:C 01 Jun 2023 05:27:18.440 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
 9:C 01 Jun 2023 05:27:18.440 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=9, just started
 9:C 01 Jun 2023 05:27:18.440 # Configuration loaded
 9:M 01 Jun 2023 05:27:18.441 * monotonic clock: POSIX clock_gettime
 9:M 01 Jun 2023 05:27:18.441 * Running mode=standalone, port=6379.
 9:M 01 Jun 2023 05:27:18.441 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
 9:M 01 Jun 2023 05:27:18.441 # Server initialized
 9:M 01 Jun 2023 05:27:18.441 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
 9:M 01 Jun 2023 05:27:18.484 * <search> Redis version found by RedisSearch : 6.2.12 - oss
 9:M 01 Jun 2023 05:27:18.484 * <search> RediSearch version 2.6.9 (Git=HEAD-b205a1d5)
 9:M 01 Jun 2023 05:27:18.484 * <search> Low level api version 1 initialized successfully
 9:M 01 Jun 2023 05:27:18.484 * <search> concurrent writes: OFF, gc: ON, prefix min length: 2, prefix max expansions: 200, query timeout (ms): 500, timeout policy: return, cursor read size: 1000, cursor max idle (ms): 300000, max doctable size: 1000000, max number of search results:  10000, search pool size: 20, index pool size: 8, 
 9:M 01 Jun 2023 05:27:18.484 * <search> Initialized thread pool!
 9:M 01 Jun 2023 05:27:18.484 * <search> Enabled diskless replication
 9:M 01 Jun 2023 05:27:18.484 * <search> Enabled role change notification
 9:M 01 Jun 2023 05:27:18.484 * Module 'search' loaded from /opt/redis-stack/lib/redisearch.so
 9:M 01 Jun 2023 05:27:18.711 * <graph> Enabled role change notification
 9:M 01 Jun 2023 05:27:18.711 * <graph> Starting up RedisGraph version 2.10.10.
 9:M 01 Jun 2023 05:27:18.713 * <graph> Thread pool created, using 8 threads.
 9:M 01 Jun 2023 05:27:18.713 * <graph> Maximum number of OpenMP threads set to 8
 9:M 01 Jun 2023 05:27:18.713 * Module 'graph' loaded from /opt/redis-stack/lib/redisgraph.so
 9:M 01 Jun 2023 05:27:18.716 * <timeseries> RedisTimeSeries version 10810, git_sha=78d8254a4ed499e4e24566ab76c9275130c8786b
 9:M 01 Jun 2023 05:27:18.716 * <timeseries> Redis version found by RedisTimeSeries : 6.2.12 - oss
 9:M 01 Jun 2023 05:27:18.716 * <timeseries> loaded default CHUNK_SIZE_BYTES policy: 4096
 9:M 01 Jun 2023 05:27:18.716 * <timeseries> loaded server DUPLICATE_POLICY: block
 9:M 01 Jun 2023 05:27:18.716 * <timeseries> Setting default series ENCODING to: compressed
 9:M 01 Jun 2023 05:27:18.716 * <timeseries> Detected redis oss
 9:M 01 Jun 2023 05:27:18.716 * <timeseries> Enabled diskless replication
 9:M 01 Jun 2023 05:27:18.716 * Module 'timeseries' loaded from /opt/redis-stack/lib/redistimeseries.so
 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> version: 20407 git sha: a4412b3 branch: HEAD
 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> Exported RedisJSON_V1 API
 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> Exported RedisJSON_V2 API
 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> Exported RedisJSON_V3 API
 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> Enabled diskless replication
 9:M 01 Jun 2023 05:27:18.750 * <ReJSON> Created new data type 'ReJSON-RL'
 9:M 01 Jun 2023 05:27:18.750 * Module 'ReJSON' loaded from /opt/redis-stack/lib/rejson.so
 9:M 01 Jun 2023 05:27:18.750 * <search> Acquired RedisJSON_V3 API
 9:M 01 Jun 2023 05:27:18.750 * <graph> Acquired RedisJSON_V1 API
 9:M 01 Jun 2023 05:27:18.752 * <bf> RedisBloom version 2.4.5 (Git=unknown)
 9:M 01 Jun 2023 05:27:18.752 * Module 'bf' loaded from /opt/redis-stack/lib/redisbloom.so
 9:M 01 Jun 2023 05:27:18.752 * Ready to accept connections
lamusmaser commented 1 year ago

Sorry it's still not working. I'm trying to think what else could be contributing to this error - qould my PiHole have any impact?

I wouldn't expect it since it should be going through its own internal routing and DNS.

I'm not seeing a restart within the logs, so that isn't the problem.

Let me think on it and look into a few other things to see if there is an unrelated issue causing this.

One thing I'd like to confirm is intra-communication. Like earlier, can you do the same test from the ES container to the redis container? I'd expect it to work this time.

Th3M1CR0WAV3 commented 1 year ago 
elasticsearch@175730da42c0:~$ (printf "PING\r\nQUIT\r\n";) | curl -v telnet://tubearchivist-redis:6379                                                                              
*   Trying 172.18.0.3:6379...                                                                                                                                                       
* TCP_NODELAY set
* connect to 172.18.0.3 port 6379 failed: Connection timed out                                                                                                          
* Failed to connect to tubearchivist-redis port 6379: Connection timed out
Th3M1CR0WAV3 commented 1 year ago

It's the firewall blocking the connection. After fixing the firewall the container runs, however when going to the TubeArchivist url I now get a Bad Request (400) message.

lamusmaser commented 1 year ago

That's where we were expecting to get to. I'm very interested to know what firewall was setup to cause it to block.

For URL responses of Bad Request (400), that usually means that we need to update the TA_HOST line. Basically, if you are attempting to access it as http://ip-address:port, then we need to add the ip-address portion to the TA_HOST variable's line. Same principal applies for any domain, such as http://domain.any, if you have a reverse proxy. If you are using https for your proxy, please specify it before the domain, such as https://domain.any, in the list. They are space delimited, so the line could be updated to the following:

    - "TA_HOST=synology.local ip-address domain.any"
derekslenk commented 1 year ago

I can reproduce this on a docker setup on an Ubuntu VM. Was working perfectly fine until a recent shutdown and reboot.

However, I see redis crash, hence why archivist-redis can't be found in my case

archivist-redis  | 9:M 08 Jun 2023 03:40:56.156 * <graph> Enabled role change notification
archivist-redis  | 9:M 08 Jun 2023 03:40:56.156 * <graph> Starting up RedisGraph version 2.10.10.
archivist-redis  | 9:M 08 Jun 2023 03:40:56.168 * <graph> Thread pool created, using 8 threads.
archivist-redis  | 9:M 08 Jun 2023 03:40:56.168 * <graph> Maximum number of OpenMP threads set to 8
archivist-redis  | 9:M 08 Jun 2023 03:40:56.169 * Module 'graph' loaded from /opt/redis-stack/lib/redisgraph.so
archivist-redis  | 9:M 08 Jun 2023 03:40:56.170 * <timeseries> RedisTimeSeries version 10810, git_sha=78d8254a4ed499e4e24566ab76c9275130c8786b
archivist-redis  | 9:M 08 Jun 2023 03:40:56.170 * <timeseries> Redis version found by RedisTimeSeries : 6.2.12 - oss
archivist-redis  | 9:M 08 Jun 2023 03:40:56.170 * <timeseries> loaded default CHUNK_SIZE_BYTES policy: 4096
archivist-redis  | 9:M 08 Jun 2023 03:40:56.170 * <timeseries> loaded server DUPLICATE_POLICY: block
archivist-redis  | 9:M 08 Jun 2023 03:40:56.170 * <timeseries> Setting default series ENCODING to: compressed
archivist-redis  | 9:M 08 Jun 2023 03:40:56.171 * <timeseries> Detected redis oss
archivist-redis  | 9:M 08 Jun 2023 03:40:56.172 * <timeseries> Enabled diskless replication
archivist-redis  | 9:M 08 Jun 2023 03:40:56.172 * Module 'timeseries' loaded from /opt/redis-stack/lib/redistimeseries.so
archivist-redis  | 9:M 08 Jun 2023 03:40:56.173 * <ReJSON> version: 20407 git sha: a4412b3 branch: HEAD
archivist-redis  | 9:M 08 Jun 2023 03:40:56.173 * <ReJSON> Exported RedisJSON_V1 API
archivist-redis  | 9:M 08 Jun 2023 03:40:56.173 * <ReJSON> Exported RedisJSON_V2 API
archivist-redis  | 9:M 08 Jun 2023 03:40:56.173 * <ReJSON> Exported RedisJSON_V3 API
archivist-redis  | 9:M 08 Jun 2023 03:40:56.173 * <ReJSON> Enabled diskless replication
archivist-redis  | 9:M 08 Jun 2023 03:40:56.173 * <ReJSON> Created new data type 'ReJSON-RL'
archivist-redis  | 9:M 08 Jun 2023 03:40:56.174 * Module 'ReJSON' loaded from /opt/redis-stack/lib/rejson.so
archivist-redis  | 9:M 08 Jun 2023 03:40:56.174 * <search> Acquired RedisJSON_V3 API
archivist-redis  | 9:M 08 Jun 2023 03:40:56.174 * <graph> Acquired RedisJSON_V1 API
archivist-redis  | 9:M 08 Jun 2023 03:40:56.175 * <bf> RedisBloom version 2.4.5 (Git=unknown)
archivist-redis  | 9:M 08 Jun 2023 03:40:56.176 * Module 'bf' loaded from /opt/redis-stack/lib/redisbloom.so
archivist-redis  | 9:M 08 Jun 2023 03:40:56.176 * <search> Loading event starts
archivist-redis  | 9:M 08 Jun 2023 03:40:56.177 # Can't handle RDB format version 10
archivist-redis  | 9:M 08 Jun 2023 03:40:56.177 # Fatal error loading the DB: Invalid argument. Exiting.
lamusmaser commented 1 year ago

@derekslenk please open your own issue for this. It is unrelated.