[DocDB] Spooky warnings during master instance initialization

[dmagda]

Jira Link: DB-7753

Description

The following warnings appear in the master's instance logs during the bootstrap:

Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0829 14:07:48.596866 13141 catalog_manager.cc:1566] Failed to get current config: Illegal state (yb/master/catalog_manager.cc:11849): Node d9730b1ebf194f87809f89a761106a09 peer not initialized.
W0829 14:07:48.605463 13147 catalog_manager.cc:1566] Failed to get current config: Illegal state (yb/master/catalog_manager.cc:11849): Node d9730b1ebf194f87809f89a761106a09 peer not initialized.
W0829 14:07:48.632774 13146 log_util.cc:243] Could not read footer for segment: /home/dmagda/yb-dir/yb-data/master/wals/table-sys.catalog.uuid/tablet-00000000000000000000000000000000/wal-000000002: Not found (yb/consensus/log_util.cc:583): Footer not found. Footer magic doesn't match
W0829 14:07:48.632809 13146 log_util.cc:243] Could not read footer for segment: /home/dmagda/yb-dir/yb-data/master/wals/table-sys.catalog.uuid/tablet-00000000000000000000000000000000/wal-000000001: Not found (yb/consensus/log_util.cc:583): Footer not found. Footer magic doesn't match
W0829 14:07:48.632820 13146 log_reader.cc:226] T 00000000000000000000000000000000 P d9730b1ebf194f87809f89a761106a09: Log segment /home/dmagda/yb-dir/yb-data/master/wals/table-sys.catalog.uuid/tablet-00000000000000000000000000000000/wal-000000001 was likely left in-progressafter a previous crash. Will try to rebuild footer by scanning data.
W0829 14:07:48.640743 13146 log_reader.cc:221] T 00000000000000000000000000000000 P d9730b1ebf194f87809f89a761106a09: Log segment /home/dmagda/yb-dir/yb-data/master/wals/table-sys.catalog.uuid/tablet-00000000000000000000000000000000/wal-000000002 was likely left in-progressafter a previous crash. Will try to reuse this segment as writable active segment
W0829 14:07:48.672413 13151 catalog_manager_bg_tasks.cc:164] Catalog manager background task thread going to sleep: Service unavailable (yb/master/scoped_leader_shared_lock.cc:92): Catalog manager is not initialized. State: 1
W0829 14:10:30.232249 13159 catalog_manager.cc:8177] T 00000000000000000000000000000000 P d9730b1ebf194f87809f89a761106a09: Invalid tablet report from 70c4f562352648bd802e00f561ed99d4: Received an incremental tablet report when a full one was neede

While the warnings look confusing, the cluster is fully functioning.

Environment::

PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian

Steps to reproduce:

1. Download:

   wget https://downloads.yugabyte.com/releases/2.18.2.1/yugabyte-2.18.2.1-b1-linux-x86_64.tar.gz
   tar xvfz yugabyte-2.18.2.1-b1-linux-x86_64.tar.gz && cd yugabyte-2.18.2.1/
   ./bin/post_install.sh

2. Start a master and tserver instance:

   ./bin/yb-master \
   --master_addresses 127.0.0.1:7100 \
   --rpc_bind_addresses 127.0.0.1 \
   --fs_data_dirs "/home/dmagda/yb-dir" &
   
   ./bin/yb-tserver \
   --tserver_master_addrs 127.0.0.1:7100 \
   --rpc_bind_addresses 127.0.0.1 \
   --enable_ysql \
   --fs_data_dirs "/home/dmagda/yb-dir" & 

The master's WARNING logs will include the spooky warnings, while at the same time you can easily connect to the cluster:

./ysqlsh
ysqlsh (11.2-YB-2.18.2.1-b0)
Type "help" for help.

yugabyte=# select * from yb_servers();
   host    | port | num_connections | node_type | cloud  |   region    | zone  | public_ip |
   uuid
-----------+------+-----------------+-----------+--------+-------------+-------+-----------+------------
----------------------
 127.0.0.1 | 5433 |               0 | primary   | cloud1 | datacenter1 | rack1 |           | 70c4f562352648bd802e00f561ed99d4
(1 row) 

Warning: Please confirm that this issue does not contain any sensitive information

• [X] I confirm this issue does not contain any sensitive information.

[mohatagarvit]

Hi @dmagda,

Hope you are doing well!

I am Garvit Mohata, a MS CS student at UT Austin. My teammate (Kulin Shah) and I are taking the graduate Distributed Systems course and as a part of the course project, we are exploring the opportunity of contributing to the yugabyte open source repository.

This issue seems closely related to the content of the course so we were wondering if we can contribute to solve this issue.

Can you provide a bit more specific expectations and details to solve this issue and pointers if possible? Thank you!

Regards, Garvit

[ddorian]

Hi @mohatagarvit

Can you join on slack on the #contributors channel and ask there?