search

scylladb / scylladb

NoSQL data store using the seastar framework, compatible with Apache Cassandra
http://scylladb.com
GNU Affero General Public License v3.0
12.94k stars 1.24k forks source link

tablet load balancer log spam #18004

Open avikivity opened 3 months ago

avikivity commented 3 months ago 
INFO  2024-03-25 15:55:24,626 [shard 0:strm] load_balancer - Node cc63bcd3-ff6d-4de1-ae85-5a8225ab099c: rack=rack1 avg_load=0, tablets=0, shards=2, state=normal
INFO  2024-03-25 15:55:24,626 [shard 0:strm] load_balancer - Prepared 0 migrations in DC datacenter1
INFO  2024-03-25 15:55:24,626 [shard 0:strm] load_balancer - Prepared 0 migration plans, out of which there were 0 tablet migration(s) and 0 resize decision(s)
INFO  2024-03-25 15:56:24,626 [shard 0:strm] raft_topology - raft topology: Refreshing table load stats for DC datacenter1 that has 1 endpoints
INFO  2024-03-25 15:56:24,627 [shard 0:strm] load_balancer - Examining DC datacenter1
INFO  2024-03-25 15:56:24,627 [shard 0:strm] load_balancer - Node cc63bcd3-ff6d-4de1-ae85-5a8225ab099c: rack=rack1 avg_load=0, tablets=0, shards=2, state=normal
INFO  2024-03-25 15:56:24,627 [shard 0:strm] load_balancer - Prepared 0 migrations in DC datacenter1
INFO  2024-03-25 15:56:24,627 [shard 0:strm] load_balancer - Prepared 0 migration plans, out of which there were 0 tablet migration(s) and 0 resize decision(s)
INFO  2024-03-25 15:57:24,627 [shard 0:strm] raft_topology - raft topology: Refreshing table load stats for DC datacenter1 that has 1 endpoints
INFO  2024-03-25 15:57:24,627 [shard 0:strm] load_balancer - Examining DC datacenter1
INFO  2024-03-25 15:57:24,627 [shard 0:strm] load_balancer - Node cc63bcd3-ff6d-4de1-ae85-5a8225ab099c: rack=rack1 avg_load=0, tablets=0, shards=2, state=normal
INFO  2024-03-25 15:57:24,627 [shard 0:strm] load_balancer - Prepared 0 migrations in DC datacenter1
INFO  2024-03-25 15:57:24,627 [shard 0:strm] load_balancer - Prepared 0 migration plans, out of which there were 0 tablet migration(s) and 0 resize decision(s)
INFO  2024-03-25 15:58:24,627 [shard 0:strm] raft_topology - raft topology: Refreshing table load stats for DC datacenter1 that has 1 endpoints
INFO  2024-03-25 15:58:24,628 [shard 0:strm] load_balancer - Examining DC datacenter1
INFO  2024-03-25 15:58:24,628 [shard 0:strm] load_balancer - Node cc63bcd3-ff6d-4de1-ae85-5a8225ab099c: rack=rack1 avg_load=0, tablets=0, shards=2, state=normal
INFO  2024-03-25 15:58:24,628 [shard 0:strm] load_balancer - Prepared 0 migrations in DC datacenter1
INFO  2024-03-25 15:58:24,628 [shard 0:strm] load_balancer - Prepared 0 migration plans, out of which there were 0 tablet migration(s) and 0 resize decision(s)
INFO  2024-03-25 15:59:24,628 [shard 0:strm] raft_topology - raft topology: Refreshing table load stats for DC datacenter1 that has 1 endpoints
INFO  2024-03-25 15:59:24,628 [shard 0:strm] load_balancer - Examining DC datacenter1
INFO  2024-03-25 15:59:24,628 [shard 0:strm] load_balancer - Node cc63bcd3-ff6d-4de1-ae85-5a8225ab099c: rack=rack1 avg_load=0, tablets=0, shards=2, state=normal
INFO  2024-03-25 15:59:24,628 [shard 0:strm] load_balancer - Prepared 0 migrations in DC datacenter1
INFO  2024-03-25 15:59:24,628 [shard 0:strm] load_balancer - Prepared 0 migration plans, out of which there were 0 tablet migration(s) and 0 resize decision(s)
INFO  2024-03-25 16:00:24,628 [shard 0:strm] raft_topology - raft topology: Refreshing table load stats for DC datacenter1 that has 1 endpoints
INFO  2024-03-25 16:00:24,629 [shard 0:strm] load_balancer - Examining DC datacenter1
INFO  2024-03-25 16:00:24,629 [shard 0:strm] load_balancer - Node cc63bcd3-ff6d-4de1-ae85-5a8225ab099c: rack=rack1 avg_load=0, tablets=0, shards=2, state=normal
INFO  2024-03-25 16:00:24,629 [shard 0:strm] load_balancer - Prepared 0 migrations in DC datacenter1
INFO  2024-03-25 16:00:24,629 [shard 0:strm] load_balancer - Prepared 0 migration plans, out of which there were 0 tablet migration(s) and 0 resize decision(s)
INFO  2024-03-25 16:01:24,628 [shard 0:strm] raft_topology - raft topology: Refreshing table load stats for DC datacenter1 that has 1 endpoints
INFO  2024-03-25 16:01:24,629 [shard 0:strm] load_balancer - Examining DC datacenter1
INFO  2024-03-25 16:01:24,629 [shard 0:strm] load_balancer - Node cc63bcd3-ff6d-4de1-ae85-5a8225ab099c: rack=rack1 avg_load=0, tablets=0, shards=2, state=normal
INFO  2024-03-25 16:01:24,629 [shard 0:strm] load_balancer - Prepared 0 migrations in DC datacenter1
INFO  2024-03-25 16:01:24,629 [shard 0:strm] load_balancer - Prepared 0 migration plans, out of which there were 0 tablet migration(s) and 0 resize decision(s)
INFO  2024-03-25 16:02:24,629 [shard 0:strm] raft_topology - raft topology: Refreshing table load stats for DC datacenter1 that has 1 endpoints
INFO  2024-03-25 16:02:24,629 [shard 0:strm] load_balancer - Examining DC datacenter1
INFO  2024-03-25 16:02:24,629 [shard 0:strm] load_balancer - Node cc63bcd3-ff6d-4de1-ae85-5a8225ab099c: rack=rack1 avg_load=0, tablets=0, shards=2, state=normal
INFO  2024-03-25 16:02:24,629 [shard 0:strm] load_balancer - Prepared 0 migrations in DC datacenter1
INFO  2024-03-25 16:02:24,629 [shard 0:strm] load_balancer - Prepared 0 migration plans, out of which there were 0 tablet migration(s) and 0 resize decision(s)

There's no information in these logs.

tgrabiec commented 3 months ago

What's the scenario? The re-examination shouldn't happen without a triggering event.

avikivity commented 3 months ago

Idle single-node cluster.

avikivity commented 3 months ago

Probably empty, too. I was looking at the auth schema.

michoecho commented 3 months ago

2209c7440cbac66d7a27f8f8be050cb22376598a added the code which wakes up the load balancer every 60s.

raphaelsc commented 2 months ago

2209c74 added the code which wakes up the load balancer every 60s.

maybe we could use an exponential backoff retry which is reset to 60s when work is found