yb-master cost far more cpu than postgres

[bnuzhouwei]

Jira Link: DB-4449

I use docker-compose built a cluster to test. When i call top to see what cost cpus, I find the yb-master cost cpu rate always larger than 10%, while other databases such as tidb, crdb cost much lower.

It is strange that the cluster has very low load, but cost much cpu. Why?

CONTAINER ID   NAME               CPU %     MEM USAGE / LIMIT    MEM %     NET I/O           BLOCK I/O         PIDS
479ed6e97ac4   ybdb_web_1         0.06%     162.8MiB / 3.83GiB   4.15%     519kB / 219kB     143MB / 1.02MB    33
1b4e6f2bc2ff   ybdb_tserver_1     13.29%    1.377GiB / 3.83GiB   35.94%    42.6MB / 99.1MB   526MB / 1.47GB    152
8d8771a9807b   ybdb_redis_1       0.10%     3.465MiB / 3.83GiB   0.09%     1.85kB / 0B       5.24MB / 0B       5
de95345ee3b3   ybdb_master_1      3.12%     105.9MiB / 3.83GiB   2.70%     97.6MB / 42.5MB   76.9MB / 83MB     47
79cc54ad3f17   ybdb_adminer_1     0.01%     4.895MiB / 3.83GiB   0.12%     1.7kB / 0B        7.92MB / 0B       1
412206df19c0   mysql2_web_1       0.08%     85.31MiB / 3.83GiB   2.18%     57.4kB / 30.7kB   42MB / 75.1MB     36
9304fa7c61f3   mysql2_adminer_1   0.00%     3.859MiB / 3.83GiB   0.10%     1.2kB / 0B        8.19kB / 0B       1
474cefc4be8d   mysql2_redis_1     0.10%     3.121MiB / 3.83GiB   0.08%     21.5kB / 12kB     97.2MB / 41kB     5
507d6a9e2517   mysql2_db_1        0.38%     87.41MiB / 3.83GiB   2.23%     12.2kB / 44.5kB   204MB / 131MB     41
bd32737f8028   fbdb_web_1         0.10%     94.07MiB / 3.83GiB   2.40%     75kB / 32.7kB     129MB / 72.5MB    36
c9c02327cd28   fbdb_db_1          0.00%     16.71MiB / 3.83GiB   0.43%     33.9kB / 73.8kB   91.3MB / 21.1MB   9
fb6bb73cd758   fbdb_adminer_1     0.00%     8.852MiB / 3.83GiB   0.23%     1.12kB / 0B       34.4MB / 0B       1
dbb6e559b455   fbdb_redis_1       0.09%     6.914MiB / 3.83GiB   0.18%     1.56kB / 0B       25.7MB / 0B       5
3b7ceac8b3d6   crdb_web_1         0.14%     85.66MiB / 3.83GiB   2.18%     44.3kB / 10.1kB   48.5MB / 61.8MB   33
cf7315d1fa71   crdb_roach2_1      3.57%     275.2MiB / 3.83GiB   7.02%     53.7MB / 48.4MB   577MB / 1.18GB    20
8f2d14309ade   crdb_redis_1       0.11%     2.723MiB / 3.83GiB   0.07%     1.37kB / 0B       49.2kB / 0B       5
a016f9a42d5a   crdb_roach1_1      3.30%     355.3MiB / 3.83GiB   9.06%     37.9MB / 40.5MB   745MB / 1.19GB    19
121aac62784b   crdb_roach3_1      3.80%     387.9MiB / 3.83GiB   9.89%     37.6MB / 41.2MB   685MB / 1.15GB    18
906986f0ef49   crdb_adminer_1     0.00%     3.863MiB / 3.83GiB   0.10%     1.37kB / 0B       8.19kB / 0B       1

version: '3.5'
services:
  master:
    image: yugabytedb/yugabyte:2.14.5.0-b18
    volumes:
      - "./master:/mnt/master"
    command: ["/home/yugabyte/bin/yb-master", "--fs_data_dirs=/mnt/master", "--master_addresses=master:7100", "--rpc_bind_addresses=master:7100", "--replication_factor=1"]
    ports:
      - "7000:7000"
    environment:
      SERVICE_7000_NAME: yb-master
    restart: always
    networks:
      app_net:
        ipv4_address: 10.10.2.2
  tserver:
    image: yugabytedb/yugabyte:2.14.5.0-b18
    volumes:
      - "./tserver:/mnt/tserver"
    command: ["/home/yugabyte/bin/yb-tserver", "--fs_data_dirs=/mnt/tserver", "--enable_ysql", "--rpc_bind_addresses=tserver:9100", "--tserver_master_addrs=master:7100"]
    ports:
      - "9042:9042"
      - "5433:5433"
      - "9000:9000"
    environment:
      SERVICE_5433_NAME: ysql
      SERVICE_9042_NAME: ycql
      SERVICE_6379_NAME: yedis
      SERVICE_9000_NAME: yb-tserver
    depends_on:
      - master
    restart: always
    networks:
      app_net:
        ipv4_address: 10.10.2.4
networks:
  app_net:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 10.10.2.1/24

It seems everythings is OK, but cost much CPU..

PostgreSQL 11.2-YB-2.14.5.0-b0 on x86_64-pc-linux-gnu, compiled by clang version 12.0.1 (https://github.com/yugabyte/llvm-project.git bdb147e675d8c87cee72cc1f87c4b82855977d94), 64-bit

[ddorian]

Hi @bnuzhouwei

What is your server hardware?

What's the replication factor?

How many tablets do you have? (posting a screenshot of http://master-ip:7000/tablet-servers

[ddorian]

Also note that it's yb-tserver and not yb-master.

[bnuzhouwei]

Server hadrware is

Intel i5 1235u with 32GB RAM and 2T NVME SSD

[ddorian]

It's probably because of the big number of tablets (User Tablet-Peers column).

Can you try dropping the tables and see if the load is lower?

Think having 620 tablets instead of 3100.

[bnuzhouwei]

I have only about 80 tables, and each table has no more than 2000 rows, why it create so many tablets. drop tables will lose data.... Can it recreate much less tablets when keep the data?

[ddorian]

I have only about 80 tables

You probably also have secondary indexes on them.

why it create so many tablets

It's using this flag https://docs.yugabyte.com/preview/reference/configuration/yb-tserver/#ysql-num-shards-per-tserver

Can it recreate much less tablets when keep the data?

It can't for now merge tablets. You can shut down the db, create a new one using a new data-directory and recreate the tables with less tablets (by changing the option above) and re-chcking CPU usage.

3112 tablets ÷ 8 (number of tablets per table/index) = 389 (tables + indexes in total that you probably have)

[ddorian]

Or you can use a colocated db https://docs.yugabyte.com/preview/architecture/docdb-sharding/colocated-tables/