[VOQ][saidump] Enhance saidump with new option -r to parser the JSON file and displays/format the right output

JunhongMao commented 1 year ago

Why I did it

Fix issue: https://github.com/sonic-net/sonic-buildimage/issues/13561 The existing saidump use https://github.com/sonic-net/sonic-swss-common/blob/master/common/table_dump.lua script which loops the ASIC_DB more than 5 seconds and blocks other processes access.

This solution uses the redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table. Related PRs: https://github.com/sonic-net/sonic-utilities/pull/2972 https://github.com/sonic-net/sonic-buildimage/pull/16466

MSFT ADO: 25892357

How I did it

To use the Redis-db SAVE option to save the snapshot of DB each time and recover later, instead of looping through each entry in the table and saving it.

    (1) Updated platform/broadcom/docker-syncd-brcm-dnx/Dockerfile.j2, install Python library rdbtools into the syncd containter.
    (2) Updated sonic-buildimage/src/sonic-sairedis/saidump/saidump.cpp, add a new option -r, which updates the rdbtools's output-JSON files' format.
    (3) Updated sonic-buildimage/build_debian.sh, to add a new script file: files/scripts/saidump.sh into the host. This shell file does the below steps:
      For each ASIC0, such as ASIC0,

      1. Save the Redis data.
      sudo sonic-db-cli -n asic$1 SAVE > /dev/null

      2. Move dump files to /var/run/redisX/
      docker exec database$1 sh -c "mv /var/lib/redis/dump.rdb /var/run/redis$1/"

      3. Run rdb command to convert the dump files into JSON files
      docker exec syncd$1 sh -c "rdb --command json /var/run/redis$1/dump.rdb | tee /var/run/redis$1/dump.json > /dev/null"

      4. Run saidump -r to update the JSON files' format as same as the saidump before. Then we can get the saidump result in standard output.
      docker exec syncd$1 sh -c "saidump -r /var/run/redis$1/dump.json -m 100"

      5. clear
      sudo rm -f /var/run/redis$1/dump.rdb
      sudo rm -f /var/run/redis$1/dump.json

    (4) Update sonic-buildimage/src/sonic-utilities/scripts/generate_dump, to check the asic db size and if it is larger than xxx entries, then do with REDIS SAVE, otherwise, to do with old method: looping through each entry of Redis DB.

How to verify it

1) On T2 setup with more than 96K routes, execute CLI command -- generate_dump 2) No error should be shown 3) Download the generate_dump result and verify the saidump file after unpacking it.

Which release branch to backport (provide reason below if selected)

[ ] 201811
[ ] 201911
[ ] 202006
[ ] 202012
[ ] 202106
[ ] 202111
[x] 202205
[x] 202211
[x] 202305

Tested branch (Please provide the tested image version)

[x] 202205

kcudnik commented 1 year ago

change all \r\n to std::endl

kcudnik commented 1 year ago

please fix build errors

JunhongMao commented 1 year ago

please fix build errors I found the CodeQL has below error. How to fix it? nlohmann/json.hpp is already in source tree. Thanks.

SaiSerialize.cpp:6:10: fatal error: nlohmann/json.hpp: No such file or directory
6 | #include <nlohmann/json.hpp>
|          ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[4]: *** [Makefile:871: libsaimeta_la-SaiSerialize.lo] Error 1
make[4]: *** Waiting for unfinished jobs....
make[4]: Leaving directory '/home/runner/work/sonic-sairedis/sonic-sairedis/meta'
make[3]: Leaving directory '/home/runner/work/sonic-sairedis/sonic-sairedis/meta'
make[3]: *** [Makefile:502: all] Error 2
make[2]: *** [Makefile:457: all-recursive] Error 1
make[2]: Leaving directory '/home/runner/work/sonic-sairedis/sonic-sairedis'
make[1]: *** [Makefile:389: all] Error 2
dh_auto_build: error: make -j2 returned exit code 2
make[1]: Leaving directory '/home/runner/work/sonic-sairedis/sonic-sairedis'
make: *** [debian/rules:46: build] Error 25
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
Error: Process completed with exit code 2.

JunhongMao commented 1 year ago

@kcudnik, please help to review it again, thanks.

mlok-nokia commented 11 months ago

@judyjoseph Need to cherry-pick all related to Ps to 202205 branch

gechiang commented 11 months ago

@StormLiangMS , @yxieca , Please review/approve for 202205 and 202305. MSFT ADO: 25892357

StormLiangMS commented 10 months ago

hi @JunhongMao @gechiang in 202305, we should use "#include " swss/json.hpp " " other than #include <nlohmann/json.hpp>, otherwise it will cause compile issue like below, make[3]: Entering directory '/home/runner/work/sonic-swss/sonic-sairedis/saidump' g++ -DHAVE_CONFIG_H -I. -I.. -Wdate-time -D_FORTIFY_SOURCE=2 -I/home/runner/work/sonic-swss/usr/include -g -I../SAI/inc -I../SAI/experimental -I../SAI/meta -I../lib -ansi -fPIC -std=c++14 -Wall -Wcast-align -Wcast-qual -Wconversion -Wdisabled-optimization -Werror -Wextra -Wfloat-equal -Wformat=2 -Wformat-nonliteral -Wformat-security -Wformat-y2k -Wimport -Winit-self -Wno-inline -Winvalid-pch -Wmissing-field-initializers -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-noreturn -Wno-aggregate-return -Wno-padded -Wno-switch-enum -Wno-unused-parameter -Wpacked -Wpointer-arith -Wredundant-decls -Wshadow -Wstack-protector -Wstrict-aliasing=3 -Wswitch -Wswitch-default -Wunreachable-code -Wunused -Wvariadic-macros -Wwrite-strings -Wno-switch-default -Wconversion -Wno-psabi -Wcast-align=strict -Wno-error -c -o saidump-saidump.o test -f 'saidump.cpp' || echo './'saidump.cpp saidump.cpp:17:10: fatal error: nlohmann/json.hpp: No such file or directory 17 | #include <nlohmann/json.hpp> | ^~~~~~~ compilation terminated. make[3]: [Makefile:477: saidump-saidump.o] Error 1 make[3]: Leaving directory '/home/runner/work/sonic-swss/sonic-sairedis/saidump' make[2]: [Makefile:444: all-recursive] Error 1 make[2]: Leaving directory '/home/runner/work/sonic-swss/sonic-sairedis' make[1]: [Makefile:376: all] Error 2 make[1]: Leaving directory '/home/runner/work/sonic-swss/sonic-sairedis' dh_auto_build: error: make -j1 returned exit code 2 make: [debian/rules:69: binary-syncd-vs] Error 25 Error: Process completed with exit code 2.

I will revert this PR from 202305, pls file separate PR to 202305 for this one.

gechiang commented 10 months ago

hi @JunhongMao @gechiang in 202305, we should use "#include " swss/json.hpp " " other than #include <nlohmann/json.hpp>, otherwise it will cause compile issue like below, make[3]: Entering directory '/home/runner/work/sonic-swss/sonic-sairedis/saidump' g++ -DHAVE_CONFIG_H -I. -I.. -Wdate-time -D_FORTIFY_SOURCE=2 -I/home/runner/work/sonic-swss/usr/include -g -I../SAI/inc -I../SAI/experimental -I../SAI/meta -I../lib -ansi -fPIC -std=c++14 -Wall -Wcast-align -Wcast-qual -Wconversion -Wdisabled-optimization -Werror -Wextra -Wfloat-equal -Wformat=2 -Wformat-nonliteral -Wformat-security -Wformat-y2k -Wimport -Winit-self -Wno-inline -Winvalid-pch -Wmissing-field-initializers -Wmissing-format-attribute -Wmissing-include-dirs -Wmissing-noreturn -Wno-aggregate-return -Wno-padded -Wno-switch-enum -Wno-unused-parameter -Wpacked -Wpointer-arith -Wredundant-decls -Wshadow -Wstack-protector -Wstrict-aliasing=3 -Wswitch -Wswitch-default -Wunreachable-code -Wunused -Wvariadic-macros -Wwrite-strings -Wno-switch-default -Wconversion -Wno-psabi -Wcast-align=strict -Wno-error -c -o saidump-saidump.o test -f 'saidump.cpp' || echo './'saidump.cpp saidump.cpp:17:10: fatal error: nlohmann/json.hpp: No such file or directory 17 | #include <nlohmann/json.hpp> | ^~~~~~~ compilation terminated. make[3]: [Makefile:477: saidump-saidump.o] Error 1 make[3]: Leaving directory '/home/runner/work/sonic-swss/sonic-sairedis/saidump' make[2]: [Makefile:444: all-recursive] Error 1 make[2]: Leaving directory '/home/runner/work/sonic-swss/sonic-sairedis' make[1]: [Makefile:376: all] Error 2 make[1]: Leaving directory '/home/runner/work/sonic-swss/sonic-sairedis' dh_auto_build: error: make -j1 returned exit code 2 make: [debian/rules:69: binary-syncd-vs] Error 25 Error: Process completed with exit code 2.

I will revert this PR from 202305, pls file separate PR to 202305 for this one.

@JunhongMao , can you please address the commnets made by @StormLiangMS so we can make progreess with this PR which is blocking your other PR #1298 ... Thanks!

JunhongMao commented 10 months ago

@gechiang, @StormLiangMS , I have created a new PR: https://github.com/sonic-net/sonic-sairedis/pull/1335, please help to review. Thanks.

sonic-net / sonic-sairedis