search

sripathikrishnan / redis-rdb-tools

Parse Redis dump.rdb files, Analyze Memory, and Export Data to JSON
https://rdbtools.com
MIT License
5.09k stars 743 forks source link

redis-rdb-tools integration into FastoRedis/FastoNoSQL #123

Open topilski opened 6 years ago

topilski commented 6 years ago

Hi maintainers, i have idea to integrate rdb into FastoRedis/FastoNoSQL. What you think about it? Pros and Cons of this integration?

oranagra commented 6 years ago

@topilski can you elaborate on what exactly you have in mind? what are features such integration will offer? and also how you intend to achieve that integration?

topilski commented 6 years ago

Hi @oranagra , i want to start integration usfull tools into FastoRedis application, i think it will help users. I thinking about execution python script and handling progress of dumping. It will be simple find script , pass needed args , handle execution in GUI. Users should specify path to rdb after that can execute commands. pip install or i can pack tools into installer package.

oranagra commented 6 years ago

So if i understand correctly this does not really consist of any integration between the redis client features and the offline rdb parsing features, but rather just having GUI for redis-rdb-tools inside the same GUI that's used as a redis client. I guess it may be useful, especially to lazy users that don't like to bother with command line arguments. On top of that, maybe you can have a nicer display of some reports (allowing sorting, search, and filtering)

How would you get the redis-rdb-tools code into the FastoRedis runtime? will you download it from pip / github on demand?

Two ways of actually integrating the Redis client features with the rdb parsing features are:

  1. use redis-rdb-tools protocol parser in order to send commands to a target Redis server (effectively loading the rdb into a running redis)
  2. send SYNC command to a Redis server in order to download an rdb over the network and then use redis-rdb-tools to analyze it.
topilski commented 6 years ago

Hi @oranagra , in mostly users use ssh connections, how i can understand rdb can be used only on same machine. I think that users should setup path to rdb and python. The solutions can be load dump file via scp (libssh2 library) and exec rdb with needed args, or call rdb remotely but it is not GUI side issue. Can you describe your 2 ways in more details?

oranagra commented 6 years ago

regarding 1) redis-rdb-tools has a tool that converts the RDB file into a stream of redis commands. if you send these commands to a redis server via a client connection, you effectively load that RDB into the target redis (merging it with whatever is already there, unless you FLUSHALL in advance)

regarding 2) What i mean is that if you're already connected to redis as a client, you can download an RDB file directly form redis (without doing BGSAVE and any ssh/scp).

You do this by sending SYNC command (like slaves do), download the RDB file that you get a response and either save it, or better yet, feed the stream of bytes directly into redis-rdb-tools and avoid the slowdown of writing and reading from the local disk.

Please note that you can send REPLCONF CAPA EOF, and if the master is set with repl-diskless-sync then the master sends you an RDB file without storing it on the master's disk, but note that the response to the SYNC command is slightly different in this case since the master doesn't know the size of the RDB file ahead of time.

chengkala commented 3 years ago

regarding 1) redis-rdb-tools has a tool that converts the RDB file into a stream of redis commands. if you send these commands to a redis server via a client connection, you effectively load that RDB into the target redis (merging it with whatever is already there, unless you FLUSHALL in advance)

regarding 2) What i mean is that if you're already connected to redis as a client, you can download an RDB file directly form redis (without doing BGSAVE and any ssh/scp).

You do this by sending SYNC command (like slaves do), download the RDB file that you get a response and either save it, or better yet, feed the stream of bytes directly into redis-rdb-tools and avoid the slowdown of writing and reading from the local disk.

Please note that you can send REPLCONF CAPA EOF, and if the master is set with repl-diskless-sync then the master sends you an RDB file without storing it on the master's disk, but note that the response to the SYNC command is slightly different in this case since the master doesn't know the size of the RDB file ahead of time.

how can I feed the stream of bytes directly into redis-rdb-tools ？