search

google / mysql-ripple

Ripple, a server that can serve as a middleman in MySQL replication
Apache License 2.0
373 stars 46 forks source link

Generating new binlog.index from logs #15

Open ctutte opened 5 years ago

ctutte commented 5 years ago

Hi!

Currently binlog.index format is like this: filename= start_pos= start_master_pos= end_pos= last_master_pos=

Is it possible to add a new flag to generate a new binlog.index file from a list (or dir) of binlogs ? i.e: parse all binlogs and generate resulting binlog.index ? So that a slave can connect and apply all the binlogs ripple has?

pivanof commented 5 years ago

I think it should be possible to do this. Can you clarify what would be the use case for this?

ctutte commented 5 years ago

Hi pivanof, thanks for your reply.

The objective is to use ripple as a "fake master". Copy binlogs from another server (or a backup server) and start streaming from ripple to another slave/s.

Some reasons for using a fake master instead of directly executing binlogs on slave are: 1) No need to parse/convert binlogs (except maybe for the initial procedure to convert them to ripple) 2) No need to worry about binary strings/blobs having null and ending stream. 3) No need to worry about encoding 4) Possibility to skip events (SQL_SLAVE_SKIP_COUNTER) , and to apply "in batches" using "START SLAVE UNTIL" 5) In case of timeout or error, replication can be stopped/restored smoothly

Being able to convert regular binlogs to ripple binlogs can help ease using fake master.

pivanof commented 5 years ago

You seem to explain only the benefits of using the "fake server" in the first place, i.e. the benefits of using Ripple at all. But why would you want to copy binlogs from another server and make Ripple use them, rather than point Ripple to replicate from that server, so that Ripple downloads the binlogs itself?

ctutte commented 5 years ago

For example in case I need to restore an old backup + binlogs which are not currently on master. Or in case master have crashed and need to recover applying binlogs.

pivanof commented 5 years ago

Thanks! That makes sense. But I think instead of generating the index file from the available binlog files it would make more sense to recognize the MySQL's index file in that case. Overall I agree it would be a useful feature to have.

techwolf359 commented 5 years ago

We store our data and binary logs offsite and if we lose access to the current data, we'd like to be able to restore our data and then play the binlogs back into a server via native replication. In this scenario, I don't need ripple to pull from anywhere, I just want it to serve binary logs to a server.