Maxwell async bootstrap mode - Disabling buffering

[IonutArhire]

Hi Maxwell team.

I'd like to be able to asynchronously replicate rows from the currently bootstrapping table, without them being queued and sent only at the end of the bootstrapping process. I'd like them not to be buffered so as to get sent as fast as maxwell can.

Is there a way for you to make this improvement? Would be of great help.

[osheroff]

this is certainly possible, but it's not implemented as the downstream consumer will have no idea which copy of the row -- the replicated one or the bootstrapped one -- is authoritative in a given situation. While I'm in favor of flexibility, I've tried to shy away from places where users could shoot themselves in the foot.

And believe you me, if you write a feature, people will turn it on and then proceed to blast their little tootsies off.

anyway, I'd accept a PR around this if it were adequately commented about how dangerous it is.

[IonutArhire]

Sorry for the late response Ben.

Seeing your response I thought I'd go with creating another maxwell custom producer that only does bootstrapping (using a small hack that makes the whole process shutdown gracefully if no bootstrapping rows are getting pushed to my custom producer anymore).

The problem I'm facing right now is having an old position in the positions table for my maxwell bootstrap process. This happens because the process stops after bootstrapping, leaving behind the position which will turn old in a maximum of 24 hours (binlog retention time).

Would you accept a merge request that ensures a valid position at startup? Let's say the client's position is old, meaning it points to a non existent binlog (like in my case explained above, but I think it can also happen for any maxwell client that, for some reason, misses the binlog change event). In this case, we delete the position. The position will then be recalculated as always (get other client position, or SHOW MASTER STATUS or the others that are already there).

What do you think?

[osheroff]

yeah so we already have --recapture_schema=TRUE|FALSE, I could see us supporting --recapture_schema=ON_MISSING or something

[osheroff]

ok I read your comment more closely. can we more precisely define the problem you're facing?

As I understand it:

• you're doing a large amount of bootstrapping, during which no rows can be processed.
• as this happens, the binlogs get rotated away and everything crashes.

And that's the fundamental issue that you were having initially when you opened this PR and also the issue in your second comment?

Impish question asks: "why not increase the retention period?"

but there may be something in here about how/why we buffer rows during a bootstrap. maybe we can improve significantly upon the process.

[redcape]

I'd like to understand this use-case better. For the custom producer, it sounds like the position is old because it takes a long time to bootstrap / the changes sent from maxwell are blocked, so the position does not move forward. Since it takes over 24 hours to bootstrap, this is a large table.

Anytime the position is too old, it needs to recapture the entire schema from scratch because it may have missed DDL changes. Similarly, the data may have changed compared to what was bootstrapped requiring another full bootstrap

Hopefully my poor ASCII art can help walk through this

time========================================================================>
=start binlog position=
==start bootstrap of table A==>> 
        ==writes to table A=  == writes to table A==                      == writes to table A==
                                      ===finish bootstrap of table A===
             === DDL changes==
                                                                        == start bootstrap of table B==>
                                                                                                      ==oldest available binlog after bootstrap==

I'm trying to understand what level of correctness is acceptable for this use-case. If the position is reset to oldest available binlog after bootstrap or any later position that is not start binlog position, you will lose writes that occurred to table A during or after the initial bootstrap.

Would you accept a merge request that ensures a valid position at startup? Let's say the client's position is old, meaning it points to a non existent binlog (like in my case explained above, but I think it can also happen for any maxwell client that, for some reason, misses the binlog change event). In this case, we delete the position. The position will then be recalculated as always (get other client position, or SHOW MASTER STATUS or the others that are already there).

What do you think?

I think this would be good as an option. I've run into the position running out a few times, but it's always been due to misconfigurations like the binlog retention incorrectly set too short or misconfiguration of maxwell. I wrote something similar that just drops the whole maxwell database if it happens for dev environments. However, I wouldn't want to enable this automatic reset in a production environment as running into it indicates something must have gone terribly wrong that needs investigation.

[IonutArhire]

Thank you guys for answering.

I will give you more context and go from the very start to make it easier for you.

I deploy my microservices in a kubernetes cluster, one of these microservices being built on top of maxwell, adding a custom producer to the mix. The architecture is multi-tenant and so one example of when I would like to run a maxwell bootstrap is when provisioning a new tenant. I do this so that the new tenant has access to the same data as the other tenants.

Problem is, while bootstrapping runs for the new tenant, the other tenants don't get their data replicated since the maxwell microservice is busy with bootstrapping. They need to wait, but bootstrapping can take a lot of time, sometimes even more than a day, depending mainly on the speed of the message broker. There's also the risk that newly created table records are no longer in the binlog if the bootstrapping takes more than 24 hours. The retention period could theoretically be increased to more than 24 hours, but I do not own that database, it is the responsability of another team unfortunately.

To solve this, I opened this github issue and asked if bootstrapping can be done in parallel with replication. After reading Ben's response, about this feature being potentially dangerous, I gave up on the idea and went for another route.

I wanted to create a sidecar container that only does bootstrapping and have this container, along with the replication container, running in parallel, inside the same deployment. The main reason why I didn't go for this is because the bootstrap container would have run indefinitely, when I actually wanted it to stop after done with bootstrapping (to put it in other words, I wanted the resources used for bootstrapping to be freed upon bootstrap completion).

As a result, I went with using a kubernetes Job that did the bootstrapping, since kubernetes Jobs are meant to eventually stop on their own (unlike Pods).

To make the job stop after bootstrap completion, I had to add code that, when run in bootstrapOnlyMode (as I named it) would gracefully exit the process after bootstrap completion.

I arrived to a point where I could run maxwell bootstrapping in a kubernetes job in parallel with the kubernetes pod used for maxwell replication.

Problem was, when starting the bootstrap job after not running it anymore for let's say, a whole week, I would get an error saying the client position points to a non-existent binlog position, which made sense, since the job didn't stick around and so it didn't get its client position updated.

To fix this, I wrote some "cleanup" code that would run at the start of my custom producer factory, that deletes the current client's position from the positions table. This would force maxwell to later update the position, and so to fix my problem.

I thought that this solution could be used by others as well, and so I wrote my second comment on this issue.

However, I never thought about missing schema updates i.e. DDL events. In my case, the table I'm replicating/bootstrapping is probably going to have the same schema well into the future, but it's still something that I need to keep in mind (thanks a lot for the info again).

Looking back, I think it would have been wiser to take the time to understand the library code so that I could do a pull request that offers the possibility of running bootstrapping in parallel with replication (it's never too late though).

Additional info (that may or may not help):

• The replicated/bootstrapped data is wrapped in an event that gets sent to the message broker and that event is handled idempotently, meaning I never have to shy away from doing a bootstrap.
• For triggering the bootstrap, I use an application built on top of the maxwell-bootstrap utility. It doesn't add much functionality on top of the utility. I guess the only differences are that it triggers bootstrap for multiple tenants at once and that it is idempotent, meaning if a bootstrap has already been completed, then it won't trigger a new one.

[osheroff]

Problem is, while bootstrapping runs for the new tenant, the other tenants don't get their data replicated since the maxwell microservice is busy with bootstrapping.

This isn't how bootstrapping is designed.

Let's say I'm bootstrapping database A, while replicating databases A, B, C.

While the bootstrap is running for A, rows for B and C should continue to flow through maxwell. Replicated rows for A will get buffered until the end of the bootstrap process.

Are you experiencing something else in production?

[IonutArhire]

You're right, the other tenants are impacted just because of how my wrapper application built on top of the maxwell-bootstrap utility is configured. It is configured such that, if there is no completed bootstrap registered in the bootstrap table let's say, it triggers bootstrap for all tenants (so if I drop the maxwell schema for example and recreate it it will trigger bootstrap for all).

This can be fixed on my part, but the issue still remains with running bootstrap for a single tenant: it blocks replication for that tenant for extended periods of time + newly inserted records can disappear from the binlog if retention period is passed.

[osheroff]

it blocks replication for that tenant for extended periods of time + newly inserted records can disappear from the binlog if retention period is passed.

Technically it doesn't do that either; it buffers the newly inserted records to disk and then replays them at the end of the bootstrap process, allowing replication to proceed forward. So it shouldn't (in a bug-free world 😄) actually block replication.

HAVING SAID ALL THIS:

It could be better. need to do a white boarding something, there's lots of moving parts, but maybe it can be made better.

[IonutArhire]

I understand. Bootstrapping in async mode would mean no records should be lost.

Then the only problem remaining I guess is the availability problem, i.e. replication is delayed by bootstrapping.