uselagoon / lagoon-sync

Apache License 2.0
6 stars 5 forks source link

Support post-dump stream processing for synchers #107

Open bomoko opened 11 months ago

bomoko commented 11 months ago

We've found that in certain cases (for instance, the "DEFINER" statements added by https://www.drupal.org/project/dynamic_entity_reference) require some post mysqldump processing - for instance, piping the dump stream through sed and filtering out certain lines.

The idea here is to be able to support this kind of post-dump processing.

There are a few options here.

  1. Add the usual cases to the synchers themselves - have them do whatever filtering is reasonable given these edge cases we find.
  2. Allow the user to pass post dump processing instructions that will affect the mysqldump (among others) stream - imagine, for instance, passing through some regex to identify lines to be replaced, or phrases to be substituted
  3. Allow users to explicitly add commands to the current synchers
  4. Support custom synchers (https://github.com/uselagoon/lagoon-sync/pull/73)

Thoughts/feedback would be appreciated.

shreddedbacon commented 11 months ago

I think if users have the ability to do this using custom sync commands, then this would be the best path forward. Giving the power to the users to create their own command scripts that the sync tool can use definitely makes more sense.

I'm mainly against the idea of building post dump processing or streaming support into lagoon-sync because there are so many ways that people may want to do things, and capturing all these to then support within lagoon-sync seems like a battle that could be never ending. Let alone then having to continue to maintain that list or add new functionality to it over time because someone wants it to do something slightly different.