search

NANU-Wandel-Wege-entdecken / NANU-magazin-website

Maintenance of the existing system and the legacy theme of the NANU website.
https://nanu-magazin.org/
2 stars 0 forks source link

Make it damn cheap to refresh staging #35

Open ibes opened 4 years ago

ibes commented 4 years ago

The original needs

the plan

ibes commented 4 years ago

@michael-voit please check if there are more tasks that are needed for cloning and or needed after WPMigrate

ibes commented 4 years ago

There might be a cheap way to selectively use filters for that job depending on environment. With WordPlate we have .env-Files with knowledge about the current environment.

michael-voit commented 4 years ago

I think you got everything.... If I do refresh the staging next time, I can write down every step I make. But I'm quite sure that the list is a wonderful startingpoint.

ibes commented 4 years ago

change SMTP settings

This could be accomplished by https://github.com/wordplate/mail via .env settings that are not set or set so specific data on staging

Would require to introduce an .env file that will anyway come with WordPlate

ibes commented 4 years ago

deactivate CDN

what needs to be done here? Could it be enough to simple deactivate the CDN plugin?

Could CDN be a solution for syncing images to staging without the need to download them? I guess we talked about that option at some point for a minute but I don't remember the conclusion

alternatively there could be a function to trigger after-db-pull option overwrite. So a function that is triggered after WP Migrate DB did it's job that runs an option_update() function against the database to overwrite settings for eg. the CDN plugin. In the plugin we use, I have not found good filters or a way to overwrite the settings by configuration.

ibes commented 4 years ago

deactivate Backups

that could also be as cheap as deactivating the plugin, right?

ibes commented 4 years ago

maybe there are plugins to selectively activate or deactivate eg query monitor or caching

that is most likely worth to build.

Could be a MU plugin that filters the 'active_plugins'-Setting like in the shy-ajax-handler: https://github.com/ibes/shy-ajax-handler/blob/master/shy-ajax-handler.php

If switching to WordPlate these plugins would need to stay out of the mu-activation.

The configuration for that most likely should go in the .env-file so this can be configurated by instance.

Maybe I want to have some other plugins auto activated/deactivated on my local installation.

Live could be the main source and we add/substract what we don't want on staging / local or there could be a set of must use plugins that definitely need to be on everywhere and than each setup can define "also_active" plugins.

Hm - that would ultimately overpower the database. Which would be fine most of the time. But I feel like that could cause troubles.

Could it be a thing to deactivate the possibility to activate/deactivate plugins in live all together?

I guess this should be an action that runs after refresh -> so after pulling db change and after deployment After that there is the possibility to change settings/try out things.

ibes commented 4 years ago

add anonymization of DB

https://github.com/deliciousbrains/wp-migrate-db-anonymization

Install on live site - have deactivated on others

ibes commented 4 years ago

is there DB overhead to reduce?

Already trying to reduce overhead but will open a separate issue for that.

ibes commented 4 years ago

minimal meaningful sub-portion to pull

The goal is to reduce wait time for DB pull to encourage us to keep the database fresh It should be easy to have more than one pull profile in WP Migrate DB Pro.

What is the MVsP (minimal valuable sub portion)

Rarely important:

Yes, Options, Posts and Post_Meta should be enough for a quick update

ibes commented 4 years ago

WP Migrate DB Pro CLI

WP Migrate DB Pro has a CLI - so there would be the option to run a pull on each deployment to staging: https://deliciousbrains.com/wp-migrate-db-pro/doc/cli-push-pull-subcommand/

negative consequences: DB would be reset on each code deployment and could overwrite the current test

There could be an automatic pull on deploying branch "develop" only. As experiments are not supposed to happen by deploying "develop" this could even enforce good practice. Also checking develop on staging would be a good step before a release and this should definitely happen with a "as close as possible" DB to live

ibes commented 4 years ago

WP Migrate DB Pro Hooks

there is the possibility to preserve individual options (eg 'active_plugins') https://github.com/deliciousbrains/wp-migrate-db-pro-tweaks/blob/master/wp-migrate-db-pro-tweaks.php#L55 (https://snippets.webaware.com.au/snippets/use-wp-migrate-db-pro-and-keep-your-development-environment-settings/)

Run something after the migration https://github.com/deliciousbrains/wp-migrate-db-pro-tweaks/blob/master/wp-migrate-db-pro-tweaks.php#L99

this might be handy: 

global $wp_current_filter;
$is_remote = strstr($wp_current_filter[0], 'nopriv') ? true : false;
ibes commented 4 years ago

speed up migration a bit

up the size of rows per segment to lower the number of requests needed increase work load for server but live should be in great shape to at least double or quadruple that

add_filter( 'wpmdb_rows_per_segment', function () { return 200; // default is 100 } );

ibes commented 4 years ago

This will be maintained in the original issue (all the way up) now

so here is the plan

OPEN - image handling

michael-voit commented 4 years ago

change SMTP settings

This could be accomplished by https://github.com/wordplate/mail via .env settings that are not set or set so specific data on staging

Would require to introduce an .env file that will anyway come with WordPlate

Traumhaft. Vor der Umstellung auf production halt auf staging gut testen - aber das ist eh klar.... ;)

michael-voit commented 4 years ago

Could CDN be a solution for syncing images to staging without the need to download them? I guess we talked about that option at some point for a minute but I don't remember the conclusion

I think it could work for all images you also have on production. But if you test something with new images on staging, the CDN plugin rewrites the url and the CDN than looks on production for this file - and will not find it there....

So it should work in the most cases - but not always...

to be tested...

michael-voit commented 4 years ago

so here is the plan

* [ ]  optimal: switch to WordPlate now (for plugin sync and .env)

* [ ]  optimal: add sub portion pull setup to staging (just posts, post_meta, options)

* [ ]  minimal: add WP Migrate DB filter to not sync active_plugins (so locally deactivate WP Mail, CDN, backup)

OPEN - image handling

Sounds good to me (the optimal version).

The second point includes the option to run a pull on each deployment to staging, right? So we would include the CLI commands to deployHQ. Pull via CLI after the deployment, after that flush the cache via CLI.

Like it!

ibes commented 4 years ago

Sounds good to me (the optimal version).

The second point includes the option to run a pull on each deployment to staging, right? So we would include the CLI commands to deployHQ. Pull via CLI after the deployment, after that flush the cache via CLI.

Like it!

I forgot about the automatic db pull after deploying staging, good catch. I like the idea but I am worried that this could be over-automized as this might overwrite text database entries. Eg. tweaking around with new menu settings, then noticing that there needs to be a code update, deploying it, and needing to redo the temporary database setting.

But just limiting it on the develop branch feels like a great idea. That way feature branches can live in their own world for as long as that is needed (but can manually synchronize any time) but develop will always be (forced) on the newest database state.

Cache should be flushed on every deployment (all branches, all version), so should be the rewrite rules and transients.

ibes commented 4 years ago

For the images/attachments, I want to test Plugins/Solutions that "fall back" to server files. Oh, actually they would need to "fall back" on CDN files, right?

This is how it should work:

I started testing such tools but was not yet convinced/sucessful

possible solutions

michael-voit commented 4 years ago

Sounds good to me (the optimal version). The second point includes the option to run a pull on each deployment to staging, right? So we would include the CLI commands to deployHQ. Pull via CLI after the deployment, after that flush the cache via CLI. Like it!

I forgot about the automatic db pull after deploying staging, good catch. I like the idea but I am worried that this could be over-automized as this might overwrite text database entries. Eg. tweaking around with new menu settings, then noticing that there needs to be a code update, deploying it, and needing to redo the temporary database setting.

But just limiting it on the develop branch feels like a great idea. That way feature branches can live in their own world for as long as that is needed (but can manually synchronize any time) but develop will always be (forced) on the newest database state.

If we can limit it to only the develop branch, would be good. Even better could be to limit it to release-branches, if possible? In that way we would get a fresh clone for the final tests (I always do them in a release-branch) but can work as we are used to in develop.

Cache should be flushed on every deployment (all branches, all version), so should be the rewrite rules and transients.

absolutely. deployHQ should have the cli command to flush the cache after every deployment. I can check that once we implement this.

michael-voit commented 4 years ago

For the images/attachments, I want to test Plugins/Solutions that "fall back" to server files. Oh, actually they would need to "fall back" on CDN files, right?

This is how it should work:

* Client: I need a file

* Server: Ok, I am searching for the file

* Server is searching for the file locally - found it? great - delivering it!
  if not

* Server is searching on remote (-> CDN) - found it? great - delivering it!
  if not

* responding 404

I started testing such tools but was not yet convinced/sucessful

Exactly. I searched for such solutions, too, back than - but not found anything working.

ibes commented 4 years ago

Ich aktualisiere die initiale Issue-Karte um dort den Zwischenstand des Plans festzuhalten. Fühlt sich nach und nach immer runder an.

ibes commented 4 years ago

First update - Plugins

as WordPlate is already initiated in the "next" repository this task was checked from the beginning.

I have now configurated the plugin loading mechanism.

This plugins are not yet solved:

These are three "premium" plugins that are not open to being grabbed via composer and not popular enough to have workarounds (in contrast to ACF Pro)

I contacted the support of BackWpUp -> they use composer internally but not yet externally and will ask the support. I have not too much hopes right now but could be a thing to check their update mechanism. This plugin was in question before.

I contacted MailPoet but have not yet heard from them. There is nothing in their docs + nothing on google so I have not big hopes. I know this is a very essential plugin.

YARPP Experiments seems to be an old experiment thing of YARPPs. Do you know what that adds to the mix? Is it good? Is it still needed (sounds like the experiments might have reached the core in the meantime?!)

For these three there would be the possibility to just throw them in version control and update them the old fashion way. Not thrilled by that, but at least all the other plugins are in the new loop. There might be a way to figure out how to require the premium plugins as a zip - same way it is done for ACF Pro.

Any thoughts @michael-voit

michael-voit commented 4 years ago

Hey @ibes - thank you very much....

let's try it the other way round: is there a backup-plugin (just database is enough) and a newsletter plugin (that can at least send a weekly automated newsletter with the new articles) that would work with composer and ideally an env-file (as we have sensible access data for both of them)?

YARPP Experiments will be checked by me. I think, we will remove it...

michael-voit commented 4 years ago

YARPP Experiments is removed as it was not doing anything anymore with the newer versions of YARPP.

FYI: A new plugin we are using is "Lightweight Subscribe To Comments"

ibes commented 4 years ago

MailPoet is quite an advanced plugin for that task. I am not sure if there will be a replacement we actually want.

For backup: What does the pro-Version add? Oh, seems like playing back backups is a pro feature (which is super strange if that is not part of the free deal)

https://wordpress.org/plugins/updraftplus/ could be an alternative. I have used it in a couple of websites but never had shit hitting the fan - but as it is one of the popular ones I imagine it would be fine. It allows to store backups in the cloud. Which cloud service would you like to store the backup to? Oh - we just need backup of "/upload" and the database.

Cool - YARPP Experiments is removed.

Lightweight Subscribe To Comments is already in my list :)

michael-voit commented 4 years ago

Newsletter I also do not see any equivalent replacement for MailPoet.

Backup updraftplus could really be a solution. I will have a look soon. It's about database backups all 4 hours during the day and also a weekly update of the uploads folder (incremental would be fine - but with updraftplus this is also premium). The whole website is backuped every night via the server. I also thought about using mariabackup and automate it via a script. So we wouldn't need a plugin.

ibes commented 4 years ago

BackWpUp Pro could you give me login data to the site where BackWpUp is managed? Maybe it is possible to make it composer compatible similar to how ACF Pro is handled.

ibes commented 4 years ago

Newsletter The same for MailPoet Pro would be great. Maybe I can make it work with Composer