Magento 2 Recipe: Cache prefix generated different cache prefixes when deploying to multiple hosts

[dverkade]

Situation:

We have Magento 2 projects which are load balanced between different servers. For instance 2 frontend web servers for visitors and a seperate backend service for the Magento backoffice. All servers point to the same Redis instance which is located on the database server. We're deploying with the configuration:

after: deploy:shared: magento:set_cache_prefix deploy:magento: magento:cleanup_cache_prefix

Actual result:

The set cache prefix is different per host because it will take the alias and release number. So for one server it will be:

production_web35_12_

And on the other server the prefix is set to:

production_web40_9_

Issue:

The issue is that cache will not be cleared correctly. The prefix should be the same for all 3 webservers in order for 1 webserver to clear the cache for the other two as well.

Expected result:

The generated cache key should be the same for all hosts within the same "stage" and release. So if we deploy to "stage" production to 3 webservers, the generated key should be the same for all 3 servers for Magento to work correctly.

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[dverkade]

@valguss Hope you have any suggestions how we could resolve this.

[valguss]

For the moment i'd suggest disabling using the cache_prefixes Is there a way for you to sync up the releases, so they are the same number?

[dverkade]

@valguss, yes, I can get the release in sync quite easily. However I'm stuck with the first / "alias" part of the key.

[valguss]

You could override the task in your deployer.php with something like the following (not ideal I know):

desc('Update cache id_prefix');
task('magento:set_cache_prefix', function () {
    //download current env config
    $tmpConfigFile = tempnam(sys_get_temp_dir(), 'deployer_config');
    download('{{deploy_path}}/shared/' . ENV_CONFIG_FILE_PATH, $tmpConfigFile);
    $envConfigArray = include($tmpConfigFile);
    //set prefix to `alias_releasename_`
    $prefixUpdate = 'abc_' . get('release_name') . '_';

    //check for preload keys and update
    if (isset($envConfigArray['cache']['frontend']['default']['backend_options']['preload_keys'])) {
        $oldPrefix = $envConfigArray['cache']['frontend']['default']['id_prefix'];
        $preloadKeys = $envConfigArray['cache']['frontend']['default']['backend_options']['preload_keys'];
        $newPreloadKeys = [];
        foreach ($preloadKeys as $preloadKey) {
            $newPreloadKeys[] = preg_replace('/^' . $oldPrefix . '/', $prefixUpdate, $preloadKey);
        }
        $envConfigArray['cache']['frontend']['default']['backend_options']['preload_keys'] = $newPreloadKeys;
    }

    //update id_prefix to include release name
    $envConfigArray['cache']['frontend']['default']['id_prefix'] = $prefixUpdate;
    $envConfigArray['cache']['frontend']['page_cache']['id_prefix'] = $prefixUpdate;

    //Generate configuration array as string
    $envConfigStr = '<?php return ' . var_export($envConfigArray, true) . ';';
    file_put_contents($tmpConfigFile, $envConfigStr);
    //upload updated config to server
    upload($tmpConfigFile, '{{deploy_path}}/shared/' . TMP_ENV_CONFIG_FILE_PATH);
    //cleanup tmp file
    unlink($tmpConfigFile);
    //delete the symlink for env.php
    run('rm {{release_or_current_path}}/' . ENV_CONFIG_FILE_PATH);
    //link the env to the tmp version
    run('{{bin/symlink}} {{deploy_path}}/shared/' . TMP_ENV_CONFIG_FILE_PATH . ' {{release_path}}/' . ENV_CONFIG_FILE_PATH);
});

I've just updated

$prefixUpdate = get('alias') . '_' . get('release_name') . '_';

to

$prefixUpdate = 'abc_' . get('release_name') . '_';

[dverkade]

@valguss, thanks. We can do a workaround but would like to see this fixed in the official recipe as well and would like to contribute to a solution.

Best practice is to have the same redis DB per environment. So in theory we could drop the "get('alias')" part of the prefix all together. Or we could use the "stage" of the host, so that the first part of the prefix is the same for all hosts in that stage. Can you just do get('stage') in order to get the host stage?

[valguss]

Using stage sounds like a better solution. The premise I was going for was for our specific setup where staging sites sit on the same server as the live env, so was trying to make the key unique to each environment so that they didn't clash. Will get a PR up shortly to change it though.

Thanks

[dverkade]

Using stage sounds like a better solution. The premise I was going for was for our specific setup where staging sites sit on the same server as the live env, so was trying to make the key unique to each environment so that they didn't clash. Will get a PR up shortly to change it though.

Thanks

Stage sounds good to me. I get your use case and with using "stage" this still works. I do think it's better to use a different Redis database number for a test environment and a live environment when they are both on the same server. Something like this:

            'page_cache' => [
                'backend' => 'Cm_Cache_Backend_Redis',
                'backend_options' => [
                    'server' => '127.0.0.1',
                    'port' => '6379',
                    'database' => '1',  ## HAVE A DIFFERENT DB NUMBER HERE FOR TEST OR PRODUCTION ENVIRONMENTS
                    'compress_data' => '0'
                ]
            ]

[valguss]

Now updated to check if stage is available and fall back to alias

[barryvdh]

What do you think about something like this:

desc('Update cache id_prefix');
task('magento:set_cache_prefix', function () {
    // Copy shared file to current release
    run('rm {{release_path}}/app/etc/env.php');
    run('cp {{deploy_path}}/shared/app/etc/env.php {{release_path}}/app/etc/env.php');

    $prefix = uniqid() .'_';
    run('{{bin/php}} {{release_or_current_path}}/bin/magento setup:config:set --cache-id-prefix='.$prefix.' --page-cache-id-prefix='.$prefix.' -n');
});

/**
 * After successful deployment, move the .env back to a symlink
 */
desc('Cleanup cache id_prefix env files');
task('magento:cleanup_cache_prefix', function () {
    run('mv {{deploy_path}}/shared/app/etc/env.php {{deploy_path}}/shared/app/etc/env.php.backup');
    run('mv {{release_path}}/app/etc/env.php {{deploy_path}}/shared/app/etc/env.php');
    run('{{bin/symlink}} {{deploy_path}}/shared/app/etc/env.php {{release_path}}/app/etc/env.php');
    run('rm {{deploy_path}}/shared/app/etc/env.php.backup');
});

So

• Copy env to release
• Run command to update prefix
• Run build with new env
• Move env back to shared.

You don't actually need the previous number do you? And to sync accross installs, maybe you can use a static variable to generate it once?

desc('Update cache id_prefix');
task('magento:set_cache_prefix', function () {

    // Copy shared file to current release
    run('rm {{release_path}}/app/etc/env.php');
    run('cp {{deploy_path}}/shared/app/etc/env.php {{release_path}}/app/etc/env.php');

    static $magentoCachePrefix;
    $magentoCachePrefix = $magentoCachePrefix ?: uniqid();

    run('{{bin/php}} {{release_or_current_path}}/bin/magento setup:config:set --cache-id-prefix='.$magentoCachePrefix.' --page-cache-id-prefix='.$magentoCachePrefix.' -n');
});

[github-actions[bot]]

This issue has been automatically closed. Please, open a discussion for bug reports and feature requests.

Read more: [https://github.com/deployphp/deployer/discussions/3888]

[github-actions[bot]]

This issue has been automatically closed. Please, open a discussion for bug reports and feature requests.

Read more: [https://github.com/deployphp/deployer/discussions/3888]

[dverkade]

Not sure why this is being closed as the PR is still open.

[peterjaap]

The bot went a bit wild I guess. Reopening.

[github-actions[bot]]

This issue has been automatically closed. Please, open a discussion for bug reports and feature requests.

Read more: [https://github.com/deployphp/deployer/discussions/3888]

[barryvdh]

lol

[mautz-et-tong]

How good that we have bots to help us by taking over annoying tasks and taking some of the burden off our shoulders these days. I can't wait for AI to step in and help the bot to be even more productive for us.

[peterjaap]

Apparently the bot was a bit drunk. Reopening. Let's see if the gets the hint :-P

[github-actions[bot]]

This issue has been automatically closed. Please, open a discussion for bug reports and feature requests.

Read more: [https://github.com/deployphp/deployer/discussions/3888]

[mautz-et-tong]

Ahh, I think @antonmedv has disallowed issues in general and just want to have discussion 👀

[barryvdh]

Maybe set exempt-issue-labels and tag it with one of those? https://github.com/actions/stale?tab=readme-ov-file#exempt-issue-labels

[peterjaap]

@mautz-et-tong you're right, I hadn't seen that https://github.com/deployphp/deployer/discussions/3888

[antonmedv]

Ahh, I think @antonmedv has disallowed issues in general and just want to have discussion 👀

Yes. =)