Feature Request - Manual entries in local config file to be merged with crawled results

[gwharton]

Provide the option for a local config file, that could be read, which is in the same format as the magepack.config.js file, and merge these entries with the ones produced from the crawl. Thus allowing users to manually add things to the bundles that don't get picked up by the automatic crawl (perhaps they only appear on one page somewhere that wasn't crawled)

[gwharton]

An example of a use case for this. I set magepack to crawl urls on my staging environment during deployment. However the staging environment does not have Google Tag Manager enabled, to avoid it firing off to google during deployment. Therefore I could add this entry manually as I know it will be skipped. Another use case, my staging environment does not contain any sales infomation, so my homepage that usually includes a list of best selling products along with some javascript slider functions also does not appear. This could also be added manually.

[krzksz]

Would it be enough that the newly generated config would merge into (instead of overwriting) an existing one if it finds it?

[gwharton]

It would be ok for me as my initial magepack.config.js file with manual entries would be grabbed from source control, then added to by the magepack program and finally used to deploy the site. For every deploy a fresh copy of my local file would be grabbed from source control. Not everybody operates like that though. I can see that approach being problematic for the way that some users operate. I can hear some screaming "Why does it overwrite my file!!!! Now ive gotta remove all these entries if I want it to recrawl from fresh". Successive reruns of magepack would end up crawling the site and then deferring mostly to what is in the existing config file. Entries that are no longer valid would only be removed by manual intervention. A better approach would be to merge contents from magepack.config.local.js into the final magepack.config.js leaving the users local file unchanged.

[gaiterjones]

@gwharton @krzksz do you have any tips for troubleshooting console errors seen after bundling. I have a couple an errors affecting a product slider, although the slider js appears to be included in the bundled file.

[gwharton]

Have you looked at PR 27690 aswell as the required 25587. 27690 was needed to resolve js errors for me.

[gwharton]

@krzksz I was thinking it would probably be easier to do the merging at the "magepack bundle" stage, as it just needs to read magepack.config.js and magepack.config.local.js (if exists) and merge the two arrays in memory and get rid of duplicates.

[Paulsky]

@gwharton May I ask you what your workaround on this is? Because we have somewhat the same scenario (different JS code in different environments).

Maybe ignore the magepack.config.js in GIT for now? And then on each deployment:

• disable Magepack bundling
• compile, clear/deploy static files and refresh the cache
• generate for this environment
• create a new bundle
• enable Magepack bundling
• refresh the cache

Not sure if this is the right way to go?

[gwharton]

Hi Paul,

Our deployment goes something like this.

On Development Node

• Clear Staging dir
• Clear Staging database
• Clone git repo into staging folder (app/code, app/design, composer.json, compose.lock, app/etc/env.production.php|env.staging.php|env.development.php/*|patches)
• rm app/etc/env.php (symbolic link to env.development.php)
• rm app/etc/config.php (we dont have disabled modules, so everything will be enabled by removing config)
• composer install
• php bin/magento setup:install .......... with staging db credentials and staging website urls
• mysql dump core_config table from dev database to staging database (Adjust ID's to start at 1). (Ensure env.staging.php contains overrides for all config that differs from dev config to staging config. Therefore no need to fiddle with config)
• mysql dump product, store, cms, eav and any other tables required to have products and add to cart in staging store from dev database to staging database
• rsync dev media folder to staging media folder
• symbolic link env.php to env.staging.php
• php bin/magento app:config:import
• php bin/magento setup:upgrade
• php bin/magento setup:di:compile
• php bin/magento setup:static-content:deploy
• php bin/magento indexer:reindex
• php bin/magento cache:clean
• php bin/magento cache:enable
• magepack generate giving urls to product in staging site - Adjust magepack.config.js by manually awking and grepping to remove or add lines as required
• magepack bundle
• symbolic link env.php to env.production.php
• Clear stagings media folder
• Symbolic link to media folder to actual location of media mount on production nodes.
• tar up the staging folder, excluding any files/folders not required on production node
• transfer to production node

On production Node

• extract tar to tmp folder
• php bin/magento maintenance:enable
• Backup old www folder
• Move tmp folder into place
• php bin/magento app:config:import
• php bin/magento setup:upgrade --keep-generated
• php bin/magento cache:clean
• php bin/magento maintenance:disable

You can see we create the magepack.config.js file from scratch every time we deploy by running magepack against our staging site which we have preprepared with all of the necessary cms pages and products to make it browseable and contain products. You only need to ensure you have enough content for the three magepack urls, plus ability to add product to cart.

Once that is done we manually use grep and awk (it would be nice if there was an easier way to add/remove entries from the config without having to manually awk and grep it) to modify the set to remove items that result in errors after bundling, or add things that we know will have been missed.

Then the staging site files are tar'd up and transfered to production nodes, followed by db upgrade on production node. The staging site is then culled.

We also run some MFTF tests against our staging site to make sure some key customisations are not broken.

The whole thing is wrapped up in a single deploy.sh script.

Probably more detail than you asked for here, but may spark some ideas.

[Paulsky]

@gaiterjones Thank you so much for your detailed answer! I really appreciate it! Indeed a little more detail what I asked for but this is helping me a lot, thank you once again!