search

forrestkirby / herzogdupont-joomla

Herzog Dupont is an extension which adds more elements to the YOOtheme Pro page builder developed by YOOtheme.
https://herzog-dupont.de/yootheme-pro
GNU General Public License v3.0
20 stars 7 forks source link

Certain Assets Not Crawled by Googlebot due to robots.txt #68

Open ibweb opened 2 months ago

ibweb commented 2 months ago

We love your plugin! This is somewhat a mix of a bug and feature request. Front end-functionality is fine, but we are concerned with Googlebot crawling and rendering which can affect Google understanding and indexing our content properly. Using Joomla's default robots.txt file, certain assets (typically CSS and JS) get blocked by Googlebot. This is the case in multiple elements. For example, HD Comparison Slider: ../plugins/system/herzogdupont/modules/elements/elements/hd-image-comparison/css/hd-image-comparison.css?4.4.5 ../plugins/system/herzogdupont/modules/elements/elements/hd-image-comparison/js/hd-image-comparison.js?4.4.5

HD Counter: ../plugins/system/herzogdupont/modules/elements/elements/hd-counter/css/hd-counter.css?4.4.5 ../plugins/system/herzogdupont/modules/elements/elements/hd-counter/js/hd-counter.js?4.4.5

I would imagine this is true with other HD elements as well.

Would it make more sense to locate those assets in a crawlable folder, that way Googlebot can crawl and render those pages successfully?

Note that this is tested in 1.8.0. I didn't see anything in the 1.9.0 changelog that would imply this has been addressed - correct me if I'm wrong.

ibweb commented 2 months ago

@forrestkirby is there any update on this?