Closed tfantina closed 3 years ago
You should see errors in your storage/logs/web.log
-- what are the logged errors?
related: https://nystudio107.com/blog/zen-and-the-art-of-craft-cms-log-file-reading
The fact that the robots.txt
on their staging server (or their production server for that matter) is 404'ing should not be affecting any ranking, though.
I'm looking into an issue for a client whose rankings have dropped off considerably.
What does this mean, exactly?
Looks like it is working on the live site:
https://www.oneworkplace.com/robots.txt
...but I think that's not coming from SEOmatic. If you want the SEOmatic-generated robots.txt
you will need to remove that location = /robots.txt
from your Nginx config, as per:
https://nystudio107.com/docs/seomatic/configuring.html#robots
...and the meta tag is correct on the homepage's HTML:
<meta name="robots" content="all">
Hey Andrew thanks for the quick response. We added a temporary robots.txt to the live site for now but I do want robots.txt to be the SEOmatic-generated one. As I mentioned, we don't have a location
block to robots.txt
in the Nginx configs.
I'm afraid the error may be a little obfuscated because we don't have a 404 page so it's rendering that error, maybe I need to create one to get a more clear picture of what's happening to the robots.txt file?
2021-12-03 16:09:47 [-][-][l40r7slv4ue3d52bnkti1l12b2][error][craft\web\twig\TemplateLoaderException] craft\web\twig\TemplateLoaderException: Unable to find the template “templates/404.twig”. in /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/twig/TemplateLoader.php:116
Stack trace:
#0 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/twig/TemplateLoader.php(72): craft\web\twig\TemplateLoader->_resolveTemplate('templates/404.t...')
#1 /var/www/html/staging.oneworkplace.com/vendor/twig/twig/src/Environment.php(299): craft\web\twig\TemplateLoader->getCacheKey('templates/404.t...')
#2 /var/www/html/staging.oneworkplace.com/vendor/twig/twig/src/Environment.php(381): Twig\Environment->getTemplateClass('templates/404.t...')
#3 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/twig/Environment.php(41): Twig\Environment->loadTemplate('templates/404.t...', NULL)
#4 /var/www/html/staging.oneworkplace.com/vendor/twig/twig/src/Environment.php(359): craft\web\twig\Environment->loadTemplate('templates/404.t...')
#5 /var/www/html/staging.oneworkplace.com/vendor/twig/twig/src/Environment.php(318): Twig\Environment->load('templates/404.t...')
#6 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/View.php(389): Twig\Environment->render('templates/404.t...', Array)
#7 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/View.php(450): craft\web\View->renderTemplate('templates/404.t...', Array)
#8 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/Controller.php(241): craft\web\View->renderPageTemplate('templates/404.t...', Array, 'site')
#9 /var/www/html/staging.oneworkplace.com/vendor/dolphiq/redirect/src/controllers/RedirectController.php(116): craft\web\Controller->renderTemplate('templates/404.t...', Array)
#10 [internal function]: dolphiq\redirect\controllers\RedirectController->actionIndex()
#11 /var/www/html/staging.oneworkplace.com/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#12 /var/www/html/staging.oneworkplace.com/vendor/yiisoft/yii2/base/Controller.php(180): yii\base\InlineAction->runWithParams(Array)
#13 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/Controller.php(190): yii\base\Controller->runAction('index', Array)
#14 /var/www/html/staging.oneworkplace.com/vendor/yiisoft/yii2/base/Module.php(528): craft\web\Controller->runAction('index', Array)
#15 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/Application.php(274): yii\base\Module->runAction('redirect/redire...', Array)
#16 /var/www/html/staging.oneworkplace.com/vendor/yiisoft/yii2/web/Application.php(103): craft\web\Application->runAction('redirect/redire...', Array)
#17 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/Application.php(259): yii\web\Application->handleRequest(Object(craft\web\Request))
#18 /var/www/html/staging.oneworkplace.com/vendor/yiisoft/yii2/base/Application.php(386): craft\web\Application->handleRequest(Object(craft\web\Request))
#19 /var/www/html/staging.oneworkplace.com/web/index.php(21): yii\base\Application->run()
#20 {main}
2021-12-03 16:09:47 [-][-][l40r7slv4ue3d52bnkti1l12b2][info][application] $_GET = []
$_POST = []
$_FILES = []
$_COOKIE = [
'CraftSessionId' => 'l40r7slv4ue3d52bnkti1l12b2'
'CRAFT_CSRF_TOKEN' => '•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••'
'1031b8c41dfff97a311a7ac99863bdc5_username' => '7b0ae12da6b63ad94617571ae36fae785b0d8b453d0a50ca937a8d4e25532fd0a:2:{i:0;s:41:\"1031b8c41dfff97a311a7ac99863bdc5_username\";i:1;s:23:\"travis@objectiveinc.com\";}'
'__stripe_mid' => '65ce8857-2932-438a-8542-d40e963e95d0f7752c'
]
$_SESSION = [
'bd62416aa8538ede709019a5e113eea5__flash' => []
]
$_SERVER = [
'USER' => 'ubuntu'
'HOME' => '/home/ubuntu'
'HTTP_COOKIE' => 'CraftSessionId=l40r7slv4ue3d52bnkti1l12b2; CRAFT_CSRF_TOKEN=b6326c9497df8ecb6f683fa6fb62662c0f8f76c5c2c65db33c0724eac981e9d5a%3A2%3A%7Bi%3A0%3Bs%3A16%3A%22CRAFT_CSRF_TOKEN%22%3Bi%3A1%3Bs%3A212%3A%22iBn4m9Q3smGheGBajATB2g-vnlTYH5zNZT2Atxfk%7C12a94f4d77247d427c8475094abc8c7f3f76d8b471850b0b301f08b0590e4f90iBn4m9Q3smGheGBajATB2g-vnlTYH5zNZT2Atxfk%7C16353%7C%242y%2413%241zrGJxz6QxdRnAJfHPWckuHE0vFXA83S82SNE5xfo9XL09S9MOj2K%22%3B%7D; 1031b8c41dfff97a311a7ac99863bdc5_username=7b0ae12da6b63ad94617571ae36fae785b0d8b453d0a50ca937a8d4e25532fd0a%3A2%3A%7Bi%3A0%3Bs%3A41%3A%221031b8c41dfff97a311a7ac99863bdc5_username%22%3Bi%3A1%3Bs%3A23%3A%22travis%40objectiveinc.com%22%3B%7D; __stripe_mid=65ce8857-2932-438a-8542-d40e963e95d0f7752c'
'HTTP_ACCEPT_LANGUAGE' => 'en-US,en;q=0.9'
'HTTP_ACCEPT_ENCODING' => 'gzip, deflate, br'
'HTTP_SEC_FETCH_DEST' => 'document'
'HTTP_SEC_FETCH_USER' => '?1'
'HTTP_SEC_FETCH_MODE' => 'navigate'
'HTTP_SEC_FETCH_SITE' => 'none'
'HTTP_SEC_GPC' => '1'
'HTTP_ACCEPT' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
'HTTP_USER_AGENT' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36'
'HTTP_UPGRADE_INSECURE_REQUESTS' => '1'
'HTTP_CACHE_CONTROL' => 'max-age=0'
'HTTP_HOST' => 'staging.oneworkplace.com'
'PATH_INFO' => ''
'SCRIPT_FILENAME' => '/var/www/html/staging.oneworkplace.com/web/index.php'
'REDIRECT_STATUS' => '200'
'SERVER_NAME' => 'staging.oneworkplace.com'
'SERVER_PORT' => '443'
'SERVER_ADDR' => '50.116.6.247'
'REMOTE_PORT' => '50152'
'REMOTE_ADDR' => '108.180.1.219'
'SERVER_SOFTWARE' => 'nginx/1.18.0'
'GATEWAY_INTERFACE' => 'CGI/1.1'
'HTTPS' => 'on'
'REQUEST_SCHEME' => 'https'
'SERVER_PROTOCOL' => 'HTTP/2.0'
'DOCUMENT_ROOT' => '/var/www/html/staging.oneworkplace.com/web'
'DOCUMENT_URI' => '/index.php'
'REQUEST_URI' => '/robots.txt'
'SCRIPT_NAME' => '/index.php'
'CONTENT_LENGTH' => ''
'CONTENT_TYPE' => ''
'REQUEST_METHOD' => 'GET'
'QUERY_STRING' => ''
'FCGI_ROLE' => 'RESPONDER'
'PHP_SELF' => '/index.php'
'REQUEST_TIME_FLOAT' => 1638576587.2647
'REQUEST_TIME' => 1638576587
'ENVIRONMENT' => 'staging'
'SECURITY_KEY' => '••••••••••••••••••••••••••••••••'
'DB_DRIVER' => '----'
'DB_SERVER' => '----'
'DB_USER' => '----'
'DB_PASSWORD' => '••••••••••••••••••'
'DB_DATABASE' => '----'
'DB_SCHEMA' => 'public'
'DB_TABLE_PREFIX' => ''
'DB_PORT' => '----'
'DEV_SITE_URL' => 'https://staging.oneworkplace.com'
'ASSETS_BASE_URL' => 'https://oneworkplace.com'
]
So SEOmatic doesn't appear anywhere in that stack trace -- instead, for whatever reason, your Redirects plugin is kicking in:
#9 /var/www/html/staging.oneworkplace.com/vendor/dolphiq/redirect/src/controllers/RedirectController.php(116): craft\web\Controller->renderTemplate('templates/404.t...', Array)
...and it's attempting to render a template that apparently doesn't exist, with the path templates/404.twig
So two take-aways here:
dolphiq/Redirects
plugin/robots.txt
file throwing a 404 error would not cause a diminishing in any Google SERP rankingsShould you end up needing it, I do make a redirects plugin called Retour as well: https://nystudio107.com/plugins/retour
If there's anything else I can help with, let me know.
I'm looking into an issue for a client whose rankings have dropped off considerably. Likely this is because their page is 404'd. Unfortunately, I can't say when this started exactly. I just upgraded to the latest version of SEOmatic, the settings seem to be pretty standard. I checked the nginx configs for
location = /robots.txt { access_log off; log_not_found off; }
and we don't have that location block (in eithernginx.confd
orsites-available
).Oddly I'm not seeing any new entries in web-404.log.
Craft CMS: 3.5.17.1 SEOMatic: 3.4.20 PHP: 7.0.33 Nginx: 1.18.0