search

nystudio107 / craft-seomatic

SEOmatic facilitates modern SEO best practices & implementation for Craft CMS 3. It is a turnkey SEO system that is comprehensive, powerful, and flexible.
https://nystudio107.com/plugins/seomatic
Other
165 stars 70 forks source link

Internal server error for robots.txt, humans.txt etc. #1025

Closed tfantina closed 2 years ago

tfantina commented 2 years ago

I'm looking into an issue for a client whose rankings have dropped off considerably. Likely this is because their page is 404'd. Unfortunately, I can't say when this started exactly. I just upgraded to the latest version of SEOmatic, the settings seem to be pretty standard. I checked the nginx configs for location = /robots.txt { access_log off; log_not_found off; } and we don't have that location block (in either nginx.confd or sites-available).

Oddly I'm not seeing any new entries in web-404.log.

Craft CMS: 3.5.17.1 SEOMatic: 3.4.20 PHP: 7.0.33 Nginx: 1.18.0

khalwat commented 2 years ago

You should see errors in your storage/logs/web.log -- what are the logged errors?

related: https://nystudio107.com/blog/zen-and-the-art-of-craft-cms-log-file-reading

The fact that the robots.txt on their staging server (or their production server for that matter) is 404'ing should not be affecting any ranking, though.

I'm looking into an issue for a client whose rankings have dropped off considerably.

What does this mean, exactly?

khalwat commented 2 years ago

Looks like it is working on the live site:

https://www.oneworkplace.com/robots.txt

...but I think that's not coming from SEOmatic. If you want the SEOmatic-generated robots.txt you will need to remove that location = /robots.txt from your Nginx config, as per:

https://nystudio107.com/docs/seomatic/configuring.html#robots

...and the meta tag is correct on the homepage's HTML:

<meta name="robots" content="all">
tfantina commented 2 years ago

Hey Andrew thanks for the quick response. We added a temporary robots.txt to the live site for now but I do want robots.txt to be the SEOmatic-generated one. As I mentioned, we don't have a location block to robots.txt in the Nginx configs.

I'm afraid the error may be a little obfuscated because we don't have a 404 page so it's rendering that error, maybe I need to create one to get a more clear picture of what's happening to the robots.txt file?

2021-12-03 16:09:47 [-][-][l40r7slv4ue3d52bnkti1l12b2][error][craft\web\twig\TemplateLoaderException] craft\web\twig\TemplateLoaderException: Unable to find the template “templates/404.twig”. in /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/twig/TemplateLoader.php:116
Stack trace:
#0 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/twig/TemplateLoader.php(72): craft\web\twig\TemplateLoader->_resolveTemplate('templates/404.t...')
#1 /var/www/html/staging.oneworkplace.com/vendor/twig/twig/src/Environment.php(299): craft\web\twig\TemplateLoader->getCacheKey('templates/404.t...')
#2 /var/www/html/staging.oneworkplace.com/vendor/twig/twig/src/Environment.php(381): Twig\Environment->getTemplateClass('templates/404.t...')
#3 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/twig/Environment.php(41): Twig\Environment->loadTemplate('templates/404.t...', NULL)
#4 /var/www/html/staging.oneworkplace.com/vendor/twig/twig/src/Environment.php(359): craft\web\twig\Environment->loadTemplate('templates/404.t...')
#5 /var/www/html/staging.oneworkplace.com/vendor/twig/twig/src/Environment.php(318): Twig\Environment->load('templates/404.t...')
#6 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/View.php(389): Twig\Environment->render('templates/404.t...', Array)
#7 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/View.php(450): craft\web\View->renderTemplate('templates/404.t...', Array)
#8 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/Controller.php(241): craft\web\View->renderPageTemplate('templates/404.t...', Array, 'site')
#9 /var/www/html/staging.oneworkplace.com/vendor/dolphiq/redirect/src/controllers/RedirectController.php(116): craft\web\Controller->renderTemplate('templates/404.t...', Array)
#10 [internal function]: dolphiq\redirect\controllers\RedirectController->actionIndex()
#11 /var/www/html/staging.oneworkplace.com/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#12 /var/www/html/staging.oneworkplace.com/vendor/yiisoft/yii2/base/Controller.php(180): yii\base\InlineAction->runWithParams(Array)
#13 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/Controller.php(190): yii\base\Controller->runAction('index', Array)
#14 /var/www/html/staging.oneworkplace.com/vendor/yiisoft/yii2/base/Module.php(528): craft\web\Controller->runAction('index', Array)
#15 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/Application.php(274): yii\base\Module->runAction('redirect/redire...', Array)
#16 /var/www/html/staging.oneworkplace.com/vendor/yiisoft/yii2/web/Application.php(103): craft\web\Application->runAction('redirect/redire...', Array)
#17 /var/www/html/staging.oneworkplace.com/vendor/craftcms/cms/src/web/Application.php(259): yii\web\Application->handleRequest(Object(craft\web\Request))
#18 /var/www/html/staging.oneworkplace.com/vendor/yiisoft/yii2/base/Application.php(386): craft\web\Application->handleRequest(Object(craft\web\Request))
#19 /var/www/html/staging.oneworkplace.com/web/index.php(21): yii\base\Application->run()
#20 {main}
2021-12-03 16:09:47 [-][-][l40r7slv4ue3d52bnkti1l12b2][info][application] $_GET = []

$_POST = []

$_FILES = []

$_COOKIE = [
    'CraftSessionId' => 'l40r7slv4ue3d52bnkti1l12b2'
    'CRAFT_CSRF_TOKEN' => '•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••'
    '1031b8c41dfff97a311a7ac99863bdc5_username' => '7b0ae12da6b63ad94617571ae36fae785b0d8b453d0a50ca937a8d4e25532fd0a:2:{i:0;s:41:\"1031b8c41dfff97a311a7ac99863bdc5_username\";i:1;s:23:\"travis@objectiveinc.com\";}'
    '__stripe_mid' => '65ce8857-2932-438a-8542-d40e963e95d0f7752c'
]

$_SESSION = [
    'bd62416aa8538ede709019a5e113eea5__flash' => []
]

$_SERVER = [
    'USER' => 'ubuntu'
    'HOME' => '/home/ubuntu'
    'HTTP_COOKIE' => 'CraftSessionId=l40r7slv4ue3d52bnkti1l12b2; CRAFT_CSRF_TOKEN=b6326c9497df8ecb6f683fa6fb62662c0f8f76c5c2c65db33c0724eac981e9d5a%3A2%3A%7Bi%3A0%3Bs%3A16%3A%22CRAFT_CSRF_TOKEN%22%3Bi%3A1%3Bs%3A212%3A%22iBn4m9Q3smGheGBajATB2g-vnlTYH5zNZT2Atxfk%7C12a94f4d77247d427c8475094abc8c7f3f76d8b471850b0b301f08b0590e4f90iBn4m9Q3smGheGBajATB2g-vnlTYH5zNZT2Atxfk%7C16353%7C%242y%2413%241zrGJxz6QxdRnAJfHPWckuHE0vFXA83S82SNE5xfo9XL09S9MOj2K%22%3B%7D; 1031b8c41dfff97a311a7ac99863bdc5_username=7b0ae12da6b63ad94617571ae36fae785b0d8b453d0a50ca937a8d4e25532fd0a%3A2%3A%7Bi%3A0%3Bs%3A41%3A%221031b8c41dfff97a311a7ac99863bdc5_username%22%3Bi%3A1%3Bs%3A23%3A%22travis%40objectiveinc.com%22%3B%7D; __stripe_mid=65ce8857-2932-438a-8542-d40e963e95d0f7752c'
    'HTTP_ACCEPT_LANGUAGE' => 'en-US,en;q=0.9'
    'HTTP_ACCEPT_ENCODING' => 'gzip, deflate, br'
    'HTTP_SEC_FETCH_DEST' => 'document'
    'HTTP_SEC_FETCH_USER' => '?1'
    'HTTP_SEC_FETCH_MODE' => 'navigate'
    'HTTP_SEC_FETCH_SITE' => 'none'
    'HTTP_SEC_GPC' => '1'
    'HTTP_ACCEPT' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
    'HTTP_USER_AGENT' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36'
    'HTTP_UPGRADE_INSECURE_REQUESTS' => '1'
    'HTTP_CACHE_CONTROL' => 'max-age=0'
    'HTTP_HOST' => 'staging.oneworkplace.com'
    'PATH_INFO' => ''
    'SCRIPT_FILENAME' => '/var/www/html/staging.oneworkplace.com/web/index.php'
    'REDIRECT_STATUS' => '200'
    'SERVER_NAME' => 'staging.oneworkplace.com'
    'SERVER_PORT' => '443'
    'SERVER_ADDR' => '50.116.6.247'
    'REMOTE_PORT' => '50152'
    'REMOTE_ADDR' => '108.180.1.219'
    'SERVER_SOFTWARE' => 'nginx/1.18.0'
    'GATEWAY_INTERFACE' => 'CGI/1.1'
    'HTTPS' => 'on'
    'REQUEST_SCHEME' => 'https'
    'SERVER_PROTOCOL' => 'HTTP/2.0'
    'DOCUMENT_ROOT' => '/var/www/html/staging.oneworkplace.com/web'
    'DOCUMENT_URI' => '/index.php'
    'REQUEST_URI' => '/robots.txt'
    'SCRIPT_NAME' => '/index.php'
    'CONTENT_LENGTH' => ''
    'CONTENT_TYPE' => ''
    'REQUEST_METHOD' => 'GET'
    'QUERY_STRING' => ''
    'FCGI_ROLE' => 'RESPONDER'
    'PHP_SELF' => '/index.php'
    'REQUEST_TIME_FLOAT' => 1638576587.2647
    'REQUEST_TIME' => 1638576587
    'ENVIRONMENT' => 'staging'
    'SECURITY_KEY' => '••••••••••••••••••••••••••••••••'
    'DB_DRIVER' => '----'
    'DB_SERVER' => '----'
    'DB_USER' => '----'
    'DB_PASSWORD' => '••••••••••••••••••'
    'DB_DATABASE' => '----'
    'DB_SCHEMA' => 'public'
    'DB_TABLE_PREFIX' => ''
    'DB_PORT' => '----'
    'DEV_SITE_URL' => 'https://staging.oneworkplace.com'
    'ASSETS_BASE_URL' => 'https://oneworkplace.com'
]
khalwat commented 2 years ago

So SEOmatic doesn't appear anywhere in that stack trace -- instead, for whatever reason, your Redirects plugin is kicking in:

#9 /var/www/html/staging.oneworkplace.com/vendor/dolphiq/redirect/src/controllers/RedirectController.php(116): craft\web\Controller->renderTemplate('templates/404.t...', Array)

...and it's attempting to render a template that apparently doesn't exist, with the path templates/404.twig

So two take-aways here:

  1. I don't think SEOmatic is involved in any way with this error; it appears to be an issue in some way with the dolphiq/Redirects plugin
  2. Simply the /robots.txt file throwing a 404 error would not cause a diminishing in any Google SERP rankings

Should you end up needing it, I do make a redirects plugin called Retour as well: https://nystudio107.com/plugins/retour

If there's anything else I can help with, let me know.