search

Syransong / portfolio-2022

0 stars 0 forks source link

Missing robots.txt file #8

Open Syransong opened 1 year ago

Syransong commented 1 year ago

Description

Currently the site does not have a robots.txt file. Consequences of not having this file is that crawlers may be indexing pages that do not need or should not be indexed. In addition, it may prevent the website from being overloaded with requests.

The solution would be to create a robots.txt in the root of the website and adding rules. Some content that could be disallowed is media content and live project URLs.

Steps to Reproduce

  1. Navigate to a robots.txt validation and tester tool such as https://technicalseo.com/tools/robots-txt/
  2. Enter the URL, "https://karinasong.com"
  3. Select the "Googlebot" option under the User Agent dropdown
  4. Click the "Test" button

Expected Result

The validation tool is able to find the robots.txt file and there are no warnings or errors associated with the file.

Actual Result

There is a warning that states that the robots.txt file was not found.

Environments

This issue is present on all browsers across all environments.

Screenshots

Screen Shot 2023-03-20 at 8 13 26 PM

Syransong commented 1 year ago

preliminary robots.txt file added and tested with validator. passes validator test (200 ok response code)

Screen Shot 2023-03-20 at 10 54 47 PM

Syransong commented 1 year ago

Updated robots.txt, removed disallow for public/images