Stop adding Cache Control of 1 year for the Sitemap pages

[isaumya]

Currently a Cache-Control header of Cache-Control: max-age=31536000 is being added to the xml and xsl files by Rankmath, which basically tell the browser/CDN to cache the sitemaps, which is a really bad approach as sitemaps should never be cached as it contains always changing dynamic content.

If you want to keep it, don't set it by default instead give an option to the user asking if they want to add cache control to the sitemaps or not. Moreover adding this cache-control via PHP prevents users to overwrite it via server rule like nginx.conf as Nginx will add the header first and then PHP will overwrite it.

The cache-control is being added from /seo-by-rank-math/includes/modules/sitemap/abstract-xml.php line no 48 from this function:

protected function send_headers( $headers = [] ) {
  $expires  = gmdate( 'D, d M Y H:i:s', ( time() + YEAR_IN_SECONDS ) );
  $defaults = [
    'X-Robots-Tag'  => 'noindex',
    'Content-Type'  => 'text/xml; charset=' . $this->get_output_charset(),
    'Pragma'        => 'public',
    'Cache-Control' => 'maxage=' . YEAR_IN_SECONDS, // <-- THIS LINE
    'Expires'       => $expires . ' GMT',
    'Etag'          => md5( $expires . $this->type ),
  ];

  $headers = wp_parse_args( $headers, $defaults );

  header( $this->get_protocol() . ' 200 OK', true, 200 );

  foreach ( $headers as $header => $value ) {
    header( $header . ': ' . $value );
  }
}

Please stop adding cache-control by default. The plugin should never add the cache-control and if any user needs they can add the respective htaccess or nginx rules for that in the server.

[jcatello]

I agree cache-control should not be set within the plugin

[isaumya]

@surajv thanks man for closing this ticket. You should also consider adding the same cache-control that is added in sitemaps is also gets added to the robots.txt generated by Rankmath when there no actual robots.txt file exists. As it is generated by PHP the same thing applies there.

[balazsrm]

Hi @isaumya The virtual robots.txt is handled by WordPress core, and Rank Math just hooks into the robots_txt filter to change the contents.

If you want to add an extra header there, you could just use the do_robots action hook or the robots_txt filter hook for that.

Hope that helps.

[isaumya]

Thanks a lot, @balazsmts for showing the way. But I'm just a bit confused after checking the Source of do_robots is that should I pass the custom cache-control header on do_robots or `do_robotstxt? 🤔 Any thoughts?

Also, does Rankmath has any filter to check if it is generating the Robots.txt dynamically or is it coming from a file?

[balazsrm]

@isaumya the header could be added like this:

add_action( 'do_robots', function() {
    header( 'Cache-Control: no-cache, no-store, must-revalidate' );
    header( 'Pragma: no-cache' );
    header( 'Expires: 0' );
    return;
}, 9 );

To check if the robots.txt exists or not, you can just use the PHP function file_exists(), or the RankMath\Robots_Txt\Robots_Txt::get_robots_data() function: the exists item in the returned array shows if the file exists or not:

$robots_txt_file_exists = RankMath\Robots_Txt\Robots_Txt::get_robots_data()['exists'];

Hope that helps.