Looks like rebench.dev is caching a lot of raw data files.
Though, they should only ever be generated when someone actually requested them.
So, it's not clear to me why this happens.
This PR adds three things to hopefully minimize this unnecessary load on the server:
mark links to data files as nofollow
add the data URLs to the disallowed URLs in robots.txt
ignore browser requests that indicate they are prefetch attempts
Looks like rebench.dev is caching a lot of raw data files. Though, they should only ever be generated when someone actually requested them. So, it's not clear to me why this happens.
This PR adds three things to hopefully minimize this unnecessary load on the server:
nofollowdataURLs to the disallowed URLs in robots.txt