When pasting URLs into Horseman to crawl a list of URLs, it seems to remove the trailing slash or the URLs. This leads to redirects during the crawl if the server has been configured to redirect requests without a trailing slash to a URL with a trailing slash.
workeffortwaste
commented
1 year ago
When pasting URLs into Horseman to crawl a list of URLs, it seems to remove the trailing slash or the URLs. This leads to redirects during the crawl if the server has been configured to redirect requests without a trailing slash to a URL with a trailing slash.