Open anjackson opened 3 years ago
Avoid https://www.bl.uk/?mobile=on to prevent accidentally crawling the mobile version of www.bl.uk.
Just patch the regex block list bean. Document how it's done to the live crawler, ideally wrap as a h3cc script.
But see also #36
Avoid https://www.bl.uk/?mobile=on to prevent accidentally crawling the mobile version of www.bl.uk.
Just patch the regex block list bean. Document how it's done to the live crawler, ideally wrap as a h3cc script.
But see also #36