Open akolonin opened 4 years ago
Example: https://www.joom.com/robots.txt ... Disallow: */q.* ... not matched for https://www.joom.com/ru/search/q.xiaomi in https://github.com/aigents/aigents-java/blob/master/src/main/java/net/webstructor/cat/HttpFileReader.java#L211 and file is tried to get read with error 400
Example: https://www.joom.com/robots.txt ... Disallow: */q.* ... not matched for https://www.joom.com/ru/search/q.xiaomi in https://github.com/aigents/aigents-java/blob/master/src/main/java/net/webstructor/cat/HttpFileReader.java#L211 and file is tried to get read with error 400