-
```
What steps will reproduce the problem?
1. Find a website where robots.txt has something similar to
User-agent: *
Crawl-delay: 80
2. Run the crawler with a parser
What is the expected output? What…
-
Hi All,
I am a little bit lost I am using the following rule :
rule=:%clientip:word% %ident:word% %auth:word% [%timestamp:char-to:]%] "%verb:word% %request:word% HTTP/%httpversion:float%" %respo…
-
Hi,
Using the scheduler "Web crawler - URL database"
2 tasks in my scheduler
1 - I delete selected URL with fetch status = Gone
2 - I synchronize URL with Fetch status = Fetch
=> this truncate the …
-
Due to the following code, disallow rules that include query parameter expressions have no effect on the meta tags and headers for pages they target:
https://github.com/nuxt-modules/robots/blob/mai…
-
Search engine allow an asterisk (*) to match any sequence of characters, and a dollar sign ($) to match the end of the URL. So, to block spiders from downloading any JPEG image files, one might use:
…
-
The following are roughy in order of my interest level
- https://hackage.haskell.org/package/xlsx-0.7.2/docs/Codec-Xlsx-Lens.html
- https://hackage.haskell.org/package/giphy-api-0.6.0.1/docs/Web-G…
-
ArcParser raises exception when it encounters malformatted record:
```
Traceback (most recent call last):
File "../../fixarc.py", line 30, in
offset, record, errors = a._read_record(True)
Fi…
-
```
What steps will reproduce the problem?
1. Find a website where robots.txt has something similar to
User-agent: *
Crawl-delay: 80
2. Run the crawler with a parser
What is the expected output? What…
-
```
What steps will reproduce the problem?
1. Find a website where robots.txt has something similar to
User-agent: *
Crawl-delay: 80
2. Run the crawler with a parser
What is the expected output? What…
-
```
What steps will reproduce the problem?
1. Find a website where robots.txt has something similar to
User-agent: *
Crawl-delay: 80
2. Run the crawler with a parser
What is the expected output? What…