-
Firefox (61+) sometimes (often first login after restarting the browser) redirects the user to /robots.txt instead of the storefront.
![grafik](https://user-images.githubusercontent.com/3968629/452…
-
Hi,
Just testing wp2static 6.6.2 on a freshly installed wp-5.0.3, using the "Subdirectory on same server" output option.
I expected the static home page to look the same as the original from WP.
…
-
I understand a RandomDelay was introduced recently. My c.Limit looks like:
`
c.Limit(&colly.LimitRule{
DomainGlob: "*",
Parallelism: 1000,
Delay: 1 * time.Second,
},
…
-
Hi Pascal,
I am seeing a lot of duplicates being processed. The number after the colon shows how many times i see it committed. This is all happening in the same run. How can I prevent this from ha…
-
Good Day!
I really love your gem and implemented it after we had our first outage due to an aggressive crawler.
The only thing I miss so far is an "easier whitelist" for yahoo and the well known goo…
-
M-x stops working after installing ivy, counsel and swiper through use-package. Here is the error-log
Debugger entered--Lisp error: (cyclic-function-indirection r-mode)
commandp(r-mode)
#f(c…
-
We should provide a "robots.txt" to prevent Google / Yahoo / Bing / Baidu / etc bots from crawling the entire server
-
> "Security.txt is the equivalent of robots.txt, but for security issues."
robots.txt is "A Standard for Robot Exclusion" or the "Robots Exclusion Protocol". A robots.txt file sets out the things t…
-
`wget` is a command used to download files from the internet. The following is part of its `man` page:
```
NAME
Wget - The non-interactive network downloader.
SYNOPSIS
wget [opti…
-
Possibly related to #2274; current robots.txt advises bots to not index the entire site (at least, judging by `wget -r ` default behavior and https://en.wikipedia.org/wiki/Robots_exclusion_standard).