-
hi,
I use matomo/device-detector and it works well but I need to filter more UA for my own usage as bots, is there an easy way to do that and to avoid to repush this list on each device-detector upda…
-
At the moment we use some string to detect crawler from the user agent string:
https://github.com/pluginkollektiv/statify/blob/667518428b30b0522367fb2c955d1913e1ef672f/inc/class-statify-frontend.ph…
-
-
### SUMMARY
The Cybersecurity and Infrastructure Security Agency (CISA), the Federal Bureau of Investigation (FBI), the Multi-State Information Sharing and Analysis Center (MS-ISAC), and the Canadi…
-
```
How about a robots.txt parser that grabs & spiders through the disallow entries as well?
I've found some very interesting things through doing that, and sometimes the entries
are even comments! [d…
-
# Introduction
I’ve made this issue to add a general idea of what Dol Guldur and the mirkwood orcs are. And to add more stuff for it to the future, there might be a bit of a “big” dump, but hopefully…
-
how to pass Query Params with all Crawling links
Sample for this
https://example.com?locale=en
https://example.com/blog/details/1?locale=en
I want this, to handle crawling with multi languages…
-
I use scrapy 1.0.3 and can't discover how works CLOSESPIDER extesnion.
For command:
scrapy crawl domain_links --set=CLOSESPIDER_PAGECOUNT=1
is correctly one requst, but for two pages count:
scrapy cra…
-
**Describe the bug**
I pulled new image and now getting an error when reloading cache.
I attempted to manually rebuild the cache and getting the same error when running the cache:clear and cache:war…
-
### Symfony version(s) affected
6.2.4
### Description
The web developer toolbar fails to load with security-bundle 6.2.3.
>Uncaught PHP Exception Twig\Error\RuntimeError: "Neither the prop…