-
Hi there,
Did anyone manage to find out what the user agents are the TikTok crawlers are using? I want to block TikTok from crawling my website but I still want users to be able to visit it from th…
-
How can I control the crawling catagory so that it wouldn't give me unexpected branch?
Like I make Thai language as the root , It came back a lot of windows version items. And consum a lot of quer…
new4u updated
2 years ago
-
We shall consider using the Elastic stack as the following scenario shows:
- Beats: Data crawling and gathering
- Logstash: Data input
- Elasticsearch: Data indexing and querying
- Kibana: Data vi…
-
In #36, dedup logic was moved from spiders to db queries in pipeline. This means that if there is an article already in our database and the discover crawl meet the article link again, spiders would …
-
### versions
- SlimerJS: 0.10.2
- Firefox: 50
- Operating system: OSX 10.10.5
### Steps to reproduce the issue
It's randomly fails while crawling
### Actual results:
slimerjs: line 167: …
-
Hi,
I'm new to x-ray, and i'd like to great a wide crawling enviroment using it.
Currently, i'm trying to crawl a page - get a specific form in it, and resubmit the page.
How and is it possible?
Th…
-
### Problem statement
A lot of manual work and tuning goes into every single publisher that's currently maintained, and still requires constant monitoring if anything changes in the supported news …
-
I think PS might have some issues crawling through the multi-valued "nodes" element in the CVE "configurations" key. I recall that's why we had to switch to Python.
![image](https://user-images.gi…
-
To get started with support for Opt-Out on Italian-language websites, the following is needed:
* Text that's usually used for links to "Terms Of Service" or "Terms Of Use" or "Conditions Of Use" or…
-
When wandering around it feels like I'm crawling a bit - the running speed +10% feels about "natural" for the environment