-
probably we should port that little project from php to python/flask.
[wiki artice for ideas](https://github.com/pointhi/searx_stats/wiki/python-port)
#### Advantages
- there are various bugs inside …
-
Only for Loblaws, No Frills and Sobeys
- check how the crawler receives the raw string
- check how it is stored in db
- check if db stores it differently than before
-
Bestimmte Angaben in einer robots.txt und Meta-Tags können dazu führen, dass Suchmaschinen die Site oder Teile davon nicht erfassen. Und eine URL, die nicht erfasst wird, kann auch nicht gefunden werd…
-
I'm maybe being a little devil here, but is there a way of speeding up more aiomultiprocess with [stackless python](https://stackless.readthedocs.io/en/v3.7.3-slp/stackless-python.html) and the [stack…
-
I've downloaded other classes with no issue, except the following :
------------------------------------------------------
2020-05-17 13:43:59,120 - 3 - [Done] priority=C, ttl=3. crawl_lecture: cour…
-
I have just run your crawler trying to get smart contract source code from Etherscan, but received an error message. I noticed that the Etherscan is using Cloudflare for security purposes, making the …
-
Hello! I have a problem with
"Block 274900 crawled
Caught an error from Bitcoind RCP, Reconnecting and retrying...(1/10)
Block 275000 crawled" or
Caught an error from Bitcoind RCP, Reconnecting and …
-
```
Current behaviour:
The crawler now runs only on single-system configurations.
Desired behaviour:
The crawler must be able to run on multiple machines in parallel in a
transparent way for the us…
-
Like in Scrapy https://github.com/scrapy/scrapy/blob/c316ca45a5b1b19622c96049c9378d8c45adba60/scrapy/crawler.py#L255
We'd need to set up a communication method between the threads and the main thre…
-
Is there a way to "log in" the crawler with my credentials?
I assume it's doing web requests and I could just attach e.g. a cookie to the web request?
I know python very well, so a hint into the r…