-
The only parts of Scrapy that we take advantage of are the scheduler and the downloader. Its management of crawlers and spiders doesn't add anything to our usecase, and the abstractions provided add c…
-
Research celery to off load scrapy work load off of django.
-
As identified in the SitemapXML forum support-thread, search engines can't crawl a given site if the above setting exists.
> Recently, Googlebot crawls without cookie.
> If he is force to use co…
-
Imported from JIRA [DS-3673] created by bram
We have been working together with COUNTER to move the management of their botlist to Github. The result is available at: https://github.com/atmire/COUNTER…
-
I believe that being able to register docker instances without exported ports (suitably tagged) should be one of the things that registrator supports.
Use cases for such dockers would be portless do…
-
I have a brief question: I have been looking for some type of reliable method which would allow specific scrapy spiders to be assigned to a "realtime" style method of execution. More specifically: I o…
-
**请描述该需求尝试解决的问题**
Hello,
I'd like to suggest to improve git sync functionality in order to make it possible for scenarios where there are dozens (or even hundreds) of spiders. Currently the function…
-
```
Python 3.9.13
Daphne 4.0.0
Django 4.1.2
Channels 4.0.0
Scrapy 2.7.0
scrapy-playwright 0.0.22
```
My settings:
```python
DOWNLOAD_HANDLERS = {
"http": "scrapy_playwright.handler.Sc…
-
item.getSupportedProjectile method returns null for LMG, causing the game to crash with Supplementaries.
---- Minecraft Crash Report ----
// Don't do that.
Time: 2024-11-10 10:29:24
Descriptio…
-
The theme for LD56 is Tiny Creatures.
- Parallelism
- 5 billion fleas
- Breeding Game --> Biological Horrors
- Insect/Tiny Creature collection to perform tasks
- Evolution through collecting sma…