-
Currently, as a fail safe, `main.py` wraps the crawling code in an additional `while True` loop (as a form of accidental exception handling). We should add appropriate try catch blocks so that overarc…
-
Parent issue: #30
- FreePlus
- News #66
- Mini
-
The OEIS servers can generate this when the backend is bombarding it with too many requests. Instead of just bailing out when this error occurs, backscope should recognize the error and cool it for a …
-
### Does this bug already exist in our backlog?
- [X] I have checked and confirm this is a new bug.
### Describe the issue
We have made a comparison of the tenant data we get from Dataflows a…
-
Creating multiple repositories named "hello world" is worse than baby steps; like crawling. Get on with it and do something real @davidbukovec.
- [ ] Think of something interesting to do.
- [ ] Start …
-
Can we make an option for setting the max allowed request latency? Some of the sites I am crawling are responding very slow. Like 100000ms. I would like to be able to set an upper limit for this.
-
Hi!
Accordingly to https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap, correct ping url for google is `https://www.google.com/ping?sitemap=FULL_URL_OF_SITEMAP`, but th…
-
Tested without any other mods that affects players. I also noticed this is more likely to happen to face baubles.
![2021-12-30_00 41 10](https://user-images.githubusercontent.com/86805568/147727988-f…
-
Checking in Google Search Console, it looks like nothing's getting read from sitemaps, and it's all from crawling. I think it's resulting in a smaller index of ~10K pages, rather than the ~114K we sho…
-
python `django` 에서 정기적으로 함수를 실행하고 싶다.
해당 [블로그](https://yongbeomkim.github.io/django/dj-scheduler/)를 보면 python 함수를 정기적으로 실행할 수 있는 방법은
크게 3가지가 있는데
```
Celery 와 Redis 의 활용
리눅스의 CronTab 설정의 활용…