-
crawlers can't find any of the content currently
-
(WEB REPORT BY: magechaos REMOTE: 172.93.109.202:7777)
# Revision
e73421e2414a1f6a925861ec7e7ab62c47e76f63
# Description
Pipes are invisible when crawling through vents after a certain distance.
…
-
Can sometimes result in a "too many requests" error. Should change `/data/web/nginx/http.conn_ratelimit` to the following content (courtesy of @JeroenVanLeusden):
```
map $remote_addr $conn_limit_ma…
-
### Context
Prior reading: https://anjackson.net/2023/06/09/what-makes-a-large-website-large/
> The simplest way to deal with this risk of temporal incoherence is to have two crawls. A shallow a…
-
We currently extract the text content in Python using the Justext library. We need something similar implemented in (ideally) Rust or Javascript. The Rust should compile to WASM so we can use it in a …
-
I am a big fan of pybuilder.
I am using Scrapy web crawling framework in my project which has its own directory structure, could you please let me know if there is any plugin for integration.
Curren…
-
I would love to create a gpt out of a github repo. Can you please add this?
K thx bai
-
See accompanying Twitter thread: https://twitter.com/simonw/status/1424820203603431439
> Datasette currently has a plugin for configuring robots.txt, but I'm beginning to think it should be part of…
-
It's very nice if I can browse the crawling web inside ACHE (localhost:8080) for debugging purpose. Because sometime pages ACHE got are different with pages I get in my browser.
Also, it would save…
-
1. 갓겜판독기를 만들꺼냐 or 똥겜판독기를 만들꺼냐
- 일단은 갓겜판독기 방면으로
2. 객관적인 지표를 가지고 먼저 판단하기
- 플레이스토어 평점, 다운로드 수, 매출순위, 리뷰 수 긁어오기
- 유용하다고 평가한 리뷰 싹 다 긁어오기
- 유용하다고 생각한 리뷰 수
3. 공통기능 정의
- 로그인, 로그아웃
- …