-
**Is your feature request related to a problem? Please describe.**
bots on the internet should honor the robots.txt (see [RFC 9309](https://datatracker.ietf.org/doc/rfc9309/)
**Describe the so…
-
(excerpted from #1624 / #1626)
-
### Terms
- [X] I'm using the very latest version of ChestPreview and its dependencies.
- [X] I already searched on this [Github page](https://github.com/PluginBugs/Issues-ChestPreview/issues) to che…
-
## 향후 개선점
- 크롤링 항목 추가: 본문, 글쓴이 ID 등
- 크롤링 결과의 별도 파일 저장
- 더 많은 종목 리스트: ex. 코스피200, 상장사 전체 등
- 더 많은 페이지 수: ex. 특정 기간까지, 혹은 마지막 페이지까지 등
- 크롤링 양이 많아질 시 진행률 표시
## 기타
- `Redefining name…
-
Here's a screenshot of the problem:
-
**Describe the bug**
`robots.txt` file is set to disallow crawling while we're actively developing. Need to fix for production
-
-When crawling one of the arms start spinning(Not enought animation incompat?)
-When using bow/crossbow/shield/trident the animation is really really slow, and the crossbow doesn't rise with the arm
…
-
If everything fails, we can go with beatiful soup crawling, see: https://github.com/OmnesRes/prepub
olegs updated
4 years ago
-
At least this one
https://ipinfo.io/AS45102/47.76.0.0/17
has been crawling, ignoring robots.txt, and using a non-bot user agent header :angry:
-
Crawlers require a container to be run, and running a local crawl on a container would require a local directory to be mounted inside of the container, which would have to be done at runtime. I'm not …