Hope that users can be given the option to more precisely control crawling behavior

CWHer / PixivCrawler

Pixiv Utils implemented in Python, including Pixiv Crawler and Mosaic Puzzles, support for rankings, personal bookmarks, artist works and keyword search for personalized filtering, and provide high-performance multi-threaded parallel download. 🤗

GNU General Public License v3.0

223 stars 28 forks source link

Hope that users can be given the option to more precisely control crawling behavior #21

Open nonhana opened 3 months ago

nonhana commented 3 months ago

[ ] It is hoped that when crawling works, information such as the title of the work, the name of the author, and the avatar can be obtained at the same time.
[x] Some users using an external network environment may be given the option to control whether to enable the proxy.

CWHer commented 3 months ago

Thanks for opening this issue and sorry for the delayed reply.

You are welcome to drop a PR if you would like to help enhance it. Besides, it would be better to briefly introduce your PR design here before hands-on it.

CWHer commented 3 months ago

As for these two issues, existing APIs can partially resolve them.

By enabling download_config.with_tag, tags of artworks, including avatar description, will be collected. However, it seems the title and author information are missing. I believe expanding with_tag option into with_metadata option could be a elegant solution.
Proxy can be disabled by setting network_config.proxy["https"] = "", but this still lacks some flexibility for more complicated scenarios.