search

bitcoin-dot-org / Bitcoin.org

Bitcoin.org Website
https://bitcoin.org/
Other
1.57k stars 2.03k forks source link

Where is the bin folder? #3801

Open melroy89 opened 2 years ago

melroy89 commented 2 years ago

Hi,

I'm running a full mirror, but I'm missing the bin folder (https://bitcoin.org/bin/) in this GitHub repo..

This folder contains the bitcoin core. Please help me to sync and mirror the whole site!
Which should include the bin folder as well.

What do you use? Are you using a script to sync the bin directory? The source files are currently stored: https://bitcoincore.org/bin/

Please help me.

Regards, Melroy van den Berg

melroy89 commented 2 years ago

@Cobra-Bitcoin I created a simple project that does exactly this.

This script will scrawl the bitcoin core web page (ONLY the bin folder), and download all the files. Save the files into the correct folders, with the correct names. So, basically you are creating a mirror for the bin folder.

See project: https://gitlab.melroy.org/bitcoin-dot-org/bitcoin-core-web-scraper 👍🏽

The source url that is used: https://bitcoincore.org/bin/. I'm using a simple Python framework called Scrapy to achieve my goals. Spider source code file.

EDIT: I also containerized this web crawler app: https://hub.docker.com/r/danger89/bitcoinscraper! 🥳

The Docker image will check the website once a week for changes automatically.
Or trigger manually from a running container (for example the first time): docker exec -it bitcoincore_scraper ./start_spider.py

Result: https://bitcoin.melroy.org/bin/