PyBOMBs 镜像 (was: build-gnuradio 脚本镜像)

scateu commented 8 years ago

项目名称与简介(Project Intro.)

GNURadio在各大发行版的预编译包总是很慢，因此GNURadio的Wiki上建议使用~~build-gnuradio脚本~~PyBOMBs来安装。

另外，GNURadio还提供PyBombs(一种GNURadio专用的包管理机制)安装，也有可能进行镜像。

Detailed instructions are given in the pybombs readme, but the following worked for me:

Install pybombs

git clone https://github.com/gnuradio/pybombs.git
cd pybombs
python setup.py build
sudo python setup.py install

Set a prefix

sudo pybombs prefix init /usr/local -a globalprefix

Install GNURadio

sudo pybombs install gnuradio gr-osmosdr

上游地址与镜像方法(How to Mirror)

~~http://www.sbrac.org/files/build-gnuradio~~

这个脚本里所使用的git仓库地址，需要自动用sed修改到TUNA的git仓库地址。

之前cuckoo做过一个，不过他的git仓库已经挂了。

https://github.com/scateu/pybombs-mirror

其他信息(Other)

镜像大小(Mirror Size): (不造...)

我从build-gnuradio脚本里面找到了以下的仓库URL，注意，有一些还需要submodule处理

git://git.osmocom.org/rtl-sdr 
git://git.osmocom.org/gr-osmosdr 
git://git.osmocom.org/gr-iqbal.git    # 这个需要submodule update --init
https://github.com/EttusResearch/uhd 
https://github.com/Nuand/bladeRF.git 
https://github.com/mossmann/hackrf.git 
https://github.com/airspy/host
http://git.gnuradio.org/git/gnuradio.git
https://github.com/guruofquality/grextras
https://github.com/balint256/gr-baz.git

scateu commented 8 years ago

更新了URL列表.

bigeagle commented 8 years ago

有 submodule 就比较麻烦。

假设项目 A 包括 submodule B, C，于是我们镜像了 Am, Bm, Cm，但是 submodule 本身在 A 中的存放形式就是个链接，所以别人从 Am clone 的时候，链接仍然只想 B 而非 Bm，这个就比较麻烦了。

bigeagle commented 8 years ago

submodule 好像可以 http://stackoverflow.com/a/2168315 这样解决。

康哥有时间的话一起搞吧，这个我一个人搞不定。

scateu commented 8 years ago

好的，下次金枪鱼之夜之前/之后我去找你一起弄

不过我记得这里面的Submodule并不是特别多，貌似只有一两个

另外，我观察到最近大家用PyBOMBS来安装GNURadio的比较多。它也是一堆Recipe的集合

scateu commented 8 years ago

啊，有点头疼。

看了一圈，发现PyBombs只是一个类似于yaourt的东西，更大一部分都是一些Recipe:

而这些Recipes里面的仓库，有一部分是github，有一部分是来自各子项目自己的仓库。因此要把这些git repo都镜像下来成本会不会比较大？

不过，批量把Recipes里的repo先爬下来，再把Recipes里面的URL改掉，也是一个办法。

待我心情好的时候跟上游通信一下，看看上游有没有什么更优雅的办法...

Anyway，最根本的问题还是国内网络慢啊... 唉

scateu commented 8 years ago

另外，PyBombs是正确安装方式是从pip安装，而pip已经有镜像了。

所以我们应该做的只是那两个recipes仓库里涉及到所有仓库搞了。

scateu commented 8 years ago

我提取了下面两个Recipes仓库里的URL:

使用命令:

grep  -r -E "(git|svn|wget)\+.*$" * | rev |cut -d' '  -f1 |rev

输出:

git+https://github.com/garverp/gr-analysis
git+https://github.com/pavelyazev/gr-dect2
git+https://github.com/merbanan/rtl_433.git
git+https://github.com/gr-vt/gr-framers.git
git+https://github.com/osh/gr-pcap.git
git+https://github.com/ptrkrysik/gr-gsm.git
git+https://github.com/kit-cel/gr-lte.git
wget+http://download.oracle.com/berkeley-db/db-4.8.30.tar.gz
git+https://github.com/bistromath/gr-smartnet.git
git+https://github.com/patchvonbraun/simple_ra.git
git+https://github.com/bistromath/gr-rtty.git
wget+http://www.zeroc.com/download/Ice/3.5/Ice-3.5.1.tar.gz
git+https://github.com/argilo/gr-ham.git
git+https://github.com/gr-vt/gr-mapper.git
git+https://github.com/marcusmueller/measurement_toolbox.git
git+git://svn.berlin.ccc.de/airprobe
git+https://github.com/osh/gr-mediatools.git
git+https://github.com/bastibl/gr-keyfob.git
git+https://github.com/Tom-McDermott/gr-hpsdr.git
git+https://github.com/mutability/dump1090.git
git+https://github.com/amontefusco/gr-microtelecom
git+https://github.com/tkuester/gr-reveng.git
git+https://github.com/git-artes/gr-mer
git+https://github.com/argilo/gr-dsd.git
git+https://github.com/tkuester/gr-psk31.git
git+https://github.com/drmpeg/gr-dvbt2.git
git+https://github.com/bastibl/gr-ieee802-15-4.git
git+https://github.com/bastibl/gr-rstt.git
git+https://github.com/unsynchronized/gr-mixalot.git
git+https://github.com/osh/gr-uhdgps.git
git+https://github.com/patchvonbraun/gr-ra_blocks.git
git+https://github.com/mitul93/gr-smithchart.git
svn+https://www.cgran.org/svn/projects/gr-pocsag/trunk
git+https://github.com/martinleolehner/gr-streamsink.git
git+https://github.com/ckuethe/acarsdec.git
git+https://github.com/argilo/gr-elster.git
git+https://github.com/wnagele/gr-gpredict-doppler.git
git+https://github.com/argilo/gr-flarm.git
git+https://github.com/BogdanDIA/gr-dvbt.git
git+https://github.com/antoinet/gr-acars2.git
git+https://github.com/osh/gr-benchmark.git
git+http://github.com/ttsou/kalibrate-uhd
git+https://github.com/drmpeg/gr-dvbs2.git
git+https://github.com/git-artes/gr-isdbt
git+https://github.com/miek/inspectrum.git
git+https://github.com/ckuethe/gr-multimon.git
git+https://github.com/peterfillmore/GR_RCCar.git
git+http://github.com/gnss-sdr/gnss-sdr.git
wget+http://ftp.gnome.org/pub/gnome/sources/gdk-pixbuf/2.24/gdk-pixbuf-2.24.0.tar.bz2
git+https://github.com/EttusResearch/gr-ettus.git
git+https://github.com/gr-vt/gr-message_tools.git
git+https://github.com/manuts/gr-ldpc.git
git+https://github.com/osh/gr-theano.git
wget+http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz
wget+https://launchpad.net/ubuntu/+archive/primary/+files/pixman_0.18.4.orig.tar.gz
wget+https://download.qt.io/archive/qt/4.6/qt-everywhere-opensource-src-4.6.2.tar.gz
git+https://github.com/kit-cel/gr-radar.git
git+https://github.com/osh/gr-fsk-burst.git
wget+http://ftp.gnome.org/pub/gnome/sources/gtk+/2.21/gtk+-2.21.8.tar.bz2
wget+http://ftp.gnome.org/pub/gnome/sources/glib/2.27/glib-2.27.2.tar.bz2
wget+https://launchpad.net/ubuntu/+archive/primary/+files/bzip2_1.0.6.orig.tar.bz2
git+https://github.com/kit-cel/gr-specest.git
wget+http://downloads.sourceforge.net/project/itpp/itpp/4.3.1/itpp-4.3.1.tar.bz2
wget+http://download.zeromq.org/zeromq-3.2.4.tar.gz
git+https://github.com/airspy/host.git
git+https://github.com/iohannez/gr-zmqblocks.git
wget+http://prdownloads.sourceforge.net/numpy/numpy-1.6.2.tar.gz
wget+http://xmlsoft.org/sources/libxml2-2.7.2.tar.gz
wget+http://mirror.nexcess.net/gnu/gsl/gsl-1.13.tar.gz
wget+http://prdownloads.sourceforge.net/flex/flex-2.5.35.tar.bz2
git+git://git.osmocom.org/osmo-tetra.git
git+https://github.com/osh/grcompat.git
wget+http://ftp.gnu.org/gnu/make/make-3.82.tar.gz
wget+http://downloads.sourceforge.net/project/boost/boost/1.53.0/boost_1_53_0.tar.bz2
git+https://github.com/iZsh/pocsag-mrt.git
git+https://github.com/mossmann/hackrf.git
git+https://github.com/dl1ksv/gr-ax25.git
wget+http://files.ettus.com/binaries/niusrprio/niusrprio-installer-15.0.0.tar.gz
git+https://github.com/bistromath/gr-air-modes.git
wget+http://www.mega-nerd.com/libsndfile/files/libsndfile-1.0.25.tar.gz
wget+http://pkgs.fedoraproject.org/repo/pkgs/PyQt4/PyQt-x11-gpl-4.7.2.tar.gz/e7782e9146ec8aa0e76bcdb0ca5b9491/PyQt-x11-gpl-4.7.2.tar.gz
git+https://github.com/kit-cel/gr-drm.git
git+git://git.osmocom.org/libosmo-dsp
git+https://github.com/doxygen/doxygen.git
git+https://github.com/analogdevicesinc/libad9361-iio.git
wget+https://www.mirrorservice.org/sites/sourceware.org/pub/libffi/libffi-3.0.9.tar.gz
wget+https://gflags.googlecode.com/files/gflags-2.0.tar.gz
git+git://anonscm.debian.org/pcsclite/PCSC.git
git+git://git.osmocom.org/osmo-sdr.git
wget+http://www.fftw.org/fftw-3.3.2.tar.gz
git+https://github.com/adamgann/gr-tagutils.git
git+https://github.com/szechyjs/mbelib.git
wget+http://files.ettus.com/e4xx_images/e3xx-release-4/oecore-x86_64-armv7ahf-vfp-neon-toolchain-nodistro.0.sh
git+https://github.com/gr-vt/gr-burst.git
git+https://github.com/gr-vt/gr-mapper.git
wget+http://archive.ubuntu.com/ubuntu/pool/main/g/gmp/gmp_4.3.2+dfsg.orig.tar.gz
git+https://github.com/stwunsch/gr-nacl.git
git+https://github.com/bastibl/gr-foo
wget+http://www.mpfr.org/mpfr-3.0.0/mpfr-3.0.0.tar.gz
git+https://github.com/gnuradio/gnuradio.git
wget+http://www.netlib.org/lapack/lapack-3.3.0.tgz
git+git://git.osmocom.org/gr-osmosdr
wget+http://google-glog.googlecode.com/files/glog-0.3.3.tar.gz
wget+http://downloads.sourceforge.net/project/expat/expat/2.0.1/expat-2.0.1.tar.gz
wget+http://fontconfig.org/release/fontconfig-2.9.0.tar.gz
wget+http://sourceforge.net/projects/libusb/files/libusb-1.0/libusb-1.0.9/libusb-1.0.9.tar.bz2
git+https://github.com/Nuand/bladeRF.git
wget+http://ftp.gnome.org/pub/GNOME/sources/gobject-introspection/0.10/gobject-introspection-0.10.2.tar.bz2
wget+http://ftp.gnome.org/pub/GNOME/sources/pygtk/2.17/pygtk-2.17.0.tar.bz2
git+git://git.osmocom.org/rtl-sdr
git+https://github.com/greatscottgadgets/gr-bluetooth.git
wget+http://downloads.sourceforge.net/project/scipy/scipy/0.8.0/scipy-0.8.0.tar.gz
git+https://github.com/kantooon/gqrx_fork.git
git+https://github.com/signal11/hidapi.git
wget+http://prdownloads.sourceforge.net/log4cpp/log4cpp-1.1.1.tar.gz
git+https://github.com/csete/gqrx.git
git+git://anonscm.debian.org/pkg-samba/talloc.git
git+https://github.com/analogdevicesinc/gnuradio.git
wget+http://ftp.gnu.org/gnu/automake/automake-1.15.tar.gz
git+git://git.osmocom.org/libosmocore.git
git+https://github.com/anastas/gr-cdma.git
wget+http://prdownloads.sourceforge.net/cppunit/cppunit-1.12.1.tar.gz
git+https://github.com/osh/gr-psk-burst.git
git+https://github.com/osh/gr-eventstream.git
git+https://github.com/gnuradio/gr-tutorial.git
wget+http://xmlsoft.org/sources/libxslt-1.1.20.tar.gz
git+https://github.com/jmalsbury/gr-mac.git
git+https://github.com/balint256/gr-packetradio.git
wget+http://www.cmake.org/files/v2.8/cmake-2.8.6.tar.gz
wget+http://wiresharkdownloads.riverbed.com/wireshark/src/all-versions/wireshark-1.10.2.tar.bz2
git+https://github.com/balint256/gr-baz.git
wget+http://cairographics.org/releases/cairo-1.8.10.tar.gz
git+https://github.com/bg2bhc/gr-lilacsat.git
git+https://github.com/analogdevicesinc/libiio
git+git://git.code.sf.net/p/openlte/code
git+https://github.com/bistromath/gr-ais.git
wget+http://ftp.gnome.org/pub/gnome/sources/pango/1.28/pango-1.28.4.tar.bz2
wget+http://prdownloads.sourceforge.net/pyqwt/PyQwt-5.2.0.tar.gz
wget+http://www.tux.org/pub/sites/ftp.alsa-project.org/lib/alsa-lib-1.0.27.2.tar.bz2
wget+https://polarssl.org/download/polarssl-1.3.8-gpl.tgz
git+https://github.com/dl1ksv/gr-display.git
git+https://github.com/drmpeg/gr-paint.git
git+https://github.com/dl1ksv/gr-fcdproplus.git
wget+http://prdownloads.sourceforge.net/swig/swig-2.0.4.tar.gz
git+https://github.com/EttusResearch/uhd.git
git+https://github.com/jedisct1/libsodium.git
wget+http://pkgconfig.freedesktop.org/releases/pkg-config-0.28.tar.gz
wget+http://www.netgull.com/gcc/releases/gcc-4.5.2/gcc-4.5.2.tar.bz2
git+https://github.com/glfw/glfw
wget+http://cairographics.org/releases/py2cairo-1.8.10.tar.gz
wget+http://ftp.gnome.org/pub/gnome/sources/atk/1.30/atk-1.30.0.tar.bz2
svn+http://googletest.googlecode.com/svn/trunk/
git+https://github.com/wnagele/gr-adsb.git
git+git://git.osmocom.org/gr-fosphor
git+https://github.com/amontefusco/libperseus-sdr
wget+http://ftp.gnu.org/gnu/bison/bison-2.5.tar.gz
wget+https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.gz
git+git://op25.osmocom.org/op25.git
wget+http://gperftools.googlecode.com/files/gperftools-2.0.tar.gz
git+https://github.com/greatscottgadgets/libbtbb.git
wget+http://aptproxy.willowgarage.com/archive.ubuntu.com/ubuntu/pool/universe/m/mcpp/mcpp_2.7.2.orig.tar.gz
git+https://github.com/ckuethe/gr-nmea.git
git+https://github.com/cheetahtemplate/cheetah.git
wget+http://prdownloads.sourceforge.net/cheetahtemplate/Cheetah-2.0.1.tar.gz
wget+http://ftp.gnome.org/pub/GNOME/sources/pygobject/2.27/pygobject-2.27.91.tar.gz
wget+http://pkgs.fedoraproject.org/repo/pkgs/sip/sip-4.10.1.tar.gz/9fa0b0d17ad355bde004317f67c819f9/sip-4.10.1.tar.gz
wget+https://launchpad.net/ubuntu/+archive/primary/+files/libpng_1.2.42.orig.tar.bz2
wget+https://pypi.python.org/packages/source/P/PySSTV/PySSTV-0.1.9.tar.gz
wget+download.savannah.gnu.org/releases/freetype/freetype-2.4.0.tar.bz2
wget+http://ftpmirror.gnu.org/libtool/libtool-2.4.2.tar.gz
git+https://github.com/apache/thrift.git
git+https://github.com/osh/gr-pyqt.git
wget+http://www.multiprecision.org/mpc/download/mpc-0.9.tar.gz
wget+http://www.tcpdump.org/release/libpcap-1.3.0.tar.gz
wget+http://prdownloads.sourceforge.net/qwt/qwt-5.2.0.tar.bz2
git+https://github.com/bastibl/gr-ieee802-11.git
git+git://git.osmocom.org/gr-iqbal
wget+http://prdownloads.sourceforge.net/wxpython/wxPython-src-2.8.11.0.tar.bz2
wget+http://lxml.de/files/lxml-2.3.3.tgz
git+https://github.com/bastibl/gr-rds.git
wget+http://sourceforge.net/projects/arma/files/armadillo-4.400.1.tar.gz
wget+https://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz

下面的想法是，把这个列表里的每个URL都加上mirrors.tuna.tsinghua.edu.cn/pybombs/的前缀:

例如:

wget+https://mirrors.tuna.tsinghua.edu.cn/pybombs/pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz

不过，还有一个问题，URL的类别有以下几种:

git+http
git+https
git+git
wget+http
wget+https
wget+
svn+https
svn+http

倒也可以把这些所有的协议全部给替换成wget+https，这样能够省服务器资源

@bigeagle 求评论

scateu commented 8 years ago

刚刚测试了一下，全部拖下来大约1.7G，花了半天的时间。

另外，PyBOMBs有时候会checkout不同的git分支....看来还是得提供git协议

scateu commented 8 years ago

啊，要不然我先弄一个travis-ci，参照emacs-china.org那样子？

bigeagle commented 8 years ago

要不给 tunasync-scripts 提个PR？

scateu commented 8 years ago

好，我先去读读其它的scripts好了

scateu commented 8 years ago

做了一个简单的脚本。

https://github.com/scateu/pybombs-mirror

求建议。 @bigeagle

bigeagle commented 8 years ago

好棒！

scateu commented 8 years ago

Done!

tuna / issues