search

etix / mirrorbits

Mirrorbits is a geographical download redirector written in Go for distributing files efficiently across a set of mirrors.
MIT License
497 stars 90 forks source link
cdn checksum cli distribution dockerfile ftp go http https ipv6 mirror open-source realtime rsync

readme

Build Status Go Report Card

Mirrorbits

Mirrorbits is a geographical download redirector written in Go for distributing files efficiently across a set of mirrors. It offers a simple and economic way to create a Content Delivery Network layer using a pure software stack. It is primarily designed for the distribution of large-scale Open-Source projects with a lot of traffic.

mirrorbits_screenshot

Main Features

Is it production ready?

Yes! Mirrorbits has served billions of files already and is known to be running in production at:

Yet some things might change before the 1.0 release. If you intend to deploy Mirrorbits in a production system it is advised to notify the author first so we can help you to make any transition as seamless as possible!

Previous projects which have used Mirrorbits:

Quick start

Prerequisites

:warning: GeoIP-legacy is not supported anymore, please use the new GeoIP2 mmdb databases!

Optional:

Upgrading

Before upgrading to the latest version, please check this guide.

Installation

You can either get a prebuilt version or choose to build it yourself.

Docker

A docker "quick start" can be found on the wiki.

Manual build

Go >= 1.11:

$ git clone https://github.com/etix/mirrorbits.git
$ cd mirrorbits
$ sudo make install

Go < 1.11:

$ go get -u github.com/etix/mirrorbits
$ cd $GOPATH/src/github.com/etix/mirrorbits
$ sudo make install

The resulting executable should now live in your /usr/local/bin directory. You can also specify a PREFIX or DESTDIR if necessary:

sudo make install PREFIX=/usr

Configuration

A sample configuration file can be found here.

Running

Mirrorbits is a self-contained application and can act, at the same time, as the server and the cli.

To run the server:

mirrorbits daemon

Additional options can be found with mirrorbits -help.

To run the cli:

mirrorbits help

Add a mirror:

mirrorbits add -ftp="ftp://ftp.mirrors.example/myproject/" -http="http://ftp.mirrors.example/myproject/" mirrors.example

Enable the mirror:

mirrorbits enable mirrors.example

Realtime file availability

By appending ?mirrorlist to any file served by mirrorbits, you'll be able to get some useful realtime informations about the given file. You can see a live example here.

Realtime mirrors statistics

Mirror statistics are available by querying mirrorbits with the ?mirrorstats argument. You can see a live example here.

Clustering / High availability

Multiple instances of mirrorbits can be started simultaneously on different servers, discovery of other nodes should be automatic as long as all the instances are connected to the same redis server. In addition to the clustering it is advised to use redis-sentinel to monitor the database and gracefully handle failover.

Upgrading

Mirrorbits has a mode called seamless binary upgrade to upgrade the server executable at runtime without service disruption. Once the binary has been replaced on the filesystem just issue the following command in the cli:

mirrorbits upgrade

Considerations

We're social!

The best place to discuss about mirrorbits is to join the #VideoLAN IRC channel on Libera.chat. For the latest news, you can follow @mirrorbits on Twitter.

License MIT

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.