webysther / packagist-mirror

๐Ÿ“ฆโœ‚๏ธ๐Ÿ“‹๐Ÿ“ฆ Create a mirror of packagist.org metadata for use locally with composer
https://packagist.org/mirrors
MIT License
195 stars 70 forks source link
composer composer-packages crawler mirror packagist packagist-mirror php

๐Ÿ“ฆ Packagist Mirror

docker pulls Minimum PHP Version Packagist Codecov Quality Score Mentioned in Awesome composer

โค๏ธ Recommended by packagist.org โค๏ธ

Announcement: Composer 2 is now available!

This mirror is for Composer 1; Composer 2 is very fast on its own. We will update to support the version 2 for those need solve the slow internet access or availability problem with the main repository.

A mirror for packagist.org that regularly caches packages from one or more main mirrors to add to a distributed package repository.

Mirror creation

If you're using PHP Composer, commands like create-project, require, update, remove are often used. When those commands are executed, Composer will download information from the packages that are needed also from dependent packages. The number of json files downloaded depends on the complexity of the packages which are going to be used. The further you are from the location of the packagist.org server, the more time is needed to download json files. By using a mirror, it will save you time when downloading json because the server location is closer.

โš™๏ธ How it works?

This project aims to create a local mirror with ease, allowing greater availability for companies/countries that want to use composer without depending on the infrastructure of third parties. It is also possible to create a public mirror to reduce the load on the main repository and better distribute requests around the world, helping make the packagist ecosystem faster as a whole!

When creating a mirror, you add a list of other mirrors to use for initial sync, which pulls all packages to your local machine. After the mirror is created and synced, the next runs will only pull updates. If any mirror fails to deliver a metadata file, the client will fallback to its configured main mirror, whether that be packagist.org or otherwise. If the client encounters an installation problem or loses connection to a mirror, it can return from where it stopped running.

Mirror creation

๐ŸŒŽ Packagist public metadata mirrors observatory around the world

๐Ÿ›ซ Amazing data mirrors used to download repositories metadata built using this recommended repository or another:

Lists are ordered by country and sync frequency.

Location Mirror Maintainer Github Sync Since
Brazil packagist.com.br Webysther main Continuously Q3'17
China php.cnpkg.org Eagle Wu fork Every minute Q3'18
China packagist.mirrors.sjtug.sjtu.edu.cn Shanghai Jiao Tong University fork Every hour Q2'19
Czech Republic packagist.hostuj.to HOSTUJ TO fork Every 5 minutes ๐Ÿ†•Q1'20
Finland packagist.fi Niko Granรถ fork Continuously ๐Ÿ†•Q2'20
Germany packagist.hesse.im Benjamin Hesse fork Every minute ๐Ÿ†•Q3'20
Germany composer.mg100.net Alex Gummenscheimer fork Every minute ๐Ÿ†•Q1'21
India packagist.in Varun Sridharan fork Every minute Q2'19
India packagist.vrkansagara.in Vallabh Kansagara fork Every 5 minutes Q4'19
Indonesia packagist.phpindonesia.id Indra Gunawan fork Every 30 seconds Q3'18
Indonesia packagist.ianmustafa.com Ian Mustafa fork Every 30 seconds Q3'19
Indonesia packagist.telkomuniversity.ac.id Telkom University fork Every 5 minutes ๐Ÿ†•Q1'20
Japan packagist.dev.studio-umi.jp Studio Umi fork Every minute ๐Ÿ†•Q1'20
Russia packagist.org.ru Konstantin Tarasov fork Every 15 minutes Q3'22
South Africa packagist.co.za SolidWorx fork Every 5 minutes Q3'18
South Korea packagist.kr PackagistKR fork Every minute Q3'18
Thailand packagist.mycools.in.th Jarak Kritkiattisak fork Every 5 minutes Q4'19
USA packagist-mirror.wmcloud.org Wikimedia fork Every 5 minutes Q3'18
Taiwan packagist.tw Peter fork Every 5 minutes ๐Ÿ†•Q2'20
Vietnam packagist.ondinh.net Long Nguyen main Every 5 minutes ๐Ÿ†•Q3'20

โš ๏ธ Not based on this source code:

Location Mirror Maintainer Github Sync Since
China mirrors.aliyun.com Aliyun Every 5 minutes
China mirrors.cloud.tencent.com Tecent Cloud Every day
Japan packagist.jp Hiraku forked Every 2 minutes Q4'14
Japan packagist.kawax.biz Kawax another Every hour Q4'18

๐Ÿ›‘ Not working as a mirror of packagist.org (checked at Q1'20):

Location Mirror Maintainer Github Reason At least
China mirrors.huaweicloud.com Huawei Cloud Outdated Q3'19
China packagist.phpcomposer.com Outdated Q4'19

If you know any new mirror based or not on this one, please create a issue or a pull request with the new data.

Check status page for health mirror's.

World Map

This map shows working mirrors from above at the country level. The colors represent the topology drawn below.

๐Ÿš€ Create your own mirror

Topology

๐Ÿ’กTip: use a machine with at least 2GB of RAM to avoid using the disk or swap space during sync.

โš ๏ธ When syncing from DATA_MIRROR or MAIN_MIRROR, your server encodes and decodes all packages as .gz files to save disk space. You may need to enable server-side decoding for legacy composer clients that ask for decompressed packages.

There are currently three supported methods for creating your own mirror.

In all three methods, you need to clone the repository and copy .env.example to .env and modify to include your values instead of the defaults.

# Clone this repository
$ git clone https://github.com/websyther/packagist-mirror.git

# Setup environment variables
$ cd packagist-mirror
$ cp .env.example .env
$ nano .env

Docker Compose

Run the following commands to start a container for Nginx, PHP-FPM, and a worker that runs cron jobs.

# Start all Docker containers
$ docker-compose up -d

# Follow log output
$ docker-compose logs -f

Once the initial sync has finished, open https://localhost:9248 to see your site.

๐Ÿ’กTip: Add -f docker-compose.prod.yml between docker-compose and up or down while running the above commands. If you are using traefik, the services in this docker-compose file contain labels used by a running traefik container to automatically route traffic matching those labels to that container. It even auto-renews LetsEncrypt certificates for you.

Docker Nginx PHP

First, add the following line to /etc/crontab to tell the host to start a container for the packagist-mirror image on boot, replacing the values for each -e flag with your own. This will start the initial sync and generate the website files to be served by nginx.

Learn about more the available options for this docker image here.

* * * * * root docker run --name mirror --rm -v /var/www:/public \
-e MAINTAINER_REPO='packagist.com.br' \
-e APP_COUNTRY_NAME='Brazil' \
-e APP_COUNTRY_CODE='br' \
-e MAINTAINER_MIRROR='Webysther' \
-e MAINTAINER_PROFILE='https://github.com/Webysther' \
-e MAINTAINER_REPO='https://github.com/Webysther/packagist-mirror' \
-e URL='packagist.com.br' \
webysther/packagist-mirror

Next, add the following to /etc/nginx/sites-available/packagist.com.br.conf to host the website files:

server {
    index.html;

    server_name packagist.com.br www.packagist.com.br;

    location / {
        try_files $uri $uri/ =404;
        gzip_static on;
        gunzip on;
    }
}

To monitor sync progress, run the following command:

docker logs --follow --timestamps --tail 10 mirror

Nginx PHP

After cloning the repository, run the following commands to configure for your host.

$ cd packagist-mirror && composer install
$ cp .env.example .env

Then, schedule the command to create and update the mirror:

$ php bin/mirror create -vvv

Nginx will now serve your mirror at the configured URL.

๐Ÿง Development & Contributing

Please see CONTRIBUTING and CONDUCT for details.

๐Ÿ“‹ Requirements

The following versions of PHP are supported by this version.

๐Ÿงช Testing

$ vendor/bin/phpunit

๐Ÿฅ‚ Credits

๐Ÿ’™ Other correlated projects

โ˜ฎ๏ธ License

MIT License. Please see License File for more information.

FOSSA Status