Packages.json missing V2 update not working

NikoGrano commented 3 years ago

./mirror.php --resync -v

...----------------------------------------------------RRRRRRRRRRRR??????
Downloaded 602900 files

After this I see there is only p2 folder created without packages.json.

If I try run --v2 it just fails.

root@packagist:/usr/local/www/mirror # ./mirror.php --v2
Mirror v2 job failed at 2021-05-04 18:38:54
[Exception] Too many retries, could not update /p2/rebing/graphql-laravel~dev.json as the origin server returns an older file (1620109991, expected 1620153524)

Fatal error: Uncaught Exception: Too many retries, could not update /p2/rebing/graphql-laravel~dev.json as the origin server returns an older file (1620109991, expected 1620153524) in /usr/local/www/mirror/mirror.php:278
Stack trace:
#0 /usr/local/www/mirror/mirror.php(329): Mirror->{closure}(Object(Symfony\Component\HttpClient\Response\CurlResponse), Array)
#1 /usr/local/www/mirror/mirror.php(151): Mirror->downloadV2Files(Array)
#2 /usr/local/www/mirror/mirror.php(796): Mirror->syncV2()
#3 {main}
  thrown in /usr/local/www/mirror/mirror.php on line 278

Config

return [
    // directory where metadata files will get saved
    'target_dir' => __DIR__.'/public'/* TODO */,
    // user agent describing your mirror node, if possible include domain name of mirror, and a contact email address
    'user_agent' => 'xxxxxxxx'/* TODO Mirror for foo.com (mycontact@example.org) */,
    // source repository URL
    'repo_url' => 'https://repo.packagist.org',
    // source repository hostname (optional, will guess from repo_url)
    //'repo_hostname' => 'repo.packagist.org',
    // source API URL
    'api_url' => 'https://packagist.org',
    // how many times the script will run the mirroring step before exiting
    'iterations' => 120,
    // how many seconds to wait between mirror runs
    'iteration_interval' => 5,
    // set this to false if you do not run the --v1 mirror job, to ensure that the v2 will then take care of syncing packages.json
    'has_v1_mirror' => false,
];

System

root@packagist:/usr/local/www/mirror # php -v
PHP 7.4.18 (cli) (built: May  2 2021 01:15:47) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
------------------------------------------------------------
root@packagist:/usr/local/www/mirror # cat /etc/os-release
NAME=FreeBSD
VERSION=12.2-RELEASE
VERSION_ID=12.2
ID=freebsd
ANSI_COLOR="0;31"
PRETTY_NAME="FreeBSD 12.2-RELEASE"
CPE_NAME=cpe:/o:freebsd:freebsd:12.2

peter279k commented 3 years ago

This issue I've never suffered before. I get this issue #6 and issue #8, but it's resolved.

As far as I know, using the ./mirror.php --v2 is good enough to setup the Composer V2 mirror at most of usual moments.

The ./mirror.php --resync -v command is to force to sync all Composer mirror.

Maybe it can do following steps:

Removing all releated Composer mirror folders.
Running the ./mirror.php --v2 command.
If above command running result is not expected, running the ./mirror.php --resync -v.

NikoGrano commented 3 years ago

Well, I'll try purge the downloaded files... It just takes with ZFS ages to complete 😓

NikoGrano commented 3 years ago

12 hours later, 10M inodes purged, lot of inodes rebuilt and it magically created the packages.json...

I decided to not support v1. It's anyways deprecated and probable going away in close future.

Stef-33560 commented 3 years ago

Hi,

Thanks @peter279k ;)

mirror.config.php

<?php

return [
    // directory where metadata files will get saved
    'target_dir' => '/var/repos/mirror',
    // user agent describing your mirror node, if possible include domain name of mirror, and a contact email address
    'user_agent' => 'Just testing mirror script',
    // source repository URL
    'repo_url' => 'https://repo.packagist.org',
    // source repository hostname (optional, will guess from repo_url)
    //'repo_hostname' => 'repo.packagist.org',
    // source API URL
    'api_url' => 'https://packagist.org',
    // how many times the script will run the mirroring step before exiting
    'iterations' => 120,
    // how many seconds to wait between mirror runs
    'iteration_interval' => 10,
    // set this to false if you do not run the --v1 mirror job, to ensure that the v2 will then take care of syncing packages.json
    'has_v1_mirror' => true,
];

System: $ php -v

PHP 7.4.3 (cli) (built: Oct  6 2020 15:47:56) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.3, Copyright (c), by Zend Technologies

$ cat /etc/os-release

NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Following steps done:

[x] all related Composer mirror folders removed (mv /var/repos/mirror/p2{,.old}).
[x] ./mirror.php --v2 -v. FIrst time :

[Exception] Too many retries, could not update /p2/orangesoft/retry ~ dev.json as the origin server returns an older file (1623049587, expected 1623230730) PHP Fatal error: Uncaught Exception: Too many retries, could not update /p2/orangesoft/retry~dev.json as the origin server returns an older file (1623049587, expected 1623230730) in /var/repos/mirror-composer/mirror.php:278

2nd :

[Exception] Too many retries, could not update /p2/discourse/wp-discourse ~ dev.json as the origin server returns an older file (1622833635, expected 1623230778) PHP Fatal error: Uncaught Exception: Too many retries, could not update /p2/discourse/wp-discourse~dev.json as the origin server returns an older file (1622833635, expected 1623230778) in /var/repos/mirror-composer/mirror.php:278

==> change iteration_interval' to 20 :(

Exception] Too many retries, could not update /p2/myfatoorah/omnipay.json as the origin server returns an older file (1623230171, expected 1623230877) PHP Fatal error: Uncaught Exception: Too many retries, could not update /p2/myfatoorah/omnipay.json as the origin server returns an older file (1623230171, expected 1623230877) in /var/repos/mirror-composer/mirror.php:278

==> Switch to https://packagist.fr/ HTTP404 Not Found returned for "https://packagist.fr/metadata/changes.json?since=16232308555191". ==> Switch to https://packagist.fi/ HTTP404 returned for "https://packagist.fi/metadata/changes.json?since=16232308555191". ==> Switch to https://repo.packagist.org/ HTTP200 https://repo.packagist.org/metadata/changes.json?since=16232308555191 BUT

[Exception] Too many retries, could not update /p2/pdir/convert-to-bundle.json as the origin server returns an older file (1615280716, expected 1623231541) PHP Fatal error: Uncaught Exception: Too many retries, could not update /p2/pdir/convert-to-bundle.json as the origin server returns an older file (1615280716, expected 1623231541) in /var/repos/mirror-composer/mirror.php:278

[x] Running the ./mirror.php --resync -v.

lots of MMMMMMMMM and some --

Downloaded 611091 files

and still no package.json nor packages.json.gz on mirror's root :(

drwxrwxr-x 1 me me 2842624 juin 9 12:18 p2/ drwxrwxr-x 1 me me 2654208 juin 8 23:51 p2.old/

peter279k commented 3 years ago

Sorry. I cannot help you because I cannot reproduce your issue.

I follow your steps to create Composer mirror and it doesn't have any error.

I guess that the Network bandwidth is not good enough and it causes syncing mirror is not done correctly.

Stef-33560 commented 3 years ago

I give another try

rm -Rf /var/repos/mirror
./mirror.php --v2 -v

1st time :

could not update /p2/symfony/mime~dev.json as the origin server returns an older file (1623236285, expected 1623236315)

2nd time

Downloaded 920 files and finally

could not update /p2/symfony/symfony ~ dev.json as the origin server returns an older file (1623236300, expected 1623236338)

3rd

/p2/laravel-enso/addressesmanager ~ dev.json as the origin server returns an older file (1623231058, expected 1623236348)

4th

Downloaded 10 files
No work
No work
MM
Downloaded 2 files
M
Downloaded 1 files
MM
Downloaded 2 files
No work
MM
Downloaded 2 files
MM
Downloaded 2 files
No work
MM
Downloaded 2 files

and finally

could not update /p2/bearlord/esd-yii.json as the origin server returns an older file (1607506142, expected 1623236422)

OK let's brute force :-1:

while [ $? != 0 ]; do ./mirror.php --v2 -v; done gave lots of stack trace but 4 hours later still no packages.json :(

Stef-33560 commented 3 years ago

Sorry. I cannot help you because I cannot reproduce your issue.

I follow your steps to create Composer mirror and it doesn't have any error.

I guess that the Network bandwidth is the good enough and it causes syncing mirror is not done correctly.

Tested from a VPS @ OVH :-1: :(

$ ./mirror.php --v2 -v

MMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMMRMRMRMRMRRRRRRRRRRRMirror v2 job failed at 2021-06-09 16:30:12 [Exception] Too many retries, could not update /p2/lullabot/drupal-symfony-lock ~ dev.json as the origin server returns an older file (1607990824, expected 1623256201) PHP Fatal error: Uncaught Exception: Too many retries, could not update /p2/lullabot/drupal-symfony-lock~dev.json as the origin server returns an older file (1607990824, expected 1623256201) in /home/steph/mirror-composer/mirror.php:278 Stack trace:

0 /home/steph/mirror-composer/mirror.php(383): Mirror->{closure}(Object(Symfony\Component\HttpClient\Response\CurlResponse), Array)

1 /home/steph/mirror-composer/mirror.php(151): Mirror->downloadV2Files(Array)

2 /home/steph/mirror-composer/mirror.php(796): Mirror->syncV2()

3 {main}

thrown in /home/steph/mirror-composer/mirror.php on line 278

Fatal error: Uncaught Exception: Too many retries, could not update /p2/lullabot/drupal-symfony-lock~dev.json as the origin server returns an older file (1607990824, expected 1623256201) in /home/steph/mirror-composer/mirror.php:278 Stack trace:

0 /home/steph/mirror-composer/mirror.php(383): Mirror->{closure}(Object(Symfony\Component\HttpClient\Response\CurlResponse), Array)

1 /home/steph/mirror-composer/mirror.php(151): Mirror->downloadV2Files(Array)

2 /home/steph/mirror-composer/mirror.php(796): Mirror->syncV2()

3 {main}

thrown in /home/steph/mirror-composer/mirror.php on line 278

Relaunched ... new error elsewhere

peter279k commented 3 years ago

Just share my experiences.

I do same steps about running ./mirror.php --v2 -v command on different hosts that provided by different host providers and I got different results.

And one of host that I need to run the ./mirror.php --resync -v command to be forced to sync Composer mirror.

Stef-33560 commented 3 years ago

Shall we hence reopen the issue ?

peter279k commented 3 years ago

I think it doesn't have to reopen this issue because it's out of scope for this repository.

Stef-33560 commented 3 years ago

I think I've found the workaround. See PR #11 :)

Stef-33560 commented 3 years ago

:frowning: I've alas understood why packages.json was missing... 'has_v1_mirror' must be set to false ! :confounded:

Seen in mirror.php#L67, I didn't understood this param correctly

composer / mirror

Packages.json missing V2 update not working #10

0 /home/steph/mirror-composer/mirror.php(383): Mirror->{closure}(Object(Symfony\Component\HttpClient\Response\CurlResponse), Array)

1 /home/steph/mirror-composer/mirror.php(151): Mirror->downloadV2Files(Array)

2 /home/steph/mirror-composer/mirror.php(796): Mirror->syncV2()

3 {main}

0 /home/steph/mirror-composer/mirror.php(383): Mirror->{closure}(Object(Symfony\Component\HttpClient\Response\CurlResponse), Array)

1 /home/steph/mirror-composer/mirror.php(151): Mirror->downloadV2Files(Array)

2 /home/steph/mirror-composer/mirror.php(796): Mirror->syncV2()

3 {main}