"do not match its signature"

Stef-33560 commented 3 years ago

Hi,

I've mirrored a packagist repo sucessfully

$php webysthe-packagist-mirror/bin/mirror create -vvv
Loading providers from https://packagist.hesse.im/
packages.json updated

My private NGINX server shows me on https://private.repo.lan/packages.json : { "packages": [], "notify-batch": "https:\/\/packagist.org\/downloads\/", "providers-url": "\/p\/%package%$%hash%.json", "search": "https:\/\/packagist.org\/search.json?q=%query%&type=%type%", "list": "https:\/\/packagist.org\/packages\/list.json", "providers-api": "https:\/\/packagist.org\/providers\/%package%.json", "warning": "You are using an outdated version of Composer. Composer 2 is now available and you should upgrade. See https:\/\/getcomposer.org\/2", "warning-versions": "<1.99", "provider-includes": { "p\/provider-2013$%hash%.json": { "sha256": "d0dc77fd03e5732f1bdb80037c61df42ed94764bddb5f24774761c59739791cc" }, "p\/provider-2014$%hash%.json": { "sha256": "ab43f689a426163c7bd97eee1359d66270cf969532a976b03d4d4e608395b25c" }, "p\/provider-2015$%hash%.json": { "sha256": "5dd99e47c377393321c91a2b7cede7c1dff20f23728c89dd551c20543750cf5e" }, "p\/provider-2016$%hash%.json": { "sha256": "7b8f1749c41d2c755e25992d763588fa527308b20f53fb15523c5a938d960bc4" }, "p\/provider-2017$%hash%.json": { "sha256": "62c94621ae4a052b1d7e00cb9398703ef09bc8f69fe8fce22e309a831b6011f9" }, "p\/provider-2018$%hash%.json": { "sha256": "df4e171e392f024639a924670d6c2a243a645277d43fb3b141cef9efe7b9df12" }, "p\/provider-2019$%hash%.json": { "sha256": "166db562306e9fd8967eb173a9dbf601c65ce4a2e67bf3d18c132730f1f1f6f9" }, "p\/provider-2020$%hash%.json": { "sha256": "ff7e5318ac974ec8ce64a1d0232f038f9fbac3d987a2146719ff31c6fc578d40" }, "p\/provider-2020-07$%hash%.json": { "sha256": "22517059027c992f913d337d6b79f67b1f3305c1d969e99bfd8ec859e54d9866" }, "p\/provider-2020-10$%hash%.json": { "sha256": "9d48effa58bf64e0764d894494902ffdee59924ad8f63fe1126b315601a2cb9b" }, "p\/provider-2021-01$%hash%.json": { "sha256": "0d37334fff077049a79655aa7f4ecd4ad7a4c01b66a4fe4b24c8aa6170ae5789" }, "p\/provider-2021-04$%hash%.json": { "sha256": "070fc16ec215d0422e7791c4fb0e1279d00704fecaac221863b56f0d68bb3514" }, "p\/provider-archived$%hash%.json": { "sha256": "60d5f896e444581ddd4baae5eaf7f04b63ec618d00f1d8ddf3af1fcf41264015" }, "p\/provider-latest$%hash%.json": { "sha256": "be8f9b5a0f6e46b14e976d6f56ae297509644bbd11916bd90d6894aea4946d27" } } }

But when requiring diffferent libs, I got each time an error ; example with drush

$ composer require drush/drush

Using version ^10.5 for drush/drush ./composer.json has been created Running composer update drush/drush Loading composer repositories with package information Installation failed, deleting ./composer.json. [Composer\Repository\RepositorySecurityException]
The contents of https://private.repo.lan/p/consolidation/site-process%24f510087614b0f459ed763ade5338c9bbade5e69b1b574df465e4e13105e11643.json do not match its signature. This could indicate a man-in-the-middle attack or e.g. antivirus software corrupting files. Try running composer again and report this if you think it is a mistake.
require [--dev] [--dry-run] [--prefer-source] [--prefer-dist] [--prefer-install PREFER-INSTALL] [--fixed] [--no-suggest] [--no-progress] [--no-update] [--no-install] [--no-scripts] [--update-no-dev] [-w|--update-with-dependencies] [-W|--update-with-all-dependencies] [--with-dependencies] [--with-all-dependencies] [--ignore-platform-req IGNORE-PLATFORM-REQ] [--ignore-platform-reqs] [--prefer-stable] [--prefer-lowest] [--sort-packages] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--apcu-autoloader-prefix APCU-AUTOLOADER-PREFIX] [--] []...

What's wrong with my mirror ?? Thanks a lot :)

peter279k commented 3 years ago

It seems that signature is not valid for original Composer mirror.

You may sync local mirror from original mirror again.

If it's not worked, I think you should consider using the composer/mirror.

Stef-33560 commented 3 years ago

Alas, it does not work :(

I've considered composer/mirror and even more your kickstarter script, but I do not have the same result :-/

With this script, I got this :

-rw-rw-r-- 1 me me 8307 juin 6 17:22 apple-touch-icon.png -rw-rw-r-- 1 me me 8307 juin 6 17:22 apple-touch-icon-precomposed.png -rw-rw-r-- 1 me me 20603 juin 6 17:22 favicon.ico -rw-rw-r-- 1 me me 14 juin 5 22:26 .gitignore -rw-rw-r-- 1 me me 9297 juin 7 22:33 index.html -rw-r--r-- 1 root root 20 juin 5 22:49 info.php -rw-rw-r-- 1 me me 14437 juin 6 17:22 logo.drawio -rw-rw-r-- 1 me me 52764 juin 6 17:22 logo.svg -rw-rw-r-- 1 me me 247878 juin 6 17:22 mirror-creation.gif -rw-rw-r-- 1 me me 16178 juin 6 17:22 network.drawio -rw-rw-r-- 1 me me 85986 juin 6 17:22 network.svg drwxr-xr-x 1 me me 2871296 juin 7 22:17 p/ lrwxrwxrwx 1 me me 16 juin 7 22:17 packages.json -> packages.json.gz -rw-rw-r-- 1 me me 973 juin 7 22:17 packages.json.gz -rw-rw-r-- 1 me me 106 juin 6 17:22 robots.txt -rw-rw-r-- 1 me me 8930 juin 6 17:22 touch-icon-192x192.png -rw-rw-r-- 1 me me 1106464 juin 6 17:22 world_map.svg

And with composer/mirror only this :

drwxr-xr-x 4 www-data www-data 4096 juin 8 21:53 ./ drwxr-xr-x 13 root root 4096 juin 7 22:41 ../ drwxrwxr-x 4 www-data www-data 4096 juin 8 08:19 mirror/ drwxrwxr-x 1 www-data www-data 2654208 juin 8 22:11 p2/

How did you achieve to get the other files on root folder to make the mirror usable ??

peter279k commented 3 years ago

Do you want to have the p1 folder?

The p1 folder is for the Composer V1 and p2 folder is for Composer V2.

If you've the p1 folder is presented, please change has_v1_mirror to be true.

More details about mirror setting can be found here.

Stef-33560 commented 3 years ago

Well, it seems that a packages.json on the root folder is mandatory to make it work throught a nginx vhost ? In any case using

"local": {
            "type": "composer",
            "url": "https://mirror-composer.lan",
        "options": {
        "ssl": {
                    "verify_peer": false,
                    "allow_self_signed": true,
                }
        }
        }

misses me this file !

Edit : Using a composer.json with

"repositories": {
    "folder": {
        "type": "path",
            "url": "/var/repos/mirror/p2"
    }
    }

works like a charm, but is quite inadapted for my needs (share an offlline repo for a drupal team offline from Internet, not on the same site)

peter279k commented 3 years ago

It seems that your issue is very similar with https://github.com/composer/mirror/issues/10.

peter279k commented 3 years ago

Just notice that the Composer mirror is not synced correctly if you miss the pckages.json file on the mirror root directory.

Here is the normal/correct captured picture:

bertploeger commented 3 years ago

I have the same problem. I dig into it and i found that some json.gz files are double-gzipped. gunzipping it twice produces the original json.

I am looking into this. Maybee it has something to do with Gzip.php and this code:


    public function isGzip(string $gzip):bool
    {
        if (mb_strpos($gzip, "\x1f"."\x8b"."\x08") === 0) {
            return true;
        }

        return false;
    }

When i find the issue i will report back.

bertploeger commented 3 years ago

I have the same problem. I dig into it and i found that some json.gz files are double-gzipped. gunzipping it twice produces the original json.

I am looking into this. Maybee it has something to do with Gzip.php and this code:
    public function isGzip(string $gzip):bool
    {
        if (mb_strpos($gzip, "\x1f"."\x8b"."\x08") === 0) {
            return true;
        }

        return false;
    }
When i find the issue i will report back.

I tested it with the specific double gzipped package.json file, in my case polyfill-php54$f9df1f5a1df7b1d216477f306bbeb226f4afb19b380cac394cf8456fea97127f.json.gz.

I gunzipped it once, then ran this test script:

$gzip = file_get_contents("php54$f9df1f5a1df7b1d216477f306bbeb226f4afb19b380cac394cf8456fea97127f.json");
if (mb_strpos($gzip, "\x1f"."\x8b"."\x08") === 0) {
    echo "gzip";
}else {
    echo "no gzip";
}

The output is "gzip", so it correctly detects whether a file is downloaded as gzip from the source mirror. I will dig deeper to see if there is any reason i can find in the code a file is double gzipped.

bertploeger commented 3 years ago

I have the same problem. I dig into it and i found that some json.gz files are double-gzipped. gunzipping it twice produces the original json. I am looking into this. Maybee it has something to do with Gzip.php and this code:
    public function isGzip(string $gzip):bool
    {
        if (mb_strpos($gzip, "\x1f"."\x8b"."\x08") === 0) {
            return true;
        }

        return false;
    }
When i find the issue i will report back.
I tested it with the specific double gzipped package.json file, in my case polyfill-php54$f9df1f5a1df7b1d216477f306bbeb226f4afb19b380cac394cf8456fea97127f.json.gz.

I gunzipped it once, then ran this test script:
$gzip = file_get_contents("php54$f9df1f5a1df7b1d216477f306bbeb226f4afb19b380cac394cf8456fea97127f.json");
if (mb_strpos($gzip, "\x1f"."\x8b"."\x08") === 0) {
    echo "gzip";
}else {
    echo "no gzip";
}
The output is "gzip", so it correctly detects whether a file is downloaded as gzip from the source mirror. I will dig deeper to see if there is any reason i can find in the code a file is double gzipped.

Strange thing was that half an hour later php54$f9df1f5a1df7b1d216477f306bbeb226f4afb19b380cac394cf8456fea97127f.json.gz was nog longer available and replaced with another hash after the $ sign. Maybe files are double gzipped at the source?

Stef-33560 commented 3 years ago

@bertploeger I did not experienced double-zipped files... but may you check if my PR composer/mirror #11 solves the problem for you too ?

webysther commented 3 years ago

Hey @Stef-33560 you found the problem here? I will test the drush and tell you soon.

Stef-33560 commented 3 years ago

Hi @webysther

Still broken with your mirror script :(

git clone https://github.com/webysther/packagist-mirror
cat .env|grep -v '^$'|grep -v '^#'

APP_NAME='Packagist Mirror' APP_COUNTRY_NAME='France' APP_COUNTRY_CODE='fr' PUBLIC_DIR=./public SLEEP=300 MAINTAINER_MIRROR='Webysther' MAINTAINER_PROFILE='https://github.com/Webysther' MAINTAINER_REPO='https://github.com/Webysther/packagist-mirror' MAINTAINER_LICENSE='MIT License' MAIN_MIRROR=https://repo.packagist.org URI_PATTERN='p/%s$%s.json' TZ='Europe/Paris' DATA_MIRROR=https://packagist.fr/,https://packagist.co.za,https://mirrors.aliyun.com/composer,https://packagist.mirrors.sjtug.sjtu.edu.cn,https://php.cnpkg.org,https://mirrors.cloud.tencent.com/composer,https://packagist.jp,https://packagist.kr,https://packagist.com.br,https://packagist.phpindonesia.id,https://repo-eu-uk-1.packagist.org URL=packagist.lan SINCE="Q2'21" MAX_CONNECTIONS=15 GOOGLE_ANALYTICS_ID= GOOGLE_ANALYTICS_MAIN_ID='UA-58875124-2'

php bin/mirror create -vvv

some 404 but it seems going well, serving public dir is OK

mkdir test
composer init
composer config -g repos.packagist composer https://packagist.lan
composer require phpunit/phpunit

Installation failed, reverting ./composer.json and ./composer.lock to their original content. [Composer\Repository\RepositorySecurityException] The contents of https://packagist.lan/p/doctrine/instantiator%2443f853a801f01284cb5b4bb83b105da84bc535029cad74afa4690b6b9719088c.json do not match its signature. This could indicate a man-in-the-middle attack or e.g. antivirus software corrupting files. Try running composer again and report this if you think it is a mistake.

Stef-33560 commented 3 years ago

@bertploeger: RFC 1952 suggests testing only 2 first fields

These have the fixed values ID1 = 31 (0x1f, \037), ID2 = 139 (0x8b, \213), to identify the file as being in gzip format.

But that does not change a lot of things.

I've launched again a new sync at 11:30 PM UTC

composer require phpunit/phpunit

gives :(

The contents of https://packagist.lan/p/phpunit/php-timer%24ed59f9ff1b99c0f19b427ff838c7a1bb3f5935ceb34d33f04a72e5cda97d40f5.json

webysther / packagist-mirror

"do not match its signature" #153