The amount of time composer takes to update json files from drupal-packagist is insane...

[generalredneck]

We may need to look into reducing the number of packages that are linked to by core drupal. That is where a ton of time is going when doing a composer update just because the machine has to hit up and query all those json files to figure out the state of the modules. However, drupal/drupal is the main package for drupal/user, drupal/node, drupal/menu... etc... but composer is still going through and making all of those connections.

A fully uncached grab of packages for a fairly simple site takes over 7 minutes just to update cache. This doesn't count the time it takes to actually download the packages. On slower connection (like some of my colleges that work in the Philippines), this takes around 20 minutes.

To reproduce what I'm seeing do this...

create a composer.json like the one here.

do a composer clear-cache and then time composer update --dry-run -vvv. You will see something along these lines:

https://gist.github.com/generalredneck/0145045bf4dcc8083de1

Granted if you do another dry run right away it takes ~30 seconds... but wait longer than that where provider-latest has updated... and you got a long wait in front of you. Especially if Core decides to update.

Can we get some brainstorming on how to help with this problem?

It's totally not connection issues... I did some timing using curl on packagist.drupal-composer.org vs packagist.org on packages that are shared... and packagist.drupal-composer.org is nearly a whole .1 secs faster on turn around. Granted I'm doing this from a 200Mb connection... but still..

Below are some pretty average times from the whole set.

curl -s -w '\nLookup time:\t%{time_namelookup}\nConnect time:\t%{time_connect}\nPreXfer time:\t%{time_pretransfer}\nStartXfer time:\t%{time_starttransfer}\n\nTotal time:\t%{time_total}\n' -o /dev/null https://packagist.drupal-composer.org/p/drupal/node$2b3030ba60a26ea685a44efdbbe3e2740376521c397897cad22ab33e66c9ea85.json

Lookup time:    0.004
Connect time:   0.133
PreXfer time:   0.621
StartXfer time: 0.807

Total time: 0.811

curl -s -w '\nLookup time:\t%{time_namelookup}\nConnect time:\t%{time_connect}\nPreXfer time:\t%{time_pretransfer}\nStartXfer time:\t%{time_starttransfer}\n\nTotal time:\t%{time_total}\n' -o /dev/null https://packagist.org/p/drupal/node%243220c90e8196c98484216f516bd93dbb68a94c472d68fea61e5489ba9249fdcb.json

Lookup time:    0.004
Connect time:   0.122
PreXfer time:   0.564
StartXfer time: 0.795

Total time: 0.918

[webflo]

Thanks for bringing this up. The response times are indeed not very good. We can try to reduce the package size (e.g. remove unnecessary suggestions for submodules). Maybe we can save a few requests with these optimizations.

[davidbarratt]

are we behind a CDN?

[webflo]

Nope, not yet because cache invalidation is hard.

Nevertheless, i configured fastly yesterday. The stage is accessible via (http://packagist.drupal-composer.org.global.prod.fastly.net/). Cache TTL is 3600.

[generalredneck]

BTW, to those of yall who do development on a daily basis, my suggestion is this... set up a cron to do a composer update --dry-run to update your cache on a regular interval (like every day). This will make those wait times lower when you REALLY need to do a composer update.

This is of course a work around.

[generalredneck]

@webflo has there been any progress on this?

[webflo]

Not much, i configured Fastly a few weeks ago (see https://github.com/drupal-composer/drupal-packagist/issues/48#issuecomment-177899324) and it seems to work fix. But i have to configure the certificate before we switch.

[ghost]

@webflo

The stage is accessible via (http://packagist.drupal-composer.org.global.prod.fastly.net/) ... i have to configure the certificate before we switch.

Does that me we should change

    ./composer.json

        "repositories": {
          "0": {
            "type": "composer",
-            "url": "https://packagist.drupal-composer.org"
+            "url": "https://packagist.drupal-composer.org.global.prod.fastly.net"
           },

? A 'simple'

composer -vvv update drupal/core

for my site is taking well over an hour to complete.

A drush update of a parallel site takes a minute or two.

[oxyc]

FYI; Prestissimo 3.0 added a feature to download packages.json files in parallel. This vastly improves the performance in my case. Locally fetching all the packages.json files with an empty cache went from 400 sec to 165 sec. Over at Drupal VM the build time decreased from 12min to 10min.

[webflo]

Drupal Packagist won't receive any bugfixes anymore. I think d.o is much faster and stable.