Composer Cache Miss on http://packages.firegento.com/packages.json

[astorm]

I'm not sure where the "blame" for this goes, and I'm not familiar enough the architecture of this project and composer to debug this myself, so apologies if this is the wrong place for this.

I'm running into a problem where composer doesn't seem to cache the contents of the firegento packages.json file. If I delete my ~/.composer/cache folder and run the following

$ composer.phar --profile -vvv --no-dev --repository-url=http://packages.firegento.com create-project magento-hackathon/magento-composer-installer .
[4.2MB/0.05s] Downloading http://packages.firegento.com/packages.json
[70.6MB/152.18s] Writing /Users/alanstorm/.composer/cache/repo/http---packages.firegento.com/packages.json into cache

It takes composer 152.18 seconds to download the repository's packages.json file. This makes sense since I just deleted the cache. However subsequent commands will see equally long downloading of the packages.json file

$ composer.phar --profile -vvv update
[3.7MB/0.01s] Reading ./composer.json
[4.2MB/0.02s] Executing command (CWD): git describe --exact-match --tags
[4.4MB/0.03s] Executing command (CWD): git branch --no-color --no-abbrev -v
[4.5MB/0.05s] Executing command (CWD): hg branch
[4.5MB/0.12s] Executing command (CWD): svn info --xml
[6.6MB/0.17s] Loading composer repositories with package information
[6.8MB/0.19s] Downloading http://packages.firegento.com/packages.json
[73.2MB/125.50s] Writing /Users/alanstorm/.composer/cache/repo/http---packages.firegento.com/packages.json into cache

Here's it's 125.50s. This behavior doesn't happen with the default packagist packages.json. After it's initially downloaded, composer reads it's contents from cache instantly.

Is there anything I can do on my end to fix this? If there anything firegento can do to fix this?

[Flyingmana]

I will look at this. i know the firegento repo does a few things different then packagist, but it could also be an issue with the cache ttl.

Give us a week to figure this out :)

cc: @daim2k5

[astorm]

I'm not sure how much authority is behind this, but I also opened an issue over on the composer tracker, and stof seems to think composer doesn't implement caching for the packages.json file.

Also, I did some digging on my own and the main packagist repository seems to split packages.json into multiple different files, so the problem may just be that the firegento file is over 12 MB in size. When others use this do they see the same slow performance? (or is everyone else just on proper fast internet)

[Flyingmana]

the splitting is what I meant with "does a few things different" After the splitting they could relay on the hash values to decide if he needs to refetch. But I wanted to do some tests, before doing a statement about this.

Anyway, the satis project could need some improvements, as I think we are the only user with such a big number of packages.

[bragento]

@astorm may be lacking some background information (no offense...just trying to clear things up). http://packages.firegento.com runs a composer repository based on satis, which is a "Simple static Composer repository generator" while Packagist is more complex and dynamic. AFAIK, Satis has little to do with packagist (technically spoken) as the mechanics differ from the scratch. So we are kind of comparing apples and oranges here.

Anyhow I wonder if satis is the right choice for the "main Magento composer repository" as the repo keeps growing whilst satis isn't really built for such proportions. Maybe we should start a discussion about this in another issue.

[sylvainraye]

@astorm did an article about his discovery: http://alanstorm.com/bypassing_a_slow_composer_repository Let's try to discuss on that at the next hackathon in Leipzig (cc @daim2k5) in few weeks. Having most of Magento Connect's packages may reduce the size of the packages.json, it could be splitted in a different packages.json as it is suggested into the official documentation of composer: https://getcomposer.org/doc/05-repositories.md#includes

For bigger repositories, like ours, here is the solution: https://getcomposer.org/doc/05-repositories.md#provider-includes-and-providers-url Does it really help?

As it was noticed into the alan's article, gzip compression is also missing it may help too.

Sylvain

[Flyingmana]

ok, now I have to make a few things clear.

1. I close this issue, as it is wrong context. Go to https://github.com/magento-hackathon/composer-repository/issues because this part gets managed by the FireGento e.V. which I am not part of.
2. They moved to a new and bigger server, and probably forget to activate gzip compression.
3. There are some other big limitations of the setup, which make it quite hard to manage currently. Not my responsibility as I dont manage the repository, but wanted to discuss in Leipzig what makes more sense, to extend Satis to our usecase, or switch to the software packagist uses.
4. I do this in my free time, and get either direct nor indirect money out of this work. I asked for a week time, so I could look at it to give a valuable answer instead of some guessing.
5. if you still expect a valuable response in less then 2 days, there is paid support available if you would look at the readme
6. I like lists with numbers.