Split `provider-includes` to decrease amount of data to be downloaded

[hiqsol]

This is a simple but effective optimization.

See links below, hope it is self-explanatory enough:

https://getcomposer.org/doc/05-repositories.md#provider-includes-and-providers-url

https://asset-packagist.org/packages.json

https://packagist.org/packages.json

[edgardmessias]

We need save the packages and releases into database.

From database, we can split packages by last release date and generate packages.

I created a pull request (fxpio/composer-asset-plugin#297) to add the release date in fxpio/composer-asset-plugin

[hiqsol]

I think packages' release dates are not important in this concern. What is important that's keeping older items in provider includes list unchanged and change only the latest item.

Basically my idea is:

• rename current provider-latest to something like provider-2017-05;
• empty and start writing new provider-latest.json with only fresh updates.

I think that's all what is necessary. But I'm not sure how everything works in composer so I'd like to hear other opinions.

[edgardmessias]

My opinion, save the packages and releases into database, to create better search and package pages.

Data can not be duplicated on providers, because of caching.

Flow:

• Update database
• If has change, generate new json from package and save hash in database
• Regenerate all providers (cron)

The idea is to update the provider in blocks, not every package update

[edgardmessias]

@hiqsol, after finishing improving the search, I will start working on this item.

[hiqsol]

Data can not be duplicated on providers, because of caching.

Are you absolutely sure about it?

Because then the whole idea of splitting providers is lost I'm afraid. If all the providers got rewritten all the time then they cannot be cached on composer side.

[edgardmessias]

Yes, you can see in the "COMPOSER_HOME".

Example: <composer_home>/repo/https---packagist.org contains only one cache for each package. If the package exists in multiple provider, only one is used.

See the ComposerRepository work

[hiqsol]

only one is used

That's ok as soon as the latest one is used.

What do you think of what I said earlier: splitting is useless if we rewrite providers all the time.

[edgardmessias]

Older packages are not rewrite providers, only the updated.

There are many bower and NPM packages that are not updated

[edgardmessias]

To reduce the rewrite, it would be interesting to use crontab, every 5 minutes for example, and no more per package update

[edgardmessias]

https://wpackagist.org/ rewrite every hour

[hiqsol]

What if we just don't rewrite old providers?

[edgardmessias]

The problem is when an old package is updated, it needs to be removed from the old provider and placed in the new

[hiqsol]

And what if not?

[edgardmessias]

May cause cache problem

[hiqsol]

On user's composer side? What makes you sure about it causes problems? Do you have any links where I can read more about providers?

[edgardmessias]

See the loadProviderListings method in ComposerRepository.

I learned a little better how the composer works:

If the package is in multiple providers, only the last package of the last provider is evaluated.

But the composer will download the package information for each time you find the provider.

Example:

• Package test/abc is in alpha (Hash 123), beta (Hash 456) and gama (Hash 789) providers and the cache is from gama (Hash: 789)
• Composer provider-includes (Line 566)
• Composer get the cache key (Line 569)
• Check composer cache (Hash 123 !== Hash 789)
• Composer download package info (Line 573)
• Merge the package in list (Line 576) (Hash 123)
• Check next provider
• Composer get the cache key (Line 569)
• Check composer cache (Hash 456 !== Hash 123) (Last hash is overrided in line 573)
• Composer download package info (Line 573)
• Merge the package in list (Line 576) (Hash 456)
• Check next provider
• Composer get the cache key (Line 569)
• Check composer cache (Hash 789 !== Hash 456) (Last hash is overrided in line 573)
• Composer download package info (Line 573)
• Merge the package in list (Line 576) (Hash 789).

[hiqsol]

Thank you for the research! Need time to dive in myself.

[edgardmessias]

The more you divide the provider, the greater the chance of the client to use the cache.

Maybe we can divide the last months by weeks, and the oldest by month and year.

When a package updates, it will change the most recent provider and the old it was

Example: week-1, week-2, week-3, week-4, week-5, week-this, if update package in week-2, only week-2 and week-this is updated

[edgardmessias]

@hiqsol, Any answers?

[github-actions[bot]]

Stale issue message