search

cpan-testers / cpantesters-project

A meta-project for tracking CPAN Testers project goals
6 stars 1 forks source link

Matrix times out on very highly-reported dists #8

Open preaction opened 8 years ago

preaction commented 8 years ago

http://matrix.cpantesters.org/?dist=Test-Simple+1.302014_009

We don't control the Matrix, but we do expose the APIs that allow it to work. Specifically this JSON document, which for Test-Simple clocks in at almost 30 Megabytes: http://www.cpantesters.org/distro/T/Test-Simple.json. The size of this document appears to make the matrix time out, and the regular cpantesters.org site isn't exactly pleased about it either.

The solution here is likely to provide an API that only gives summary statistics on specific axes (perl version and OS by default, other axes later), and restrict those statistics to a specific version of the dist. And then an API that gets reports, also restricted to a specific version, and paginated with a hard upper limit. The version restriction should be required on each API, and an index on the table should result in fast queries.

Then we can make the more efficient API and send a PR for the matrix to use it.

preaction commented 8 years ago

The new summary API could also help solve the release database problem, if the summary tables contained a "last updated" field and the API accepted an "updated since" field.