cpan-testers / cpantesters-api

An API in to data held by CPAN Testers: Test reports and CPAN uploads
Other
4 stars 4 forks source link

Add APIs to list/search raw test reports #10

Open preaction opened 7 years ago

preaction commented 7 years ago

We should have a way to list and search the raw test report JSON that we are storing. We should create the following endpoints:

Other fields could be added as query parameters for more constraints like ?grade=(pass|fail).

The above endpoints are only examples. If they can not perform decently, we cannot add them. We need to test these APIs against all the records (which means this cannot be released until the old test report records are migrated: cpan-testers/cpantesters-backend#6).

Since there are 80 million records and climbing, this will need to be paginated. Each endpoint should accept offset and limit parameters. The limit parameter must be less than or equal to 1000 (this number chosen because it matches the old Metabase log.txt limit). The output should contain pagination information along with the records returned, like:

{
    "offset": 0,
    "limit": 1000,
    "total": 81345619,
    "query": { },
    "order_by": [
        [ "created", "desc" ]
    ],
    "results": [
        { "result": { "grade": "pass", "output": { } } },
        { "result": { "grade": "na", "output": { } } },
        { "result": { "grade": "fail", "output": { } } }
    ]
}

There is a problem that doing a COUNT(*) on an InnoDB table in MySQL is super-slow. But, without that information, it becomes much harder to use the API.