search

estiens / world_cup_json

Rails backend for a scraper that outputs World Cup data as JSON
928 stars 107 forks source link

Long tail requests - the ALL MATCHES endpoint #141

Open estiens opened 6 years ago

estiens commented 6 years ago

Now that we have a ton of information in the JSON, the matches endpoint that returns all information for every match is really big one to return and requests to it are often topping out over 1s. Cacheing is in place, indexes are in place and query optimization is ongoing, but I'd really not like to have an unpaginated endpoint that returns all info for all matches.

But I know a lot of people are using this endpoint (many aren't, they are using /matches/today or matches/current)

ideal situation would be to return the summary view in the endpoint because you can always grab all details for a match by going to the show view /matches/[fifa_id] OR by limiting your date range, but I know anyone grabbing the entire match object and parsing data from it will have to change their app to either make multiple calls (grab details for each match) or limit dates (or i could implement pagination) - I could also still allow grabbing all details from this match endpoint by passing in a detail param, but I'd like to not just give all details on all matches to all people that hit that endpoint by default somehow.

If people have any thoughts could you weigh in here? I'm loathe to make a breaking change to an in-use API, but 1+ second response times for that endpoint tie up a lot of resources that could be shooting out live results for smaller subsets of matches quicker.

Possibly I'll cache that endpoint much longer....

estiens commented 6 years ago

Using OJ and removing partials seems to have reduced response time of all matches endpoint to like 500ms instead of 1-2s. Will run with that for awhile and see.

estiens commented 5 years ago

Thinking about changing this for the WWC in 2019 - Anyone building a new app would have to request details for individual matches and just get a summary view from the all_matches endpoint

(ie,status, score, etc, but not individual events for all matches)

estiens commented 5 years ago

Note - the index endpoint will now return truncated information by default (other endpoints matches/today, etc) will return all details. If you want the former behavior of all matches with all events pass the param details=true to the main matches endpoint