search

jikan-me / jikan-rest

The REST API for Jikan
https://docs.api.jikan.moe/
MIT License
435 stars 269 forks source link

Some endpoint responses are too large #523

Open firmart opened 7 months ago

firmart commented 7 months ago

I ran into a problem where the data returned by the Jikan API was too large, which unfortunately made some endpoints unusable on the project I'm working on, involving LLM. I have getAnimeReviews, getAnimeCharacters, getAnimeRecommendations in mind, but there are others and their "Manga" counterparts.

In my particular use case, a simple additional query parameter limit as already present in the /anime or /manga endpoints will solve the issue, but it would be even nicer to have full pagination. Actually, getAnimeReviews does implement pagination, but the page size cannot be controlled.

Thanks for the great work !

pushrbx commented 7 months ago

Hi @firmart , thanks for the feedback.

The reason there is no pagination for these endpoints is a technical limitation. All of these endpoints are just scraping the relevant pages from MAL, then they are caching the scrape result in the database for a certain period of time. If there is no pagination on these MAL pages, we can't add any pagination to these endpoints. (One request to Jikan API is exactly one request to MAL if the page is not cached yet)

A few examples to just make it easy to visualise it:

Reviews
https://myanimelist.net/anime/52991/Sousou_no_Frieren/reviews
image There is only a "next page" and "previous page" link button at the bottom of the page.

Characters
https://myanimelist.net/anime/52991/Sousou_no_Frieren/characters There is no pagination in most of the cases.

Recommendations https://myanimelist.net/anime/9253/Steins_Gate/userrecs
image There is no pagination at all, it just dumps all of them on one page under a collapsible component.

Just to be more helpful:

which unfortunately made some endpoints unusable on the project I'm working on, involving LLM

This statement raises some eyebrows for me. In case of an LLM related project the size of the data should not be a problem, it might be a problem the way you try to parameterise the LLM, or you might want to use a different model which doesn't limit the token count that much. I can give you better hints if you could elaborate on that part of your problem.