Multi-search without rate limiting, how to?

[Prid13]

Since there is no single /searchAll/ endpoint, making a search engine site for anything anime-related, like searching not only for anime titles, but also mangas, characters, people/musicians, OPs/EDs (themes), etc. (just like how it is on MAL, showing all the results in a single page) is impossible due to the rate limiting of 3 requests per second :/

Continuing with this search engine website example, suppose there's a detailed page for each anime/manga title. How am I supposed to show related animes/mangas (sequels, prequels, spin-offs, etc.), OPs/EDs (themes), pictures, and potentially news, without hitting the API limit? :(

Let's go one step further -- imagine the website allows animes to be favorited. Adding an anime to favorites would save/cache the JSON results (across all the API calls made: anime, related, themes, pictures, etc.). Obviously, this information would need to be updated occassionally, or at least checked for new updates every now and then.

• Would sending a request with If-None-Match header count against the API rate limit?
• If yes, how can I feasibly check for updates in all aspects of an anime/manga title (details, related, themes, news, pictures, etc.)?

Having endpoints like /searchAll and /getAllAnimeDetails would be really helpful when the rate limiter is (rightfully) constrained.

Possible workaround: I've thought about sending all the required API calls by spacing them out 3 calls per second, so wanting to call 5 endpoints would mean calling 3 endpoints first, then waiting 1 second, and then calling the next 2. This would've been an acceptable solution were it not for the fact that a user could search for other anime quickly one after the other, possibly blocking some of the search results due to hitting the rate limit (bc of buffered API calls), and also eating up the 60 requests / minute limit.

Any thoughts, workarounds, possible solutions being worked on, or optimal ways of doing this that I'm not aware of?

Thanks :)

[BmElectro]

You can just host your own Jikan instance and do how many requests your heart desire, the rate limit is simply there for the cloud Jikan that is available for everyone.

[irfan-dahir]

I did recently implement getAnimeFullById which returns response like v3 used to (with external links, op/ed, relations). So I hope that solves a part of this issue. :)

However, at the moment it's a technical challenge to return all the anime information in a single response because:

• the rest of the pages (anime pictures, stats, episodes, etc) are separate on MAL and are scraped separately - this means it's a separate request from our server to MAL. This increases the chance of an unsuccessful request (Jikan getting rate limited by MAL) if the other requests are not cached or outdated.
• the rest of the pages are cached separately and are not indexed like the main anime information is

---

A search all call sounds interesting but it's something that I would not prioritize right now. At the moment I'd rather see a feature stack of Jikan that has a good implementation of search overall at first (https://github.com/jikan-me/jikan-rest/issues/189). Then perhaps from there we could look into a search all call. Any feature PRs would be welcome at this point with the discussion we have going in #189 .

---

As for rate limits; increasing this is a popular demand. But right now I need to balance it so it prevents issues like #177 . I will revise the rate-limit policy once we discontinue v3 entirely because that's a resource hog of the bandwidth we have for MAL requests.

[akio-q]

You can just host your own Jikan instance and do how many requests your heart desire, the rate limit is simply there for the cloud Jikan that is available for everyone.

How much space will it take on my hosting if I host my own instance? Will it load the full database on my server? Do I need to have some kind of powerful server?

[pushrbx]

@akio-q Please in the future open a new issue for questions, or post it on our discord server in the "support" channel. I've already answered your question on discord: https://discord.com/channels/460491088004907029/1222845429847162971/1222854965240791120

Also I'm sharing it here too:

• Database, search index and the codebase is roughly around 2gb disk space
• Images are linked from MAL cdn, no images are stored on the server. MAL pages are scraped, turned into JSON, saved into mongodb and if enabled in the search index.
• "Do I need to have some kind of powerful server?" Depends on the traffic you anticipate. I'm self-hosting and my website has 25 active users per day, so it has been ok with 2vcpu and 2gb ram server. (ofc with this ram size you need to tweak bunch of stuff yourself to prevent slow downs)