Search: Improve v4 search

[Fezlight]

Hello,

I think this issue is related to spaces in search query param.

In Jikan v3 : https://api.jikan.moe/v3/search/anime?q=d%20gray%20man There are many result according to "D gray man".

In Jikan v4 : https://api.jikan.moe/v4/anime?q=d%20gray%20man No result.

Thanks for your help !

[irfan-dahir]

Hi,

v4 queries an internal database via %like% patterns instead of parsing MAL pages. The search currently expects the appropriate title if you're searching for the entire term.

e.g https://api.jikan.moe/v4/anime?q=D.Gray-man

This is not a bug/issue - rather there is currently no loose/tokenized query feature added for search.

We discussed this a bit over on our discord server: https://discord.com/channels/460491088004907029/462991054619148288/935589650930753576

[Fezlight]

Thanks for your reply :)

Is there a new feature planned to be able to use loose/tokenized query ?

[irfan-dahir]

Not at the moment

[irfan-dahir]

Reopening this because it's been added to the roadmap. https://github.com/orgs/jikan-me/projects/2

Feature branch testing with Scout+MeiliSearch: https://github.com/jikan-me/jikan-rest/tree/feature/scout

[irfan-dahir]

This feature is now highly prioritized. Development in progress...

🚧 WIP Feature branch testing with Elasticsearch: https://github.com/jikan-me/jikan-rest/tree/feature/elasticsearch ⚠ Not recommended for production or self-hosting at the moment. Do it at your own risk. 🚧 WIP Wiki: Installation, Setting up Elasticsearch

[pushrbx]

Just out of curiosity: What happened with the laravel scout approach? I find it a bit strange that a third app is required to sync mongodb contents to ES. It would be cool if less setup would be required.

[irfan-dahir]

@pushrbx Laravel Scout doesn't have official support for Elasticsearch and the other third-party ones I looked into seemed a bit outdated which might require me to test out further or additionall development.

Monstache is something I'm trying out which for me has been a very elegant out-of-the-box experience for quickly prototyping this and it works with the latest ES 8.1. Additionally, with MongoDB v4+ it works by listening for changes if the MongoDB instance is configured as a single-node replica set which I easily did in production.

I can look into a custom scout approach in the future but any PR help meanwhile for setting it up would be fantastic.

[pushrbx]

My thinking from a hosting perspective was that with scout you wouldn't need another container image configuration for a yet another app/service, you could just reuse the container image of jikan, and spin up another container with it which runs an artisan command. (I'm doing this now with queues and the schedule stuff)

I did a bit of reading, and some things look hopeful: https://github.com/handcraftedinthealps/ElasticsearchDSL/pull/18 https://github.com/matchish/laravel-scout-elasticsearch/issues/184 ^ this library seems stable and active, but they don't have ES 8.x support yet. I've read the scout branch, and that one looks almost there. 👍 If I have downtime at work again I'm going to give it a shot.

Is there a reason for the need for ES 8.x? Also I've read somewhere that you guys had trouble allowing people to search based on airing times. Is that still a problem? (I can't find it on the api docs page https://docs.api.jikan.moe/#operation/getAnimeSearch so I guess yes?)

[irfan-dahir]

Sorry for the late response, @pushrbx

What you mentioned is fair. It does make sense to use scout or a artisan command to mange this rather than something external.

I've read the scout branch, and that one looks almost there. 👍 If I have downtime at work again I'm going to give it a shot.

We can look into that and I'd be happy if you'd be up for implementing a test branch if you find the time. I'm a bit tied at the moment so this is a low priority for me right now. My priority is to get a working POC with the existing queries we have available with MongoDB.

Is there a reason for the need for ES 8.x?

No specific reason other than it's the latest version and is a bit better on performance. If we were to use an older version, we'd still have to upgrade at a certain point.

Also I've read somewhere that you guys had trouble allowing people to search based on airing times. Is that still a problem? (I can't find it on the api docs page https://docs.api.jikan.moe/#operation/getAnimeSearch so I guess yes?)

Don't think we've had that issue. start_date and end_date work fine. It's a separate matter that I forgot to include it in the docs until recently. 😅

[pushrbx]

@irfan-dahir I'm going to work on this then and make this laravel scout thing a reality in Jikan. 😄 Additionally may I recommend TypeSense instead of ES? It has better laravel scout support atm. Altough it's not a problem for me to add support for both of them. According to the comparison all the major features required by the Jikan API is there (i might have missed something): https://typesense.org/typesense-vs-algolia-vs-elasticsearch-vs-meilisearch/

[irfan-dahir]

@pushrbx That'd be epic. 🙌

Jikan API requirements would be the support for the querying we currently offer via search. Like data ranges (score, timestamps), nested JSON matches (which MeiliSearch, unfortunately, doesn't support at the moment otherwise I would've gone for it), matching for other languages like Hiragana, Kanji, etc.

Typesense looks interesting and I'm open to it. 👍

[irfan-dahir]

We've started testing out TypeSense publicly. If anyone's interested, you can get access to the staging server on our discord channel to help us tweak the results.

https://discord.com/channels/460491088004907029/461020417998454785/989259536550465586

[fremmede]

It doesn't work for me with V4 but with normal V3 what could be the error.

` public function listings() { $Filter = $this->getVariable("Filter"); $Settings = $this->getVariable("Settings");

    $this->page             = isset($_GET['page']) && is_numeric($_GET['page']) ? $_GET['page'] : 1;

    if ($this->page) $ApiFilter .= '&page=' . $this->page;

    $Client     = new \GuzzleHttp\Client();
    if(($Filter['type'] == 'movie' || $Filter['type'] == 'tv') AND $Filter['q']) {
    $Response   = $Client->request('GET', 'https://api.jikan.moe/v3//search/anime/' . '?q=' . Input::cleaner($Filter['q']) . '&limit=15');
    } else if (($Filter['type'] == 'movie' || $Filter['type'] == 'tv') AND $Filter['bulk_ids']) {
      $results = [];
      foreach (explode("\r\n", $Filter['bulk_ids']) as $bulk_id) {
        $_response = $Client->request('GET', 'https://api.jikan.moe/v3/anime/' . $bulk_id );
        $result = json_decode($_response->getBody() , true);
        array_push($results, $result);
      }
    } else {

    $Response   = $Client->request('GET', 'https://api.themoviedb.org/3/discover/'.Input::cleaner($Filter['type']).'?sort_by='.Input::cleaner($Filter['sort']).$ApiFilter . '&api_key=' . get($Settings,'data.tmdb_api','api') . '&language='.get($Settings,'data.tmdb_language','api') . '&append_to_response=release_dates');
    }

    if (!empty($Filter['bulk_ids'])) {
      $Results = [
        'results' => $results,
        'total_results' => count($results),
      ];
    } else {
      $Results = json_decode($Response->getBody() , true);
    }

    foreach ($Results['results'] as $Result) {
        if (Input::cleaner($Filter['type']) == 'movie') {

            $Listings[] = [
                'id'        => trim($Result['mal_id']),
                'type'      => $Result['type'],
                'link'      => $Result['url'],
                'title'     => $Result['title'],
                'image'     => $Result['image_url'],
                'episode'   => $Result['episodes'],
                'mpaa'      => $Result['rated']
            ];
        } elseif (Input::cleaner($Filter['type']) == 'tv') {

            $Listings[] = [
                'id'        => trim($Result['mal_id']),
                'type'      => $Result['type'],
                'link'      => $Result['url'],
                'title'     => $Result['title'],
                'image'     => $Result['image_url'],
                'episode'   => $Result['episodes'],
                'mpaa'      => $Result['status']
            ];
        }
    }

    $this->paginationLimit  = 20;
    $this->totalRecord      = $Results['total_results'];
    $this->pageCount        = ceil($this->totalRecord / $this->paginationLimit);

    $Pagination         = $this->showPagination(APP.'/admin/mal?filter='.json_encode($Filter).'&page=[page]');
    $this->setVariable("Listings",$Listings);   
    $this->setVariable("Pagination",$Pagination);

}

`

[pushrbx]

@fremmede We don't support V3 anymore. In case of V4 we are still testing the new search functionality, it's still in progress, and it's not even on the master branch yet. So what do you need help with?

[luknl]

Hi! Any news on this? (Thanks again for this amazing API btw)

[pushrbx]

Hi! Any news on this? (Thanks again for this amazing API btw)

I'm in the final stages of testing at the moment. Also thanks for letting me know about raycast. 😄

[luknl]

Ok great to know! And you're welcome for Raycast, it's a great app indeed. They just approved my PR for an extension to search animes from MAL thanks to Jikan 😀. You can download it here.

[AlbEagle]

can anyone to help me with change v3 to v4 with code.

let $ = jQuery;

$('#meta_mal_api_button').click(function () { let id = $('#meta_mal_api_input').val(); $('#meta_mal_api_input').val('Processing').attr("disabled", 'disabled'); fetch('https://api.jikan.moe/v3/anime/' + id) .then(res => res.json()) .then(res => { $('#title').val(res.title); $('#ak_synonyms').val(res.title_synonyms.join(", ")); $('#ak_english').val(res.title_english); $('#ak_japanese').val(res.title_japanese); $('#ak_episodes').val(res.episodes); $('#ak_duration').val(res.duration); $('#ak_aired').val(res.aired.string); $('#ak_source').val(res.source); $('#ak_score').val(res.score); $('#ak_rating').val(res.rating); $('#new-tag-type').val(res.type); $('#new-tag-season').val(res.premiered); $('#new-tag-status').val(res.status.replace("Currently Airing", "Ongoing")); $('#ak_trailer').val(res.trailer_url.replace("?enablejsapi=1&wmode=opaque&autoplay=1", "")); $('#ak_producers').val(res.producers.map(item => { return item.name }).join(", ")); $('#ak_licensors').val(res.licensors.map(item => { return item.name }).join(", ")); $('#new-tag-genres').val(res.genres.map(item => { return item.name }).join(",")); $('#new-tag-studios').val(res.studios.map(item => { return item.name }).join(",")); var synopsis = ""; var output = res.synopsis.replace(/\n/g, "
"); synopsis += "" + output + ""; if (typeof tinyMCE != "undefined") { var editor_id = wpActiveEditor; if ($('#wp-' + editor_id + '-wrap').hasClass('tmce-active') && tinyMCE.get(editor_id)) { tinyMCE.get(editor_id).setContent(synopsis); } else { $("#content").val(synopsis); } } }) .then(() => { $('#meta_mal_api_input').val('Success').removeAttr("enable", 'enable'); }) .catch(e => { alert('Error: Please Input Correct MAL ID!'); $('#meta_mal_api_input').val('').removeAttr("disabled", 'disabled'); console.log(e) }); })

Thank!