Adding an aggregation which is not a collection property (e.g. years) breaks search

[paulvanbladel]

Hi, This is a very interesting and extremely useful project.

I'm just encountering a slight problem when I extend the filter configuration with a pure 'scalar' value. In order to situate the problem as clear as problem, allow me to refer to your movie sample. (https://jsfiddle.net/cigol/0ef9qeos/5/) When in this sample, you change the configuration as follows:

    year: {
      title: 'Year',
      size: 10
    }

the search isn't working any longer. Did I miss something? You can find the updated jsfiddle reproducing the problem here as well: https://jsfiddle.net/paulvb/86xbs4jd/ So, basically the 'year list', the bucket list so to speak is built up correctly but when you select a year, things break.

Thanks a lot paul.

[cigolpl]

@paulvanbladel thanks for kind words regarding project. I was considering to implement scalar values as a range filters i.e. if you have 1995 year in movies it will show up as 1990 - 2000, 2001 as 2000 - 2010 (depends how you define ranges in configuration). I was afraid though that the next kind of filters (ranges) will make a search performance suffer so I didn't implemented it yet so far. Also I didn't have an idea for a simple algorithm and simple implementation.

There are two workarounds though:

• you can make a data pre-processing before you index them to the search so i.e. if you have a year 1995 you can make it a '1995' or '1990 - 2000' then everything should work smoothly
• if you have lot of items you can also consider another project https://github.com/itemsapi/elasticitems which is supporting scalar values and it is more robust but on the other hand it requires real search engine Elasticsearch. I've indexed and search through it 250 mln of records recently

[paulvanbladel]

Hi, Thanks a lot for the prompt reply. Well I have a relatively small set of data so ItemsJs so ItemsJs is exactly what I need. I do not really need the ranges, I'm perfectly ok that the different years appear as such, but I'm not clearly understanding what you mean with "make a data pre-processing before you index them". Could you maybe just look to the sample and tell me what I need to change in order to get it working? https://jsfiddle.net/paulvb/86xbs4jd/

[cigolpl]

Paul, the each year field in file you are including in jsfiddle https://cdn.rawgit.com/itemsapi/itemsapi-example-data/master/jsfiddle/imdb.js should be as a string then it should work

[paulvanbladel]

very nice. Works like a charm :)

I'm patching the data just before they enter the magic search function (that's probably what you meant with 'pre-processing before you index'):

computed: {
    searchResult: function () {
      rows.forEach(e => {
        e.year = e.year.toString()
      })
      var result = itemsjs(rows, this.configuration).search({
        per_page:100,
        query: this.query,
        filters: this.filters
      })
      return result
    }
  }

[cigolpl]

Paul, great it is working now but If you execute that code after website is being loaded / init stage / only once:

rows.forEach(e => {
  e.year = e.year.toString()
})
var client = itemsjs(rows, this.configuration)

Then it will be much faster because ItemsJS will not need to index data every time you search :)

[paulvanbladel]

That is great advice. I'll adapt accordingly :)

[paulvanbladel]

Hi, There is a slight problem with this approach. So, the year prop in the movie db is now a string. But, when I change one of the movies year to 199, it will appear nicely as a facet, but in the search results when selecting 199 as fascet, all movies which have a year starting with 199 will appear. Is there a way to specify that an exact match is required? Thanks a million. paul.

[cigolpl]

Thanks for info! Actually since the beginning it was intended that facets / filters are searchable by exact match. If it's not working like that then it seems like a bug.

I'll add a test here checking your case https://github.com/itemsapi/itemsjs/blob/master/tests/aggregationsSpec.js. If you could provide minimal configuration and data which is causing that behavior that'd be also very helpful

[paulvanbladel]

Hi, sorry for the delay. I created a github repository which can nicely illustrate the problem. It's based on the canonical movies database sample. In order to illustrate the problem, I changed the "year" property of one of the movies to 199, which is the first 3 chars of many other years like 1991,1992, ... :)

Please find: https://github.com/paulvanbladel/ItemsJs-Vue-Demo run

npm install
npm serve

then select a year different from 199, so e.g. 1997 and you'll get 4 movies. Then select 199 as year and you get a whole list of movies while there is actually one movie from the year 199 (must be black and white)

Hopefully this make sense for you. Thanks a lot. paul.

[paulvanbladel]

I found out that the problem only happens when it is about a field that has a scalar value. If for example I change one of the tag fields from ganster to ga "tags": ["ga", "organized crime", "mafia", "rise and fall", "cold blooded murder"], then the search on ga will return only one item (as expected). So, the year property is not a collection property but a scalar and that's when it goes wrong :)

[cigolpl]

Thank you for a nice example! It's strange because I am not distinguishing scalar values and strings in ItemsJS. This lib is using Lodash.js quite a lot i.e. for comparisons. My first guess is Lodash is treating / comparing scalar and strings differently.

I'll write some tests and I'll try to figure out a root cause of that bug

I like https://github.com/paulvanbladel/ItemsJs-Vue-Demo because it is using different libraries than I usually use (i.e. quasar, babel, etc) so there is a chance to learn something new. One thing I have noticed - if you could in some way initialize ItemsJS once i.e. on page load than in searchResult then the time of search response will be significantly faster (in that case even from 20ms to 3ms)

[cigolpl]

@paulvanbladel you have discovered a very interesting bug. Actually it happens not when there is a scalar value but when the field is not an array.

I've found a workaround. Please change a line in src/components/Movies.vue: From e.year = e.year.toString() to e.year = [e.year.toString()]. I assume having year as an array might be inconvenient for you but that's only one way to solve it right now.. :( Anyway ItemsJS should support not only array fields in that case but also just typical strings. I've put that into a todo list

BTW could I add your example https://github.com/paulvanbladel/ItemsJs-Vue-Demo to the DEMO's of ItemsJS ? It uses different technologies and maybe can be useful for someone

[paulvanbladel]

@cigolpl Hi Mateusz, No problem that' a great workaround. You know, VueJs has a nice plugin model. I'm planning to develop such a plugin for ItemJs in such a way the "heavy lifting" of integrating ItemsJs in vue is done by the plugin. That would bring integrating itemsjs in an app to the level of adding an html tag with a configuration object. Sure, I would be very happy if you add my sample to the demo section. Thanks a lot for the great support.

[cigolpl]

Your idea with VueJs and ItemsJs as a plugin sounds great! I've added your demo to demo page. Thanks

[paulvanbladel]

@cigolpl
FYI: I did a first level of abstracting. the whole facet machinery is now abstracted under an html component (it's not yet a vuejs plugin. Basically, the component needs 2 inputs : jsondata and a configuration object. When the facet engine has data is returned by means of an event 'searchResultUpdated'. Simple comme bonjour :) See: https://github.com/paulvanbladel/ItemsJs-Vue-Demo

 <items-js-facets :rows="jsonData" :configuration="configuration" @searchResultUpdated="searchResultUpdated">
        </items-js-facets>

So, the whole screen boils down to:

<template>
  <q-page padding>

    <div class="row">
      <div class="col-3">
        <items-js-facets :rows="jsonData" :configuration="configuration" @searchResultUpdated="searchResultUpdated">
        </items-js-facets>
      </div>
      <div class="col-1"></div>
      <div class="col-7">
        <q-list>
          <q-item v-for="item of items" :key="item.name">
            <q-item-side><img style="width: 100px;" v-bind:src="item.image"></q-item-side>
            <q-item-main :label="item.name" :sublabel="item.description">
              <q-item-tile>
                <br>
                <q-chip small tag v-for="tag in item.tags" :key="tag.key">{{ tag }} </q-chip>
              </q-item-tile>
            </q-item-main>
            <q-item-side right>{{item.year[0]}}</q-item-side>
          </q-item>
        </q-list>
      </div>
    </div>
  </q-page>
</template>
<style>
</style>
<script>

import rawJson from './imdb.json'
import ItemsJsFacets from './ItemsJsFacets.vue'
export default {
  components:{ItemsJsFacets},
  created() {
    this.jsonData = rawJson
     this.jsonData.forEach(e => {
        e.year = [e.year.toString()]
      })
  },
  data() {
    var configuration = {
      searchableFields: ['title', 'tags', 'actors'],
      sortings: {
        name_asc: {
          field: 'name', order: 'asc' }
       },
      aggregations: {
        tags: { title: 'Tags', size: 200 },
        actors: { title: 'Actors', size: 10 },
        genres: { title: 'Genres', size: 10 },
        country: { title: 'Country', size:20 },
        year: { title: 'Year', size:100 }
      }
    }
    return {
      jsonData: [],
      configuration,
      items: []

    }
  },
  methods: 
  {
    searchResultUpdated(d) {
      this.items = d
    }
  }
}
</script>

[cigolpl]

@paulvanbladel I've tested out the fresh version. Looks very promising! So in very short it will be possible to create any search engine (books, movies, games, etc) just by providing json data, configuration and rendering html like below:

<items-js-facets :rows="jsonData" :configuration="configuration" @searchResultUpdated="searchResultUpdated">
</items-js-facets>

True ?

[paulvanbladel]

Yes but inside a vue-js app. But that reminds me that native web components are possible now: https://github.com/vuejs/vue-web-component-wrapper

[domoritz]

As I understand it, itemsjs does check for exact matches in the filters. It would be great if it supported numbers and boolean in addition to strings. It seems a bit odd that as a user I have to convert numbers to strings to make them work in facets. Or am I missing something?

[cigolpl]

@paulvanbladel, @domoritz

The itemsjs version 2.0.1 should resolve this issue. I've introduced much better faceted search algorithms and now fixing it was quite easy

[vmengh]

This is a great library and I am able to clone and run it locally. Thanks for the work!
I did observed 2 minor issues

1. ItemsJs-Vue-Demo package.json is missing the serve script tag - serve:'npm run serve'
2. There's a lint issue everytime you update the index.vue or vue file Post every update you have to run - npm run lint -- --fix and then npm run serve it could just be my configuration, not sure

Are there any other libraries that support elastic search (from different cloud sources), I looked at appbase.io they do provide components for vue / react and more but all these require elasctic search at the backend and component ? Thanks