Question about setting up migration table

[chrisgrim]

Hi I am struggling to understand the mapping a bit. In my event index I have

 Index::create('events', function (Mapping $mapping, Settings $settings) {
            $mapping->search_as_you_type('name');
            $mapping->date('published_at');
            $mapping->integer('rank');
            $mapping->date('closingDate');
            $mapping->text('priceranges');
            $mapping->integer('category_id');
            $mapping->geo_point('location_latlon');
            $mapping->text('shows');
            $mapping->text('genres');
            $mapping->boolean('hasLocation');
        });

However it doesn't work when I try to do search for name. I am using the same database and have your old elasticsearch plugin on one server and the new one locally. So on the old one when I use

$event = Event::search($request->keywords)
             ->rule(EventSearchRule::class)
             ->take(5)
             ->get();

And in EventSearchRule.php 

return [
            'must' => [
                'multi_match' =>  [
                    "query" =>  $this->builder->query,
                    "type"  =>  'bool_prefix',
                    "fields"    =>  [
                        'name',
                        'name.2gram',
                        'name.3gram'
                    ],
                ]
            ]
        ];

If I type in 'c' it returns cairn, curiousor, and 3 more. However in the new system when I use that migration above and

$a = Event::multiMatchSearch()
        ->fields(['name', 'name.2gram','name.3gram'])
        ->query($request->keywords)
        ->execute();

        return $a->models();

if I type in 'c' it returns only one C(ovell) In The C(loud). If I type in a full word like 'chaos' it will then find that name and returns that event named chaos.

Do I need to do $mapping->dynamicTemplate()? If so, how would that setup be for geo_point or search_as_you_type? Thanks so much!!

[babenkoivan]

Hey @chrisgrim, you should use camelCased method names, i.e.:

$mapping->searchAsYouType('name');
$mapping->geoPoint('name');

You can find the full list of supported types here.

[chrisgrim]

Hi @babenkoivan Thanks so much for the quick response and information. That list was very helpful! Sadly it didn't seem to fix my problem. My new migration file is

final class CreateEventsIndex implements MigrationInterface
{
    /**
     * Run the migration.
     */
    public function up(): void
    {
        Index::create('events', function (Mapping $mapping, Settings $settings) {
            $mapping->searchAsYouType('name');
            $mapping->date('published_at');
            $mapping->integer('rank');
            $mapping->date('closingDate');
            $mapping->text('priceranges');
            $mapping->integer('category_id');
            $mapping->geoPoint('location_latlon');
            $mapping->text('shows');
            $mapping->text('genres');
            $mapping->boolean('hasLocation');
        });
    }

Then I ran php artisan elastic:migrate:refresh and then php artisan scout:import "App\Models\Event". It said Imported [App\Models\Event] models up to ID: 471 All [App\Models\Event] records have been imported. Then in my controller I double checked and I have

$a = Event::multiMatchSearch()
        ->fields(['name', 'name.2gram','name.3gram'])
        ->query($request->keywords)
        ->execute();

        return $a->models();

which should do the type ahead. However if I start to type in 'chaos', it doesn't return a result until I have completely typed in the word, then returns the event object for the event named chaos. I realize this might be a question now for elastic-scout-driver-plus but I figured I had already started the question here. Thanks again!!

Also is there a way to check the migrations on elastic search? When I type in http://127.0.0.1:9200/_cat/indices?v into the browser I see all the indices created by the old elasticsearch plugin but I dont see the new ones. It really helped me to see my data was actually there by doing something like http://127.0.0.1:9200/event/_search?pretty&size=60. However if I try to do http://127.0.0.1:9200/Create_Events_Index/_search?pretty&size=60 or http://127.0.0.1:9200/CreateEventsIndex/_search?pretty&size=60 it says index_not_found_exception.

[babenkoivan]

Hey @chrisgrim, if we have a look at the example on this page, we can see, that the search fields are:

"my_field",
"my_field._2gram",
"my_field._3gram"

It looks like you are missing the underscore. But you are right, it's better to check first if the documents are in the index or not. Your index name is events (the first argument in Index::create). You can still do http://127.0.0.1:9200/events/_search?pretty=1&size=100, this driver doesn't change how ES works 🙂

Also please make sure, that your actual mapping matches what you have in the migration. Maybe you should delete the index and create it again via elastic:migrate:refresh.

[chrisgrim]

Hi @babenkoivan Sorry for all the questions if I can do anything to help with this project please let me know. I am happy to write documentation or help with things like that.

Good catch with the _2gram.I realized I had to delete the old event indeces I had created before with your older plugin using curl -XDELETE localhost:9200/event; Then I did the migrate refresh and it worked. But now I am having something strange happen. In my event model I have:

/**
    * What events should be searchable for scout elastic search
    *
    * @return \Illuminate\Database\Eloquent\Relations\belongsTo
    */
    public function shouldBeSearchable()
    {
        return $this->isPublished();
    }

    /**
    * Determines which events are published for Laravel Scout
    *
    * @return bool
    */
    public function isPublished() {
        return $this->status == 'p';
    }

    /**
     * Get the indexable data array for the model.
     *
     * @return array
     */
    public function toSearchableArray()
    {
        return [
            'name' => $this->name,
            'published_at' => $this->published_at,
            'showtype' => $this->showtype,
            'rank' => $this->rank,
            'closingDate' => $this->closingDate,
            'priceranges' => $this->pricerangesSelect,
            'category_id' => $this->category_id,
            'location_latlon' => $this->location_latlon,
            'shows' => $this->showsSelect,
            'genres' => $this->genreSelect,
            'hasLocation' => $this->hasLocation
        ];
    }

and in my migration I now have

final class CreateEventsIndex implements MigrationInterface
{
    /**
     * Run the migration.
     */
    public function up(): void
    {
        Index::create('events', function (Mapping $mapping, Settings $settings) {
            $mapping->searchAsYouType('name');
            $mapping->date('published_at');
            $mapping->integer('rank');
            $mapping->date('closingDate');
            $mapping->text('priceranges');
            $mapping->integer('category_id');
            $mapping->geoPoint('location_latlon');
            $mapping->text('shows');
            $mapping->text('genres');
            $mapping->boolean('hasLocation');
        });
    }

If I run migration refresh and import events the indeces will be empty. However if I comment out

return [
            'name' => $this->name,
            'published_at' => $this->published_at,
            'showtype' => $this->showtype,
            'rank' => $this->rank,
            // 'closingDate' => $this->closingDate,
            // 'priceranges' => $this->pricerangesSelect,
            'category_id' => $this->category_id,
            'location_latlon' => $this->location_latlon,
            // 'shows' => $this->showsSelect,
            // 'genres' => $this->genreSelect,
            'hasLocation' => $this->hasLocation
        ];

Then it imports 12 events and a lot of them and some of the options are null

{
"_index": "events",
"_type": "_doc",
"_id": "83",
"_score": 1,
"_source": {
"name": "Off The Table",
"published_at": null,
"showtype": "o",
"rank": 1,
"category_id": 13,
"location_latlon": null,
"hasLocation": false
}
},

However, if I comment out the entire migration table

Index::create('events', function (Mapping $mapping, Settings $settings) {
            // $mapping->searchAsYouType('name');
            // $mapping->date('published_at');
            // $mapping->text('showtype');
            // $mapping->integer('rank');
            // $mapping->date('closingDate');
            // $mapping->text('priceranges');
            // $mapping->integer('category_id');
            // $mapping->geoPoint('location_latlon');
            // $mapping->text('shows');
            // $mapping->text('genres');
            // $mapping->boolean('hasLocation');
        });

and then uncomment out the tosearchable array and run migration and import it seems to work as it should.

{
"_index": "events",
"_type": "_doc",
"_id": "9",
"_score": 1,
"_source": {
"name": "Adventures in the Mind's Ear",
"published_at": null,
"showtype": "a",
"rank": 1,
"closingDate": "2020-12-12 00:46:16",
"priceranges": [
{
"price": "3.50"
},
{
"price": "12.00"
}
],
"category_id": 10,
"location_latlon": null,
"shows": [
{
"date": "2020-12-12 00:46:16"
}
],
"genres": [
{
"name": "ASMR",
"pivot": {
"event_id": 9,
"genre_id": 26
}
},
{
"name": "Audio",
"pivot": {
"event_id": 9,
"genre_id": 27
}
},
{
"name": "Immersive",
"pivot": {
"event_id": 9,
"genre_id": 1
}
},
{
"name": "Intimate",
"pivot": {
"event_id": 9,
"genre_id": 12
}
},
],
"hasLocation": false
}
},

Do I actually need to make a migrations table? It seems like it fills in the indices correctly with the toSearchableArray.

[chrisgrim]

I did a bit more testing and I realized that it regardless of what I name the migration table, if I run

php artisan scout:import "App\Models\Event"

It will make an events indices and fill it with my events. So its almost like if I create the events migration table it conflicts with the import instead of having the import fill it up. If I create a migrations table called event then I will have two indices, one called events and one called event.

[chrisgrim]

Hi @babenkoivan I have done more tinkering today and figured out some things/ have new questions. I realized I do have to create a migrations table because when I do http://127.0.0.1:9200/events/_mapping I can see that if I don't setup the migration table correctly then it does the wrong mapping types. This also led me to figure out why it wasn't filling out my entire indices, since I was using the incorrect migration type for some of my fields. This does lead me to another question.

Some of my fields like shows and genres return an array of shows assigned to the event. I was able to get this to work by using $mapping->nested('shows'); however when I look at the mapping this brings up a new issue.

"shows": {
"type": "nested",
"properties": {
"date": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},

How do I control the field types inside the nested and what gets pulled into the nested? Date shouldn't be a text field. I also can't control what parts of the model to bring in. This was accomplished with your older plugging using

protected $mapping = [
        'properties' => [
'shows' => [
                'properties' => [
                    'date' => [
                        'type' => 'date',
                        'format' => 'yyyy-MM-dd HH:mm:ss'
                    ],
                ]
            ],

How do I accomplish the same thing?

[chrisgrim]

Sorry for the tons of comments, I figured it out! My final migration table looks like

 Index::create('events', function (Mapping $mapping, Settings $settings) {
            $mapping->searchAsYouType('name');
            $mapping->text('showtype');
            $mapping->integer('rank');
            $mapping->integer('category_id');
            $mapping->geoPoint('location_latlon');
            $mapping->boolean('hasLocation');
            $mapping->object('priceranges', ['properties' => ['price' => ['type' => 'integer']]]);
            $mapping->object('shows', ['properties' => ['date' => ['type' => 'date', 'format' => 'yyyy-MM-dd HH:mm:ss']]]);
            $mapping->object('genres');
            $mapping->date('closingDate', ['format' => 'yyyy-MM-dd HH:mm:ss']);
            $mapping->date('published_at', ['format' => 'yyyy-MM-dd HH:mm:ss']);
        });

in case this helps anyone else. Now off to figure out the elastic scout driver plus plugin :)