Error while trying to import with tntsearch:import

[jpmurray]

Hello,

I have a table with a bit above 57 million rows that I'm trying to index. If I start the indexation with scout:import all goes well and indexation starts, but it's slow, so I'm trying to use tntsearch:import. Now with that last command I'm getting an undefined index error.

I've found out that the culprit is that I specify the primaryKey name as such in my model:

protected $primaryKey = "song_id";

Is there anyway I can make the driver play well with this line?

[pwnz22]

Show please your toSearchableArray()

[jpmurray]

public function toSearchableArray()
    {
        $array = $this->toArray();
        unset($array['title_version']);
        unset($array['original_release_date']);

        return $array;
    }

[jpmurray]

Leaving a bit more trace so maybe it can help trying to solve it.

Putting the toSearchableArray method to this:

public function toSearchableArray()
    {
        return [
            $this->getKeyName() => $this->getKey(),
            'name' => $this->name,
            'artist_display_name' => $this->artist_display_name,
            'collection_display_name' => $this->collection_display_name,
        ];
    }

will make artisan scout:import work, but not artisan tntsearch:import. At the moment, I'm trying to index 57+ million rows and it's just too slow with scout:import. And I also fear that I'll be hitting the same problem as written in #81 .

Here's the error it throws:

// sond_id being the primarykey
[PDOException (42S22)]
  SQLSTATE[42S22]: Column not found: 1054 Unknown column 'song_id' in 'field list'

// trace points at
() at /home/forge/tools.utvarp.co/vendor/teamtnt/tntsearch/src/Indexer/TNTIndexer.php:209
 PDO->query() at /home/forge/tools.utvarp.co/vendor/teamtnt/tntsearch/src/Indexer/TNTIndexer.php:209

[pwnz22]

I used to have something similar. I solved it when added in array'id' => $this->id.

[jpmurray]

Thanks for yout suggestion @pwnz22, but it didn't work :( What I can see is that the driver (or tnt search) is assuming an ID named id. Problem is, the id in this table is called something else...

At the moment, I can successfully use the tntsearch:import function if I remove the protected $primaryKey in the model and use the toSearchableArray as in my second message.

BUT.

The importer stops at 31274 rows, and there is above 57 millions :\

[nticaric]

@jpmurray what error do you get after 31274 rows?

Regarding the first issue that you had, have you tried to make the property public, like:

public $primaryKey = "song_id"

[jpmurray]

@nticaric I just tried with the public $primaryKey, didn't worked :(

As for the error, there is none! It just stops, like the command would when it reached the end.

[nticaric]

Very strange indeed, we'll try to help you with both issues, but we'll need some more debugging from you. Let's focus first on the primaryKey problem.

In our import command we have a piece of code that check the primary key, take a look here:

https://github.com/teamtnt/laravel-scout-tntsearch-driver/blob/master/src/Console/ImportCommand.php#L45

The question is, why it's failing. So could you do a dd($model->getKeyName()); on line 35 of the file above to see what we have.

The file is in your vendor/teamtnt/laravel-scout-tntsearch-driver/src/Console/ directory

[jpmurray]

Just before we do that, so we're clear and I don't bring you in a bug chase incorectly....

If I have public $primaryKey = "song_id"; set in my model, I can do scout:import but not tntsearch:import. Removing the primaryKey variable will make scout:import fail, but tntsearch:import run correctly.

Now, running the dd (at line 40, so the $model variable is set :P), will return "song_id" if the $primaryKey is set to the model, and "id" if it's not set.

Also, if you want, since we seems to be both online at the same time, we can make it on a chat somewhere (Larachat? Gitter? Something else). No pressure, just offering so it's easier ;-)

[jpmurray]

Now that is completely weird. I just tried to run with a smaller table (a bit over 2000 rows), and tntsearch:import runs up to 31199 rows :\

[nticaric]

Hmm, strange, whats the exact command that you're running?

[jpmurray]

php artisan tntsearch:import App\\Models\\EPF\\Song -v

[nticaric]

Could it be that you a have one or more databases where the data is differen?

[jpmurray]

@nticaric Absolutely, I have. Although the connections are correctly specified in the model and it accesses the right data. (EDIT: just to be clear it accesses the right data if I use the model.)

... oh, but tntsearch:import doesn't know that since it's not directly the scout driver, could that be it?

[bryandease]

I am having a similar issue. I have 3 different databases that data is getting pulled from. The connections are declared on the model instances but tntsearch:import doesn't seem to follow the model connections. Is there a way to declare the correct connection explicitly?

As a quick hack to get it to work with different databases I modified the end of the artisan command to this:

$table = $model->getConnectionName() . '.' . $model->getTable();
$indexer->query("SELECT $query FROM {$table};");

Obviously a nicer way would be allowing the config settings above this code to handle the connection - but this seems to work for now.

[marksparrish]

I think this is caused by the a double call to the primary key. When I dd the query string, it shows an entry for id and the custom primary key.

"SELECT id, person_id, first_name FROM people;"

My toSearchableArray is -

    public function toSearchableArray()
    {
        return [
            'people_id' => $this->people_id,
            'first_name' => $this->first_name,
        ];
    }

Also, you can't modify the call on $this->first_name. I wanted to add nicknames to the index through a call to my nickname helper.

public function toSearchableArray()
{
    $nickname = new Nickname();
    return [
        'people_id' => $this->people_id,
        'first_name' => $nickname->nicknames($this->first_name),
    ];
}

The only way the nicknames get indexed is to use the scout:import model artisan command.