search

LMS-Community / slimserver

Server for Squeezebox and compatible players. This server is also called Lyrion Music Server.
https://lyrion.org
Other
1.2k stars 298 forks source link

FTS is being used before the index is available, throwing errors (LMS9) #1193

Open LMSSonos opened 4 weeks ago

LMSSonos commented 4 weeks ago

Hi all

I upgraded toLyrion Music Server (v9.0.0, 1729751785) from 8.5.3 and I see these errors in the logfile. Is there a command to start migration so that the table will be created?

Thanks

[24-10-24 16:35:13.7991] Slim::Web::Cometd::handler (437) errorNeedsClient: 172.30.0.1, 00:04:20:26:c3:1b, status, -, 10, menu:menu, useContextMenu:1, subscribe:600
[24-10-24 16:35:18.8437] Slim::Web::Cometd::handler (437) errorNeedsClient: 172.30.0.1, 00:04:20:26:c3:1b, displaystatus, subscribe:showbriefly
[24-10-24 16:38:14.3307] Slim::Plugin::FullTextSearch::Plugin::_initPopularTerms (637) Fulltext index missing or outdated - re-building
[24-10-24 16:38:14.3314] Slim::Schema::Storage::throw_exception (122) Error: DBI Exception: DBD::SQLite::db selectcol_arrayref failed: no such table: fulltext_terms [for Statement "
        SELECT term FROM (
            SELECT term, SUM(documents) d
            FROM fulltext_terms
            WHERE NOT col IN ('*', 1, 0) AND LENGTH(term) > 1
            GROUP BY term
        )
        WHERE d > 500
    "]
[24-10-24 16:38:14.3317] Slim::Schema::Storage::throw_exception (122) Backtrace:

   frame 0: Slim::Utils::Log::logBacktrace (/lms/Slim/Schema/Storage.pm line 122)
   frame 1: Slim::Schema::Storage::throw_exception (/lms/CPAN/DBIx/Class/Storage/DBI.pm line 1007)
   frame 2: DBIx::Class::Storage::DBI::__ANON__ (/lms/Slim/Plugin/FullTextSearch/Plugin.pm line 644)
   frame 3: Slim::Plugin::FullTextSearch::Plugin::_initPopularTerms (/lms/Slim/Plugin/FullTextSearch/Plugin.pm line 183)
   frame 4: Slim::Plugin::FullTextSearch::Plugin::__ANON__ (/lms/Slim/Control/Request.pm line 2053)
   frame 5: (eval) (/lms/Slim/Control/Request.pm line 2053)
   frame 6: Slim::Control::Request::notify (/lms/Slim/Control/Request.pm line 858)
   frame 7: Slim::Control::Request::checkNotifications (/lms/slimserver.pl line 708)
   frame 8: main::idle (/lms/slimserver.pl line 676)
   frame 9: main::main (/lms/slimserver.pl line 1215)

[24-10-24 16:38:14.3320] Slim::Control::Request::notify (2055) Error: Failed notify: Carp::Clan::__ANON__(): DBI Exception: DBD::SQLite::db selectcol_arrayref failed: no such table: fulltext_terms [for Statement "
        SELECT term FROM (
            SELECT term, SUM(documents) d
            FROM fulltext_terms
            WHERE NOT col IN ('*', 1, 0) AND LENGTH(term) > 1
            GROUP BY term
        )
        WHERE d > 500
    "] at /lms/Slim/Schema/Storage.pm line 126
michaelherger commented 4 weeks ago

Simplest might be to delete library.db while the container is shut down. LMS9 would trigger a full wipe & rescan anyway.

LMSSonos commented 4 weeks ago

Thanks, I deleted the ones in the /config/cache/library.* and restarted. The scan is still running but when I search i get 

[24-10-25 06:21:54.0761] Slim::Schema::Storage::throw_exception (122) Error: DBI Exception: DBD::SQLite::db selectrow_array failed: no such table: fulltext [for Statement "SELECT count(1) FROM fulltext WHERE fulltext MATCH 'type:track mytrack*'"]

Are the tables only created once the scan is completely finished (takes probably 2 more days) or did I delete the wrong library.db

michaelherger commented 4 weeks ago

Yes, you can't search before the scan has finished. Creation of the full text index only happens at the end of the scanning process.

But that's actually a good pointer: I should catch this case. I've seen similar reports before, but never understood how that would happen. What you describe makes sense! Thanks for that heads up!

LMSSonos commented 4 weeks ago

Thanks! Just added more RAM to the container and restarting because else it took a few days to scan all

I assume this is because I aborted the initial scan - so will delete the /config/cache/library.* files and restart lyrion. right? 

lms    | [24-10-25 07:56:01.3192] Slim::Plugin::FullTextSearch::Plugin::_initPopularTerms (637) Fulltext index missing or outdated - re-building
lms    | [24-10-25 07:56:01.3199] Slim::Schema::Storage::throw_exception (122) Error: DBI Exception: DBD::SQLite::db selectcol_arrayref failed: no such table: fulltext_terms [for Statement "
lms    |                SELECT term FROM (
lms    |                        SELECT term, SUM(documents) d
lms    |        "]
lms    | [24-10-25 07:56:01.3203] Slim::Schema::Storage::throw_exception (122) Backtrace:
lms    |    frame 8: main::idle (/lms/slimserver.pl line 676)
lms    |    frame 9: main::main (/lms/slimserver.pl line 1215)
lms    | 
lms    | [24-10-25 07:56:01.3206] Slim::Control::Request::notify (2055) Error: Failed notify: Carp::Clan::__ANON__(): DBI Exception: DBD::SQLite::db selectcol_arrayref failed: no such table: fulltext_terms [for Statement "
lms    |                        WHERE NOT col IN ('*', 1, 0) AND LENGTH(term) > 1
lms    |                        GROUP BY term
lms    |                )
lms    |                WHERE d > 500
lms    |        "] at /lms/Slim/Schema/Storage.pm line 126
LMSSonos commented 4 weeks ago

The rescan finished, it als created the index for fulltext search

[24-10-25 14:49:10.0416] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (537) Create fulltext index for tracks  
[24-10-25 14:57:02.7934] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (547) Create fulltext index for albums  
[24-10-25 15:02:31.0095] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (556) Create fulltext index for contributors
[24-10-25 15:03:02.2509] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (566) Create fulltext indel[24-10-25 14:49:10.0416] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (537) Create fulltext index for tracks
[24-10-25 14:57:02.7934] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (547) Create fulltext index for albums
[24-10-25 15:02:31.0095] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (556) Create fulltext index for contributors
[24-10-25 15:03:02.2509] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (566) Create fulltext index for works
[24-10-25 15:10:12.0644] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (575) Create fulltext index for playlists
[24-10-25 15:10:23.4360] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (590) Optimize fulltext index
[24-10-25 15:11:23.4520] Slim::Plugin::FullTextSearch::Plugin::_rebuildIndex (605) Fulltext index build done!

but later then this error comes up

[24-10-25 17:17:31.0619] Slim::Plugin::FullTextSearch::Plugin::_initPopularTerms (637) Fulltext index missing or outdated - re-building
[24-10-25 17:17:31.0630] Slim::Schema::Storage::throw_exception (122) Error: DBI Exception: DBD::SQLite::db selectcol_arrayref failed: no such table: fulltext_terms [for Statement "
            SELECT term FROM (
                    SELECT term, SUM(documents) d
   frame 4: Slim::Plugin::FullTextSearch::Plugin::__ANON__ (/lms/Slim/Control/Request.pm line 2053)     
   frame 5: (eval) (/lms/Slim/Control/Request.pm line 2053)
   frame 6: Slim::Control::Request::notify (/lms/Slim/Control/Request.pm line 858)
   frame 7: Slim::Control::Request::checkNotifications (/lms/slimserver.pl line 708)
   frame 5: (eval) (/lms/Slim/Control/Request.pm line 2053)
   frame 6: Slim::Control::Request::notify (/lms/Slim/Control/Request.pm line 858)
   frame 7: Slim::Control::Request::checkNotifications (/lms/slimserver.pl line 708)
   frame 8: main::idle (/lms/slimserver.pl line 676)
   frame 9: main::main (/lms/slimserver.pl line 1215)

I relaunched a scan (without delete) to see if that helps

Edit: Scan completed, errors disappeart, all seems to work now. Thanks a lot!