How to check Migration seems to hanging

[AleksCee]

After the hotfix I can start the migration but it seems tu hang now, how can I check or resume it?

the count 635 is more than 10 minutes without changes.

select count() from results; select count() from results_bad_json; +----------+ | count(*) | +----------+ | 635 | +----------+ 1 row in set (0.001 sec)

+----------+ | count(*) | +----------+ | 7257 | +----------+ 1 row in set (0.007 sec)

[AleksCee]

If it’s stops I see a log output in the docker console: speedtest-tracker | ./run: line 2: 115 Killed s6-setuidgid webuser php $WEBUSER_HOME/artisan queue:work --tries=3 --no-ansi -q speedtest-tracker | ./run: line 2: 223 Killed s6-setuidgid webuser php $WEBUSER_HOME/artisan queue:work --tries=3 --no-ansi -q

Now I have try it again and stops after 800 records.

[AleksCee]

Ok, my problem is fixed - it seems that the migrate-task was terminated to fast. After the systemload was lower I have truncate the results an start the migration again. Now it was faster an finished bevor the task was killed. perhaps the taskrunner-timeout should be increased for this job?

[jaggel]

I had the same issue and ended up in doing "partial" migrations chunk by chunk... Moving some and the deleting the migrated from the _bad_json table and re-starting the migration. I agree that the timeout should be increased especially for installations with large number of records.

[sschneider]

After starting the migration, directly two messages are showing. "Starting data migration..." and "There was an issue migrating the data!"

Will look into it later to share more information.

[alexjustesen]

Ok, my problem is fixed - it seam that the migrate-task was terminated to fast. After the systemload was lower I have truncate the results an start the migration again. Now it was faster an finished bevor the task was killed. perhaps the taskrunner-timeout should be increased for this job?

I'll increase the timeout of the job. I tested with ~5000 records and didn't get this but that comes down to how fast the host system is.

[alexjustesen]

@AleksCee how long was the process running before it stopped?

[AleksCee]

@alexjustesen it looks like 6 minutes from the first request to /results till the killed entry in the logs.

[alexjustesen]

What hardware are you running on? That feels really slow

[AleksCee]

It‘s an Synology nas (ds716+) but at Update time the backup jobs were still running.

[AleksCee]

@alexjustesen btw timing, when starting the docker-compose with Maria-dB and Speedtest sometimes the dB migration starts befor the sql server is ready to connect, because of release upgrade in the docker after update to latest of the dB-Container. Can you perhaps check connection in a loop in a little retry? In case if the database not ready, the Speedtest container crash and restart 2-3 times.. is eventually in some cases after an update a problem….

[juanmanuelbc]

I'll increase the timeout of the job. I tested with ~5000 records and didn't get this but that comes down to how fast the host system is.

My Synology DS920+ wasn't able to afford the migration of 7.182 results, so I made the migration of the database with my laptop and now everything is working like a charm...

Thank you!!!

[alexjustesen]

@alexjustesen btw timing, when starting the docker-compose with Maria-dB and Speedtest sometimes the dB migration starts befor the sql server is ready to connect, because of release upgrade in the docker after update to latest of the dB-Container. Can you perhaps check connection in a loop in a little retry? In case if the database not ready, the Speedtest container crash and restart 2-3 times.. is eventually in some cases after an update a problem….

I'm planning on updating that doc page with a health check so it waits for a healthy DB connection. GitBook is having an issue with that component for the last few days so I haven't been able to make updates. Probably have to just delete it and make a new one.

[sschneider]

@alexjustesen my sqlite results_bad_json table has over 22 thousand entries. The error points to a part of the framework (function bindValues)

[2024-02-20 17:49:59] production.ERROR: Error: Object of class stdClass could not be converted to string in /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php:723

I guess this makes it not part of the previous described issue.

[alexjustesen]

@sschneider 22k! What are you running it every 5min? I think you might be at the top of the leader board lol

That was way outside of my test criteria, sounds like I'll have to split the job up into batches that can be processed separately to avoid long running processes.

[sschneider]

@alexjustesensind 2022-10-31 when I set it up the second or third time :-). The check runs every 30 minutes ;-) Thanks for all the work you put into this :)

[AleksCee]

I have 45576 in the old Speedtest (wait for the importer ;) ), 7600 in yours and about the same rows are lost by changing from sqlite to MySQL because can’t find a way to import the dump to MySQL. Test runs every 30 minutes, too. :-) we love this tool :-)

[thegodfatherrelish]

I've been having a problem with the updates for a few days now; when I run the update (Watchtower or manually via the terminal), I get the following error messages. I have now deleted and recreated everything each time, but that can't be the point... Host system is a Synology DS920+ with DSM 7.2.1 U4.

speedtest-new date,stream,content 2024/02/21 06:35:55,stderr,prog: fatal: stopping the container. 2024/02/21 06:35:55,stderr,/run/s6/basedir/scripts/rc.init: warning: s6-rc failed to properly bring all the services up! Check your logs (in /run/uncaught-logs/current if you have in-container logging) for more information. 2024/02/21 06:35:55,stderr,s6-rc: warning: unable to start service laravel-automations: command exited 1 2024/02/21 06:35:55,stdout, 2024/02/21 06:35:55,stdout,
2024/02/21 06:35:55,stdout," e, report to the developers" 2024/02/21 06:35:55,stdout, SQLSTATE[HY000] [2054] Server sent charset (0) unknown to the client. Pleas
2024/02/21 06:35:55,stdout,
2024/02/21 06:35:55,stdout,In Connector.php line 65: 2024/02/21 06:35:55,stdout, 2024/02/21 06:35:55,stdout,
2024/02/21 06:35:55,stdout," BLE', 'SYSTEM VERSIONED') order by table_name) " 2024/02/21 06:35:55,stdout, tables where table_schema = 'speedtest_tracker' and table_type in ('BASE TA
2024/02/21 06:35:55,stdout," engine as engine, table_collation as collation from information_schema. " 2024/02/21 06:35:55,stdout," name, (data_length + index_length) assize, table_comment ascomment, " 2024/02/21 06:35:55,stdout," e, report to the developers (Connection: mysql, SQL: select table_name as " 2024/02/21 06:35:55,stdout, SQLSTATE[HY000] [2054] Server sent charset (0) unknown to the client. Pleas
2024/02/21 06:35:55,stdout,
2024/02/21 06:35:55,stdout,In Connection.php line 829: 2024/02/21 06:35:55,stderr,"} 2024/02/21 06:35:55,stderr,#36 {main} 2024/02/21 06:35:55,stderr,#35 /var/www/html/artisan(35): Illuminate\Foundation\Console\Kernel->handle() 2024/02/21 06:35:55,stderr,#34 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(201): Symfony\Component\Console\Application->run() 2024/02/21 06:35:55,stderr,#33 /var/www/html/vendor/symfony/console/Application.php(175): Symfony\Component\Console\Application->doRun() 2024/02/21 06:35:55,stderr,#32 /var/www/html/vendor/symfony/console/Application.php(324): Symfony\Component\Console\Application->doRunCommand() 2024/02/21 06:35:55,stderr,#31 /var/www/html/vendor/symfony/console/Application.php(1096): Illuminate\Console\Command->run() 2024/02/21 06:35:55,stderr,#30 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(180): Symfony\Component\Console\Command\Command->run() 2024/02/21 06:35:55,stderr,#29 /var/www/html/vendor/symfony/console/Command/Command.php(326): Illuminate\Console\Command->execute() 2024/02/21 06:35:55,stderr,#28 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(211): Illuminate\Container\Container->call() 2024/02/21 06:35:55,stderr,#27 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\Container\BoundMethod::call() 2024/02/21 06:35:55,stderr,#26 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod() 2024/02/21 06:35:55,stderr,#25 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure() 2024/02/21 06:35:55,stderr,#24 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}() 2024/02/21 06:35:55,stderr,#23 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\Database\Console\Migrations\MigrateCommand->handle() 2024/02/21 06:35:55,stderr,#22 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Console/Migrations/MigrateCommand.php(83): Illuminate\Database\Migrations\Migrator->usingConnection() 2024/02/21 06:35:55,stderr,#21 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(633): Illuminate\Database\Console\Migrations\MigrateCommand->Illuminate\Database\Console\Migrations\{closure}() 2024/02/21 06:35:55,stderr,#20 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Console/Migrations/MigrateCommand.php(84): Illuminate\Database\Console\Migrations\MigrateCommand->prepareDatabase() 2024/02/21 06:35:55,stderr,#19 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Console/Migrations/MigrateCommand.php(116): Illuminate\Database\Console\Migrations\MigrateCommand->repositoryExists() 2024/02/21 06:35:55,stderr,#18 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Console/Migrations/MigrateCommand.php(140): retry() 2024/02/21 06:35:55,stderr,#17 /var/www/html/vendor/laravel/framework/src/Illuminate/Support/helpers.php(248): Illuminate\Database\Console\Migrations\MigrateCommand->Illuminate\Database\Console\Migrations\{closure}() 2024/02/21 06:35:55,stderr,#16 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Console/Migrations/MigrateCommand.php(140): Illuminate\Database\Migrations\Migrator->repositoryExists() 2024/02/21 06:35:55,stderr,#15 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(700): Illuminate\Database\Migrations\DatabaseMigrationRepository->repositoryExists() 2024/02/21 06:35:55,stderr,#14 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/DatabaseMigrationRepository.php(184): Illuminate\Database\Schema\Builder->hasTable() 2024/02/21 06:35:55,stderr,#13 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Schema/Builder.php(165): Illuminate\Database\Schema\MySqlBuilder->getTables() 2024/02/21 06:35:55,stderr,#12 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Schema/MySqlBuilder.php(41): Illuminate\Database\Connection->selectFromWriteConnection() 2024/02/21 06:35:55,stderr,#11 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(401): Illuminate\Database\Connection->select() 2024/02/21 06:35:55,stderr,#10 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(414): Illuminate\Database\Connection->run() 2024/02/21 06:35:55,stderr,#9 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(783): Illuminate\Database\Connection->runQueryCallback() 2024/02/21 06:35:55,stderr,#8 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(816): Illuminate\Database\Connection->Illuminate\Database\{closure}() 2024/02/21 06:35:55,stderr,#7 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(423): Illuminate\Database\Connection->getPdoForSelect() 2024/02/21 06:35:55,stderr,#6 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(528): Illuminate\Database\Connection->getPdo() 2024/02/21 06:35:55,stderr,#5 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(1339): call_user_func() 2024/02/21 06:35:55,stderr,#4 [internal function]: Illuminate\Database\Connectors\ConnectionFactory->Illuminate\Database\Connectors\{closure}() 2024/02/21 06:35:55,stderr,#3 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connectors/ConnectionFactory.php(184): Illuminate\Database\Connectors\MySqlConnector->connect() 2024/02/21 06:35:55,stderr,#2 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connectors/MySqlConnector.php(24): Illuminate\Database\Connectors\Connector->createConnection() 2024/02/21 06:35:55,stderr,#1 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php(44): Illuminate\Database\Connectors\Connector->createPdoConnection() 2024/02/21 06:35:55,stderr,#0 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php(65): PDO->construct() 2024/02/21 06:35:55,stderr,[stacktrace] 2024/02/21 06:35:55,stderr,"[previous exception] [object] (PDOException(code: 2054): SQLSTATE[HY000] [2054] Server sent charset (0) unknown to the client. Please, report to the developers at /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connectors/Connector.php:65)" 2024/02/21 06:35:55,stderr, 2024/02/21 06:35:55,stderr,#27 {main} 2024/02/21 06:35:55,stderr,#26 /var/www/html/artisan(35): Illuminate\Foundation\Console\Kernel->handle() 2024/02/21 06:35:55,stderr,#25 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(201): Symfony\Component\Console\Application->run() 2024/02/21 06:35:55,stderr,#24 /var/www/html/vendor/symfony/console/Application.php(175): Symfony\Component\Console\Application->doRun() 2024/02/21 06:35:55,stderr,#23 /var/www/html/vendor/symfony/console/Application.php(324): Symfony\Component\Console\Application->doRunCommand() 2024/02/21 06:35:55,stderr,#22 /var/www/html/vendor/symfony/console/Application.php(1096): Illuminate\Console\Command->run() 2024/02/21 06:35:55,stderr,#21 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(180): Symfony\Component\Console\Command\Command->run() 2024/02/21 06:35:55,stderr,#20 /var/www/html/vendor/symfony/console/Command/Command.php(326): Illuminate\Console\Command->execute() 2024/02/21 06:35:55,stderr,#19 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(211): Illuminate\Container\Container->call() 2024/02/21 06:35:55,stderr,#18 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\Container\BoundMethod::call() 2024/02/21 06:35:55,stderr,#17 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod() 2024/02/21 06:35:55,stderr,#16 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure() 2024/02/21 06:35:55,stderr,#15 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}() 2024/02/21 06:35:55,stderr,#14 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\Database\Console\Migrations\MigrateCommand->handle() 2024/02/21 06:35:55,stderr,#13 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Console/Migrations/MigrateCommand.php(83): Illuminate\Database\Migrations\Migrator->usingConnection() 2024/02/21 06:35:55,stderr,#12 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(633): Illuminate\Database\Console\Migrations\MigrateCommand->Illuminate\Database\Console\Migrations\{closure}() 2024/02/21 06:35:55,stderr,#11 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Console/Migrations/MigrateCommand.php(84): Illuminate\Database\Console\Migrations\MigrateCommand->prepareDatabase() 2024/02/21 06:35:55,stderr,#10 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Console/Migrations/MigrateCommand.php(116): Illuminate\Database\Console\Migrations\MigrateCommand->repositoryExists() 2024/02/21 06:35:55,stderr,#9 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Console/Migrations/MigrateCommand.php(140): retry() 2024/02/21 06:35:55,stderr,#8 /var/www/html/vendor/laravel/framework/src/Illuminate/Support/helpers.php(248): Illuminate\Database\Console\Migrations\MigrateCommand->Illuminate\Database\Console\Migrations\{closure}() 2024/02/21 06:35:55,stderr,#7 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Console/Migrations/MigrateCommand.php(140): Illuminate\Database\Migrations\Migrator->repositoryExists() 2024/02/21 06:35:55,stderr,#6 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/Migrator.php(700): Illuminate\Database\Migrations\DatabaseMigrationRepository->repositoryExists() 2024/02/21 06:35:55,stderr,#5 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Migrations/DatabaseMigrationRepository.php(184): Illuminate\Database\Schema\Builder->hasTable() 2024/02/21 06:35:55,stderr,#4 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Schema/Builder.php(165): Illuminate\Database\Schema\MySqlBuilder->getTables() 2024/02/21 06:35:55,stderr,#3 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Schema/MySqlBuilder.php(41): Illuminate\Database\Connection->selectFromWriteConnection() 2024/02/21 06:35:55,stderr,#2 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(401): Illuminate\Database\Connection->select() 2024/02/21 06:35:55,stderr,#1 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(414): Illuminate\Database\Connection->run() 2024/02/21 06:35:55,stdout, 2024/02/21 06:35:55,stderr,#0 /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php(783): Illuminate\Database\Connection->runQueryCallback() 2024/02/21 06:35:55,stderr,[stacktrace] 2024/02/21 06:35:55,stderr,"[2024-02-21 05:35:55] production.ERROR: SQLSTATE[HY000] [2054] Server sent charset (0) unknown to the client. Please, report to the developers (Connection: mysql, SQL: select table_name as name, (data_length + index_length) as size, table_comment as comment, engine as engine, table_collation as collation from information_schema.tables where table_schema = 'speedtest_tracker' and table_type in ('BASE TABLE', 'SYSTEM VERSIONED') order by table_name) {\"exception\":\"[object] (Illuminate\Database\QueryException(code: 2054): SQLSTATE[HY000] [2054] Server sent charset (0) unknown to the client. Please, report to the developers (Connection: mysql, SQL: select table_name as name, (data_length + index_length) as size, table_comment as comment, engine as engine, table_collation as collation from information_schema.tables where table_schema = 'speedtest_tracker' and table_type in ('BASE TABLE', 'SYSTEM VERSIONED') order by table_name) at /var/www/html/vendor/laravel/framework/src/Illuminate/Database/Connection.php:829)" 2024/02/21 06:35:55,stdout,🚛 Migrating the database... 2024/02/21 06:35:55,stdout, 2024/02/21 06:35:55,stdout,✅ Cache set. 2024/02/21 06:35:54,stdout,💰 Building the cache... 2024/02/21 06:35:54,stdout, 2024/02/21 06:35:54,stdout,✅ An application key exists. 2024/02/21 06:35:54,stdout, 2024/02/21 06:35:54,stdout,✅ Symlinks created. 2024/02/21 06:35:54,stdout,🔗 Creating symlinks for config and log files... 2024/02/21 06:35:54,stdout, 2024/02/21 06:35:54,stdout,✅ Environment file exists. 2024/02/21 06:35:54,stdout, 2024/02/21 06:35:54,stdout,✅ Permissions fixed. 2024/02/21 06:35:54,stdout,🔒 Fixing app path file permissions... 2024/02/21 06:35:54,stdout, 2024/02/21 06:35:54,stdout,🐇 Configuring Speedtest Tracker... 2024/02/21 06:35:54,stdout, 2024/02/21 06:35:54,stdout,"🔒 SSLMODE has been set to MIXED, setting the web server to work in HTTP + HTTPS..." 2024/02/21 06:35:54,stdout, 2024/02/21 06:35:54,stdout,------------------------------------- 2024/02/21 06:35:54,stdout,User gid: 100 2024/02/21 06:35:54,stdout,User uid: 1026 2024/02/21 06:35:54,stdout, 2024/02/21 06:35:54,stdout,------------------------------------- 2024/02/21 06:35:54,stdout,GID/UID 2024/02/21 06:35:54,stdout,------------------------------------- 2024/02/21 06:35:54,stdout,https://github.com/alexjustesen/speedtest-tracker 2024/02/21 06:35:54,stdout,To support Speedtest Tracker visit: 2024/02/21 06:35:54,stdout, 2024/02/21 06:35:54,stdout,-------------------------------------------------------------------- 2024/02/21 06:35:54,stdout, 2024/02/21 06:35:54,stdout, 2024/02/21 06:35:54,stdout, || || |.|||||_|| 2024/02/21 06:35:54,stdout, | | | | | || <| -| | 2024/02/21 06:35:54,stdout,| |.----.---.-.----.| |--.-----.----. 2024/02/21 06:35:54,stdout, ____ 2024/02/21 06:35:54,stdout, || 2024/02/21 06:35:54,stdout,|___|| |__|||____||||__| 2024/02/21 06:35:54,stdout,|__ || | -| -| | | -| --|| _| 2024/02/21 06:35:54,stdout,| _|.-----.-----.-----.--| | |.-----.-----.| | 2024/02/21 06:35:54,stdout, ____ __ 2024/02/21 06:35:54,stdout, 2024/02/21 06:35:54,stdout,-------------------------------------------------------------------- 2024/02/21 06:35:54,stdout, 2024/02/21 06:35:54,stdout,usermod: no changes

[alexjustesen]

@thegodfatherrelish different issue, follow #1205 for that one.

[HiiiiiHa]

hello, migration launched one hour ago on Synology 220+. No notification of completion so far. Results are still showing empty. Will wait some more but looks like something wrong happened

Any advise? thanks

update1: no update after 24h, hence updated to latest version and launched again migration...after more than 30min still nothing update2: upset by this I decided to reinstall from scratch after cleaning everything and now facing a SQLSTATE[HY000] [2002] Connection refused error

[SAS-1]

same issue running on Synology NAS 920+ and my bad results has 33.913 rows in it, kicked the migration off and it did 17,703 and just stopped. would be nice to do the migration fully but not major.

Maybe we could get a SQL we could manually run to get the data over?

[sschneider]

@alexjustesen I checked the records in more detail and it might be that my issue is related to different JSONs with (most) and without "\" (764).

[alexjustesen]

Import/export is coming in 0.21.0, closing out these migration issues as this system will be removed in favor of the framework's version.