search

kimai / kimai

Kimai is a web-based multi-user time-tracking application. Works great for everyone: freelancers, companies, organizations - everyone can track their times, generate reports, create invoices and do so much more. SaaS version available at https://www.kimai.cloud
https://www.kimai.org
GNU Affero General Public License v3.0
3.19k stars 557 forks source link

Unable to export large sets of data using csv, pdf and xls export option in kimai 1.2 #1081

Closed rarvin closed 5 years ago

rarvin commented 5 years ago

Unable to export large sets of data (say one months data) using csv, xls and pdf option in kimai 1.2.

The same works well for smaller ranges of one week. Please help

When using the xls option. Error seen in logs [2019-09-04 15:01:23] request.INFO: Matched route "export_data". {"route":"export_data","route_parameters":{"_route":"export_data","_controller":"App\Controller\ExportController::export","_locale":"en"},"request_uri":"https://chittar.stmdocs.org/en/export/data","method":"POST"} [] [2019-09-04 15:01:23] security.DEBUG: Read existing security token from the session. {"key":"_security_secured_area","token_class":"Symfony\Component\Security\Core\Authentication\Token\UsernamePasswordToken"} [] [2019-09-04 15:01:23] php.INFO: User Deprecated: Checking for the AdvancedUserInterface in "Symfony\Component\Security\Core\Authentication\Token\AbstractToken::hasUserChanged()" is deprecated since Symfony 4.1 and support for it will be removed in 5.0. Implement the Symfony\Component\Security\Core\User\EquatableInterface to check if the user has been changed, {"exception":"[object] (ErrorException(code: 0): User Deprecated: Checking for the AdvancedUserInterface in \"Symfony\Component\Security\Core\Authentication\Token\AbstractToken::hasUserChanged()\" is deprecated since Symfony 4.1 and support for it will be removed in 5.0. Implement the Symfony\Component\Security\Core\User\EquatableInterface to check if the user has been changed, at /var/www/kimai2/vendor/symfony/security-core/Authentication/Token/AbstractToken.php:325)"} [] [2019-09-04 15:01:23] php.INFO: User Deprecated: Checking for the AdvancedUserInterface in "Symfony\Component\Security\Core\Authentication\Token\AbstractToken::hasUserChanged()" is deprecated since Symfony 4.1 and support for it will be removed in 5.0. Implement the Symfony\Component\Security\Core\User\EquatableInterface to check if the user has been changed, {"exception":"[object] (ErrorException(code: 0): User Deprecated: Checking for the AdvancedUserInterface in \"Symfony\Component\Security\Core\Authentication\Token\AbstractToken::hasUserChanged()\" is deprecated since Symfony 4.1 and support for it will be removed in 5.0. Implement the Symfony\Component\Security\Core\User\EquatableInterface to check if the user has been changed, at /var/www/kimai2/vendor/symfony/security-core/Authentication/Token/AbstractToken.php:325)"} [] [2019-09-04 15:01:23] security.DEBUG: User was reloaded from a user provider. {"provider":"Symfony\Component\Security\Core\User\ChainUserProvider","username":"username"} [] [2019-09-04 15:01:23] security.DEBUG: Checking for guard authentication credentials. {"firewall_key":"secured_area","authenticators":1} [] [2019-09-04 15:01:23] security.DEBUG: Checking support on guard authenticator. {"firewall_key":"secured_area","authenticator":"App\Security\TokenAuthenticator"} [] [2019-09-04 15:01:23] security.DEBUG: Guard authenticator does not support the request. {"firewall_key":"secured_area","authenticator":"App\Security\TokenAuthenticator"} [] [2019-09-04 15:01:54] php.CRITICAL: Fatal Error: Maximum execution time of 30 seconds exceeded {"exception":"[object] (Symfony\Component\Debug\Exception\FatalErrorException(code: 0): Error: Maximum execution time of 30 seconds exceeded at /var/www/kimai2/vendor/phpoffice/phpspreadsheet/src/PhpSpreadsheet/Collection/Cells.php:95)"} [] [2019-09-04 15:01:54] request.CRITICAL: Uncaught PHP Exception Symfony\Component\Debug\Exception\FatalErrorException: "Error: Maximum execution time of 30 seconds exceeded" at /var/www/kimai2/vendor/phpoffice/phpspreadsheet/src/PhpSpreadsheet/Collection/Cells.php line 95 {"exception":"[object] (Symfony\Component\Debug\Exception\FatalErrorException(code: 0): Error: Maximum execution time of 30 seconds exceeded at /var/www/kimai2/vendor/phpoffice/phpspreadsheet/src/PhpSpreadsheet/Collection/Cells.php:95)"} [] [2019-09-04 15:01:54] php.INFO: User Deprecated: Calling the "Symfony\Component\EventDispatcher\EventDispatcherInterface::dispatch()" method with the event name as the first argument is deprecated since Symfony 4.3, pass it as the second argument and provide the event object as the first argument instead. {"exception":"[object] (ErrorException(code: 0): User Deprecated: Calling the \"Symfony\Component\EventDispatcher\EventDispatcherInterface::dispatch()\" method with the event name as the first argument is deprecated since Symfony 4.3, pass it as the second argument and provide the event object as the first argument instead. at /var/www/kimai2/vendor/symfony/event-dispatcher/EventDispatcher.php:58)"} [] [2019-09-04 15:01:54] security.DEBUG: Stored the security token in the session. {"key":"_security_secured_area"} []

When using csv option error seen in logs

[2019-09-04 15:05:30] request.INFO: Matched route "export_data". {"route":"export_data","route_parameters":{"_route":"export_data","_controller":"App\Controller\ExportController::export","_locale":"en"},"request_uri":"https://chittar.stmdocs.org/en/export/data","method":"POST"} [] [2019-09-04 15:05:30] security.DEBUG: Read existing security token from the session. {"key":"_security_secured_area","token_class":"Symfony\Component\Security\Core\Authentication\Token\UsernamePasswordToken"} [] [2019-09-04 15:05:30] php.INFO: User Deprecated: Checking for the AdvancedUserInterface in "Symfony\Component\Security\Core\Authentication\Token\AbstractToken::hasUserChanged()" is deprecated since Symfony 4.1 and support for it will be removed in 5.0. Implement the Symfony\Component\Security\Core\User\EquatableInterface to check if the user has been changed, {"exception":"[object] (ErrorException(code: 0): User Deprecated: Checking for the AdvancedUserInterface in \"Symfony\Component\Security\Core\Authentication\Token\AbstractToken::hasUserChanged()\" is deprecated since Symfony 4.1 and support for it will be removed in 5.0. Implement the Symfony\Component\Security\Core\User\EquatableInterface to check if the user has been changed, at /var/www/kimai2/vendor/symfony/security-core/Authentication/Token/AbstractToken.php:325)"} [] [2019-09-04 15:05:30] php.INFO: User Deprecated: Checking for the AdvancedUserInterface in "Symfony\Component\Security\Core\Authentication\Token\AbstractToken::hasUserChanged()" is deprecated since Symfony 4.1 and support for it will be removed in 5.0. Implement the Symfony\Component\Security\Core\User\EquatableInterface to check if the user has been changed, {"exception":"[object] (ErrorException(code: 0): User Deprecated: Checking for the AdvancedUserInterface in \"Symfony\Component\Security\Core\Authentication\Token\AbstractToken::hasUserChanged()\" is deprecated since Symfony 4.1 and support for it will be removed in 5.0. Implement the Symfony\Component\Security\Core\User\EquatableInterface to check if the user has been changed, at /var/www/kimai2/vendor/symfony/security-core/Authentication/Token/AbstractToken.php:325)"} [] [2019-09-04 15:05:30] security.DEBUG: User was reloaded from a user provider. {"provider":"Symfony\Component\Security\Core\User\ChainUserProvider","username":"username"} [] [2019-09-04 15:05:30] security.DEBUG: Checking for guard authentication credentials. {"firewall_key":"secured_area","authenticators":1} [] [2019-09-04 15:05:30] security.DEBUG: Checking support on guard authenticator. {"firewall_key":"secured_area","authenticator":"App\Security\TokenAuthenticator"} [] [2019-09-04 15:05:30] security.DEBUG: Guard authenticator does not support the request. {"firewall_key":"secured_area","authenticator":"App\Security\TokenAuthenticator"} [] [2019-09-04 15:06:02] php.CRITICAL: Fatal Error: Maximum execution time of 30 seconds exceeded {"exception":"[object] (Symfony\Component\Debug\Exception\FatalErrorException(code: 0): Error: Maximum execution time of 30 seconds exceeded at /var/www/kimai2/vendor/phpoffice/phpspreadsheet/src/PhpSpreadsheet/Cell/Coordinate.php:34)"} [] [2019-09-04 15:06:02] request.CRITICAL: Uncaught PHP Exception Symfony\Component\Debug\Exception\FatalErrorException: "Error: Maximum execution time of 30 seconds exceeded" at /var/www/kimai2/vendor/phpoffice/phpspreadsheet/src/PhpSpreadsheet/Cell/Coordinate.php line 34 {"exception":"[object] (Symfony\Component\Debug\Exception\FatalErrorException(code: 0): Error: Maximum execution time of 30 seconds exceeded at /var/www/kimai2/vendor/phpoffice/phpspreadsheet/src/PhpSpreadsheet/Cell/Coordinate.php:34)"} [] [2019-09-04 15:06:02] php.INFO: User Deprecated: Calling the "Symfony\Component\EventDispatcher\EventDispatcherInterface::dispatch()" method with the event name as the first argument is deprecated since Symfony 4.3, pass it as the second argument and provide the event object as the first argument instead. {"exception":"[object] (ErrorException(code: 0): User Deprecated: Calling the \"Symfony\Component\EventDispatcher\EventDispatcherInterface::dispatch()\" method with the event name as the first argument is deprecated since Symfony 4.3, pass it as the second argument and provide the event object as the first argument instead. at /var/www/kimai2/vendor/symfony/event-dispatcher/EventDispatcher.php:58)"} [] [2019-09-04 15:06:02] security.DEBUG: Stored the security token in the session. {"key":"_security_secured_area"} []

When using the pdf option

[2019-09-04 15:06:40] request.INFO: Matched route "export_data". {"route":"export_data","route_parameters":{"_route":"export_data","_controller":"App\Controller\ExportController::export","_locale":"en"},"request_uri":"https://chittar.stmdocs.org/en/export/data","method":"POST"} [] [2019-09-04 15:06:40] security.DEBUG: Read existing security token from the session. {"key":"_security_secured_area","token_class":"Symfony\Component\Security\Core\Authentication\Token\UsernamePasswordToken"} [] [2019-09-04 15:06:40] php.INFO: User Deprecated: Checking for the AdvancedUserInterface in "Symfony\Component\Security\Core\Authentication\Token\AbstractToken::hasUserChanged()" is deprecated since Symfony 4.1 and support for it will be removed in 5.0. Implement the Symfony\Component\Security\Core\User\EquatableInterface to check if the user has been changed, {"exception":"[object] (ErrorException(code: 0): User Deprecated: Checking for the AdvancedUserInterface in \"Symfony\Component\Security\Core\Authentication\Token\AbstractToken::hasUserChanged()\" is deprecated since Symfony 4.1 and support for it will be removed in 5.0. Implement the Symfony\Component\Security\Core\User\EquatableInterface to check if the user has been changed, at /var/www/kimai2/vendor/symfony/security-core/Authentication/Token/AbstractToken.php:325)"} [] [2019-09-04 15:06:40] php.INFO: User Deprecated: Checking for the AdvancedUserInterface in "Symfony\Component\Security\Core\Authentication\Token\AbstractToken::hasUserChanged()" is deprecated since Symfony 4.1 and support for it will be removed in 5.0. Implement the Symfony\Component\Security\Core\User\EquatableInterface to check if the user has been changed, {"exception":"[object] (ErrorException(code: 0): User Deprecated: Checking for the AdvancedUserInterface in \"Symfony\Component\Security\Core\Authentication\Token\AbstractToken::hasUserChanged()\" is deprecated since Symfony 4.1 and support for it will be removed in 5.0. Implement the Symfony\Component\Security\Core\User\EquatableInterface to check if the user has been changed, at /var/www/kimai2/vendor/symfony/security-core/Authentication/Token/AbstractToken.php:325)"} [] [2019-09-04 15:06:40] security.DEBUG: User was reloaded from a user provider. {"provider":"Symfony\Component\Security\Core\User\ChainUserProvider","username":"username"} [] [2019-09-04 15:06:40] security.DEBUG: Checking for guard authentication credentials. {"firewall_key":"secured_area","authenticators":1} [] [2019-09-04 15:06:40] security.DEBUG: Checking support on guard authenticator. {"firewall_key":"secured_area","authenticator":"App\Security\TokenAuthenticator"} [] [2019-09-04 15:06:40] security.DEBUG: Guard authenticator does not support the request. {"firewall_key":"secured_area","authenticator":"App\Security\TokenAuthenticator"} [] [2019-09-04 15:07:02] request.CRITICAL: Uncaught PHP Exception Mpdf\MpdfException: "The HTML code size is larger than pcre.backtrack_limit 1000000. You should use WriteHTML() with smaller string lengths." at /var/www/kimai2/vendor/mpdf/mpdf/src/Mpdf.php line 26661 {"exception":"[object] (Mpdf\MpdfException(code: 0): The HTML code size is larger than pcre.backtrack_limit 1000000. You should use WriteHTML() with smaller string lengths. at /var/www/kimai2/vendor/mpdf/mpdf/src/Mpdf.php:26661)"} [] [2019-09-04 15:07:02] php.INFO: User Deprecated: Calling the "Symfony\Component\EventDispatcher\EventDispatcherInterface::dispatch()" method with the event name as the first argument is deprecated since Symfony 4.3, pass it as the second argument and provide the event object as the first argument instead. {"exception":"[object] (ErrorException(code: 0): User Deprecated: Calling the \"Symfony\Component\EventDispatcher\EventDispatcherInterface::dispatch()\" method with the event name as the first argument is deprecated since Symfony 4.3, pass it as the second argument and provide the event object as the first argument instead. at /var/www/kimai2/vendor/symfony/event-dispatcher/EventDispatcher.php:58)"} [] [2019-09-04 15:07:02] security.DEBUG: Stored the security token in the session. {"key":"_security_secured_area"} []

Please help.

kevinpapst commented 5 years ago

Seems you upgraded your PHP (settings)? It worked in the past, so your settings must have changed since the last monthly export.

Maximum execution time of 30 seconds exceeded

https://www.php.net/manual/en/info.configuration.php#ini.max-execution-time

The HTML code size is larger than pcre.backtrack_limit 1000000

https://www.php.net/manual/en/pcre.configuration.php

I'd recommend to raise the max-execution-time to prevent that the script is killed after 30 seconds.

Maybe you should think about exporting weekly data to reduce the total amount of exported items per run.

rarvin commented 5 years ago

Thanks Kevin, the team sometimes have to extract one month worth of data in one go for analysis purposes. I changed the time setting to 90 seconds and also had to increase the memory setting to 1GB, with this it worked. The number of records that were extracted were 59,000. Thanks for helping.

kevinpapst commented 5 years ago

Yeah, I thought about opening the weekly files and copy&pasting them into one document. But if it works, its nor necessary.

But 90 seconds sounds rather good for that amount of data, wasn't it around 6 minutes last time? Did you change the hardware? Does the new form work better?

rarvin commented 5 years ago

no hardware changes were made. export once triggered was always fast, the problem was that one couldnt do the export since the selection were only enabled after the entire list of records were rendered on the browser. Since you have changed the scheme we can easily select the range and then hit the respective download without waiting for it to start.

The new forms are indeed better now, really usable.

kevinpapst commented 5 years ago

Aha, well then the change was really necessary ;-) Very good that you report all these smaller (and bigger) problems and possible optimizations!

lock[bot] commented 4 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. If you use Kimai on a daily basis, please consider donating to support further development of Kimai.