search

znuny / Znuny

Znuny/Znuny LTS is a fork of the ((OTRS)) Community Edition, one of the most flexible web-based ticketing systems used for Customer Service, Help Desk, IT Service Management.
https://www.znuny.org
GNU General Public License v3.0
368 stars 86 forks source link

Bug - Runtime errors since upgrading to Perl 5.38.0 #465

Open tgurr opened 1 year ago

tgurr commented 1 year ago

Environment

Expected behaviour

No (visible) errors in either the user/agent interface nor the logs.

Actual behaviour

Errors, with the user/agent visible one is when clicking through the articles a single ticket has an "Internal Server Error" is randomly appearing instead of showing the actual content of the article:

image

How to reproduce

Steps to reproduce the behavior:

  1. Ugrade your system from Perl 5.36.1 to 5.38.0
  2. Click through the articles of a ticket
  3. Randomly experience hitting "Internal Server Error"

Additional information

Apache error log: error_log.txt otrs.CheckModules.pl: modules.txt

Looks like we got the Message: Missing parameter TimeZone. error also before the upgrade so that might be irrelevant but the other one appears to be new:

ERROR: OTRS-CGI-12 Perl: 5.38.0 OS: linux Time: Fri Jul 28 08:22:29 2023

 Message: Kernel::System::Ticket::Article::Backend::Email could not be loaded: Invalid version format (non-numeric data) at /usr/lib64/perl5/vendor_perl/5.38/File/Temp.pm line 176.
BEGIN failed--compilation aborted at /usr/lib64/perl5/vendor_perl/5.38/File/Temp.pm line 176.
Compilation failed in require at /opt/otrs/Kernel/cpan-lib/MIME/Tools.pm line 14.
BEGIN failed--compilation aborted at /opt/otrs/Kernel/cpan-lib/MIME/Tools.pm line 14.
Compilation failed in require at /opt/otrs/Kernel/cpan-lib/MIME/Parser.pm line 140.
BEGIN failed--compilation aborted at /opt/otrs/Kernel/cpan-lib/MIME/Parser.pm line 140.
Compilation failed in require at /opt/otrs/Kernel/System/EmailParser.pm line 17.
BEGIN failed--compilation aborted at /opt/otrs/Kernel/System/EmailParser.pm line 17.
Compilation failed in require at /opt/otrs/Kernel/System/Ticket/Article/Backend/MIMEBase.pm line 17.
BEGIN failed--compilation aborted at /opt/otrs/Kernel/System/Ticket/Article/Backend/MIMEBase.pm line 17.
Compilation failed in require at /opt/otrs/Kernel/cpan-lib/parent.pm line 16.
BEGIN failed--compilation aborted at /opt/otrs/Kernel/System/Ticket/Article/Backend/Email.pm line 19.
Compilation failed in require at /opt/otrs/Kernel/System/ObjectManager.pm line 255.

 RemoteAddress: xxx
 RequestURI: /otrs/index.pl

 Traceback (1806): 
   Module: Kernel::System::ObjectManager::_DieWithError Line: 628
   Module: Kernel::System::ObjectManager::_ObjectBuild Line: 262
   Module: Kernel::System::ObjectManager::Get Line: 201
   Module: Kernel::System::CommunicationChannel::Email::ArticleBackend Line: 100
   Module: Kernel::System::Ticket::Article::BackendForChannel Line: 165
   Module: Kernel::System::Ticket::Article::ArticleSearchableFieldsList Line: 1123
   Module: Kernel::System::Ticket::ArticleSearchIndex::DB::ArticleSearchIndexSQLJoinNeeded Line: 216
   Module: Kernel::System::Ticket::Article::ArticleSearchIndexSQLJoinNeeded Line: 977
   Module: Kernel::System::Ticket::TicketSearch::TicketSearch Line: 512
   Module: Kernel::Output::HTML::Dashboard::TicketGeneric::Run Line: 781
   Module: Kernel::Modules::AgentDashboardCommon::_Element Line: 1002
   Module: Kernel::Modules::AgentDashboardCommon::Run Line: 432
   Module: Kernel::System::Web::InterfaceAgent::Run Line: 1144
   Module: ModPerl::ROOT::ModPerl::Registry::opt_otrs_bin_cgi_2dbin_index_2epl::handler Line: 39
   Module: (eval) (v1.99) Line: 207
   Module: ModPerl::RegistryCooker::run (v1.99) Line: 207
   Module: ModPerl::RegistryCooker::default_handler (v1.99) Line: 173
   Module: ModPerl::Registry::handler (v1.99) Line: 32

There are also various other problems visible in the attached log not sure if they're related and/or severe as well.

Screenshots

Attached above.

Additional observation

[Thu Jul 27 21:20:38 2023] QuotedPrint.pm: Argument "3.16_01" isn't numeric in numeric ge (>=) at /opt/otrs/Kernel/cpan-lib/MIME/Decoder/QuotedPrint.pm line 76.

is also triggered by

OTRS Scheduler Daemon Cron: TicketPendingCheck

rkaldung commented 1 year ago

Hi,

two questions:

1.) Which Linux distribution you use and how was the Perl updated installed (packages or source)? 2.) Have you also checked if any module build with cpan/cpanm needs to be updated?

tgurr commented 1 year ago

1.) Which Linux distribution you use and how was the Perl updated installed (packages or source)?

Gentoo Linux in this case which is a source-based distribution, perl and deps were solely updated via the offical package manager using "packages" they provide.

2.) Have you also checked if any module build with cpan/cpanm needs to be updated?

Dependencies have been updated as well as in I did a full system upgrade perl + deps the distribution offers which also means rebuilding any perl package against the newly installed perl version.

Edit: As I've reverted back to Perl 5.36.1 for now here's a list of how the update from Perl 5.36.1 -> 5.38.0 looks like packaging/versions wise on Gentoo Linux: perl-update-packages.txt

rkaldung commented 1 year ago

@tgurr One more question: have you restarted the web server and daemon after updating Perl and the modules?

tgurr commented 1 year ago

@tgurr One more question: have you restarted the web server and daemon after updating Perl and the modules?

@rkaldung Yes, multiple times and even the whole machine.

Proof also included in the apache error_log attached to the initial report, see snippet:

[...]
[Thu Jul 27 19:57:19.919883 2023] [mpm_event:notice] [pid 1789:tid 140085662100928] AH00489: Apache/2.4.57 (Unix) OpenSSL/1.1.1u mod_perl/2.0.12 Perl/v5.36.1 configured -- resuming normal operations
[Thu Jul 27 19:57:19.920137 2023] [core:notice] [pid 1789:tid 140085662100928] AH00094: Command line: '/usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D LANGUAGE -D HTTP2 -D PERL -d /usr/lib64/apache2 -f /etc/apache2/httpd.conf'
[Thu Jul 27 19:59:33 2023] -e: Use of uninitialized value $AuthenticatedUser in string eq at /opt/otrs/Kernel/System/SupportDataCollector/Plugin/OTRS/DefaultUser.pm line 60.
[Thu Jul 27 20:06:13.072037 2023] [mpm_event:notice] [pid 1789:tid 140085662100928] AH00491: caught SIGTERM, shutting down
[Thu Jul 27 21:13:30.045440 2023] [mpm_event:notice] [pid 1790:tid 140098379288000] AH00489: Apache/2.4.57 (Unix) OpenSSL/3.1.1 mod_perl/2.0.12 Perl/v5.38.0 configured -- resuming normal operations
[Thu Jul 27 21:13:30.046220 2023] [core:notice] [pid 1790:tid 140098379288000] AH00094: Command line: '/usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D LANGUAGE -D HTTP2 -D PERL -d /usr/lib64/apache2 -f /etc/apache2/httpd.conf'
[...]
hanneshal commented 11 months ago

Hi @tgurr

question: does the issue still occur? I'm asking because your stack trace shows:

 Traceback (1806): 
   Module: Kernel::System::DateTime::ToTimeZone Line: 1056
   Module: Kernel::Output::HTML::ResponseDefaultStateOutputFilter::AgentTicketComposeFilter::Run Line: 132
   Module: Kernel::Output::HTML::Layout::Template::Output Line: 287
   Module: Kernel::Modules::AgentTicketCompose::_Mask Line: 2389

Shows a custom module (ResponseDefaultStateOutputFilter) as one error source. If the error is still there, I would like to have a look on your instance, if possible.

Regards Johannes

tgurr commented 11 months ago

@hanneshal sorry my late response, I first had to setup a new test machine. After upgrading a clone of our current production machine to perl 5.38.0 I currently do not seem to be able to reproduce the issue anymore. I'll upgrade our production machine later today and report back if we can still observe the issue or if it's solved for us and the issue can be closed.

tgurr commented 11 months ago

So far I couldn't observe the visual error with "Internal Server Error" anymore yet, however I at least still see what I reported in my initial message for "Additional observation":

Additional observation

[Thu Jul 27 21:20:38 2023] QuotedPrint.pm: Argument "3.16_01" isn't numeric in numeric ge (>=) at /opt/otrs/Kernel/cpan-lib/MIME/Decoder/QuotedPrint.pm line 76.

is also triggered by

OTRS Scheduler Daemon Cron: TicketPendingCheck

Probably introduced by https://github.com/Perl/perl5/commit/4dc2cf45feb548059d2b42b85df91b054a459425 which is included in / part of perl 5.38.0.

tomglx commented 10 months ago

I have the same problem. Since perl 5.38 PDF Generation by printing is no longer possible. Znuny version is 6.5.4. We're getting many errors like this one in our error_log Argument "40,054" isn't numeric in numeric gt (>) at /opt/otrs/Kernel/System/PDF.pm line 2004. Seems, that the comma is the problem. I suspect, that a decimal point is expected.