Perl script failure with "Attempt to free unreferenced scalar"

[reshmavk]

We have a locally created perl script/XS that works with perl 5.34.1. When we updated perl to version 5.38.2, these scripts fail with the following errors

Attempt to free unreferenced scalar: SV 0xd, Perl interpreter: 0x200073a8 at XX.pm line 1224.
panic: unknown regstclass 0 at XX.pm line 1224.

We see this failure because of this commit https://github.com/Perl/perl5/commit/75acd14e43f2ffb698fc7032498f31095b56adb5 Any idea what is causing this issue?

[jkeenan]

We have a locally created perl script/XS that works with perl 5.34.1. When we updated perl to version 5.38.2, these scripts fail with the following errors

Attempt to free unreferenced scalar: SV 0xd, Perl interpreter: 0x200073a8 at XX.pm line 1224.
panic: unknown regstclass 0 at XX.pm line 1224.

We see this failure because of this commit 75acd14 Any idea what is causing this issue?

Can you provide a reduced version of your program that works in 5.34 but generates that panic in 5.38? Thanks.

[reshmavk]

Since it is proprietary code, exact code cannot be shared here. But when something like below is done , we get the error.

sub test {
  my($s,$q)=@_;
  return undef unless defined $s;
  $q = '' unless defined $q;
  return($s =~ m/^$q$/);
}

Perl 5.38.2 is built using xlc 13.1.3 and the platform is AIX 7.3.

[iabyn]

On Thu, Apr 18, 2024 at 10:20:49PM -0700, Reshma V Kumar wrote:

Since it is proprietary code, exact code cannot be shared here. But when something like below is done , we get the error. sub test { my($s,$q)=@_; return undef unless defined $s; $q = '' unless defined $q; return($s =~ m/^$q$/); }

You mention that your program uses XS code. Is this just CPAN XS modules, or does it use private XS code written for the application?

If the latter, it is most likely that there is a bug in the XS code which is being tickled by the newer perl, or that the XS code is doing something on the very limits of what is offered by the perl API and in doing so, is triggering some sort of edge condition.

The sample code you've shown is just simple, common perl, and is unlikely on its own to be triggering any bug.

Unless you can show us a standalone reproducible program, it's very unlikely that we can diagnose this issue. It needn't include any proprietary code. Just keep stripping down your real code, removing all the things not needed to trigger, until you're left with just a few lines of perl and/or XS code. Rename any functions and variables to give them generic names.

If you can't reproduce it in a simple form, at the very least build a DEBUGGING perl and run it under valgrind or ASAN or similar, and see if any issues can be identified.

-- You live and learn (although usually you just live).

[reshmavk]

Thanks for the reply. Yes, it uses private XS code written for the application. I will try to come up with a standalone program that can reproduce this problem. In the meanwhile, can you provide me some thoughts on where to look into in the XS code? (since the issue is seen after this commit https://github.com/Perl/perl5/commit/75acd14e43f2ffb698fc7032498f31095b56adb5 which makes the newSV_type an inline function)

[iabyn]

On Fri, Apr 19, 2024 at 04:25:57AM -0700, Reshma V Kumar wrote:

In the meanwhile, can you provide me some thoughts on where to look into in the XS code? (since the issue is seen after this commit https://github.com/Perl/perl5/commit/75acd14e43f2ffb698fc7032498f31095b56adb5 which makes the newSV_type an inline function)

That commit just changes slightly how new SVs are created. At a guess I would say that the XS is doing something wrong when allocating or freeing SVs, and the changes in that commit just happen to make the bug more obvious.

The first warning message you're seeing happens when two different places hold a reference to an SV, but the SV only has a reference count of 1 rather than 2.

When the first reference is deleted, the SV's ref count goes to zero, and the SV is returned to the free pool. If the second reference is deleted before the SV is reallocated, the freeing code spots that the RC is already at zero and issues a warning. If it's freed after the SV has been reallocated, or not freed at all, then all sorts of strange things can happen.

The second error coming from the regex engine is likely a side-effect of the first issue, and the bug may be unrelated to anything in the regex engine.

In addition to building perl with -DDEBUGGING and using valgrind or ASAN, if you use -DDEBUG_LEAKING_SCALARS too, you get may get more info on where the bad SV was first allocated.

If you use bleadperl (or the 5.39.9 development release) and in addition build with -DPERL_RC_STACK, that will enable an experimental feature which changes how perl handles refcounting of SVs on the stack, which might cause the fault to be detected earlier, or show up in a different way. (Or it might not).

To build a perl with those extra defines, just add them as -Accflags etc argument(s) to Configure, e.g.

$ sh Configure -des -Dusedevel -Doptimize='-g' \ -Accflags='-DPERL_RC_STACK -DDEBUGGING -ggdb -fsanitize=address' \ -Aldflags='-fsanitize=address' \ -Dcc='clang'\ && make test

-- It's not that I'm afraid to die, I just don't want to be there when it happens. -- Woody Allen

[reshmavk]

Thanks for the information. It will be helpful in debugging this issue further.

[reshmavk]

After rebuilding perl 5.38.2 on AIX with -DDEBUGGING and -DDEBUG_LEAKING_SCALARS, our local perl script fails with Segmentation fault(coredump) . Following is the stack trace.

XX.Perl_newSV_type(my_perl = 0x20007428, type = SVt_PVHV), line 417 in "sv_inline.h"
XX_init(), line 209 in "XX.xs"
unnamed block in XS_XX_fp(my_perl = 0x20007428, cv = 0x200fadd8), line 247 in "XX.xs"
XS_XX_fp(my_perl = 0x20007428, cv = 0x200fadd8), line 247 in "XX.xs"
Perl_pp_entersub(my_perl = 0x20007428), line 5555 in "pp_hot.c"
Perl_runops_debug(my_perl = 0x20007428), line 2861 in "dump.c"
S_run_body(my_perl = 0x20007428, oldscope = 1), line 2812 in "perl.c"
perl_run(my_perl = 0x20007428), line 2727 in "perl.c"

Any thoughts?

[tonycoz]

Did you rebuild your XS module for the rebuilt perl?

DEBUG_LEAKING_SCALARS changes the layout of the interpreter structure, which could cause a SEGV like this if the module hasn't been rebuilt.

Looking at the opening post, that SV pointer is strange. At the point this message is produced the SV pointer has been dereferenced many times, but 0xd is almost a NULL pointer and is mis-aligned too.

It might be worth running under a debugger and putting a breakpoint on the line (7280 in sv.c for 5.38.2) to check the value of sv at that point, in case there's some other problem with the formatted output code (which is unlikely).

You could also build with DEBUG_LEAKING_SCALARS_ABORT to produce a core dump immediately after the "Attempt to free unreferenced scalar" message to debug that sv value from a core dump.