8zero2 commented 5 years ago

Issue type

I am using an exec program(php) to get the output pairs as CoA.
I was using 3.0.16 in that it used to crash only when the server is run in threaded mode, and not in debugging. But in the latest tree of v3.0.x it crashes in both. Following are the outputs from the latest tree

Defect

How to reproduce the issue

Write a PHP code which is run through exec module

I had something like the following:

exec coa_attributes {

wait = yes

input_pairs = request

output_pairs = coa

program = "/usr/bin/php /path/to/program '%{User-Name}' '%{control:Plan-Id}' 'allcoa' '%{%{Acct-Multi-Session-Id}:-%{Acct-Session-ID}}' '%{NAS-IP-Address}'"

shell_escape = yes

timeout = 10

 }

In default server, the following:

       update coa {

                User-Name="%{User-Name}"

                Acct-Session-Id="%{%{Acct-Multi-Session-Id}:-%{Acct-Session-ID}}"

        }

        coa_attributes

radiusd -fxxl stdout OUTPUT


CONSISTENCY CHECK FAILED src/main/process.c[4093]: Expected VALUE_PAIR "Context-Name" to be parented by 0x15d3b70 (RADIUS_PACKET), instead parented by 0x15d39c0 (REQUEST)

Talloc chunk lineage: 0x15d3b70 (RADIUS_PACKET) < 0x15d39c0 (REQUEST) < 0x15d3710 (auth_listener_pool) Talloc context level 0: Talloc chunk lineage: 0x15d39c0 (REQUEST) < 0x15d3710 (auth_listener_pool) Talloc context level 0: SOFT ASSERT FAILED src/lib/pair.c[2437]: 0 CAUGHT SIGNAL: Aborted Backtrace of last 11 frames: /usr/local1/lib/libfreeradius-radius.so(fr_fault+0x12c)[0x7f113faae2cb] /usr/local1/lib/libfreeradius-radius.so(fr_assert_cond+0x4c)[0x7f113faaee81] /usr/local1/lib/libfreeradius-radius.so(fr_pair_list_verify+0x126)[0x7f113fac0db4] /usr/local1/lib/libfreeradius-server.so(+0x26600)[0x7f113fd15600] /usr/local1/lib/libfreeradius-server.so(verify_request+0x124)[0x7f113fd1572d] /usr/local1/sbin/radiusd[0x44785e] /usr/local1/sbin/radiusd[0x440ad3] /usr/local1/sbin/radiusd[0x44128f] /usr/local1/sbin/radiusd[0x43ca0b] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f113edc66ba] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f113e8c441d] No panic action set Aborted

## Full backtrace from LLDB or GDB

```text
Starting program: /usr/local1/sbin/radiusd -f
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffebe69700 (LWP 44596)]
[New Thread 0x7fffeb668700 (LWP 44597)]
[New Thread 0x7fffeae67700 (LWP 44598)]
[New Thread 0x7fffea666700 (LWP 44599)]
[New Thread 0x7fffe9e65700 (LWP 44600)]
[New Thread 0x7fffe9664700 (LWP 44601)]
[New Thread 0x7fffe8e63700 (LWP 44602)]
[New Thread 0x7fffe8662700 (LWP 44603)]
[New Thread 0x7fffe7e61700 (LWP 44604)]
[New Thread 0x7fffe7660700 (LWP 44605)]
[New Thread 0x7fffe6e5f700 (LWP 44606)]
[New Thread 0x7fffe665e700 (LWP 44607)]
[New Thread 0x7fffe5e5d700 (LWP 44608)]
[New Thread 0x7fffe565c700 (LWP 44609)]
[New Thread 0x7fffe4e5b700 (LWP 44610)]
[New Thread 0x7fffe465a700 (LWP 44611)]
[New Thread 0x7fffe3e59700 (LWP 44612)]
[New Thread 0x7fffe3658700 (LWP 44613)]
[New Thread 0x7fffe2e57700 (LWP 44614)]
[New Thread 0x7fffe2656700 (LWP 44615)]
[New Thread 0x7fffe1e55700 (LWP 44616)]
[New Thread 0x7fffe1654700 (LWP 44617)]
[New Thread 0x7fffe0e53700 (LWP 44618)]
[New Thread 0x7fffe0652700 (LWP 44619)]
[New Thread 0x7fffdfe51700 (LWP 44620)]
[New Thread 0x7fffdf650700 (LWP 44621)]
[New Thread 0x7fffdee4f700 (LWP 44622)]
[New Thread 0x7fffde64e700 (LWP 44623)]
[New Thread 0x7fffdde4d700 (LWP 44624)]
[New Thread 0x7fffdd64c700 (LWP 44625)]
[New Thread 0x7fffdce4b700 (LWP 44626)]
[New Thread 0x7fffdc64a700 (LWP 44627)]
[New Thread 0x7fffdbe49700 (LWP 44628)]
[New Thread 0x7fffdb648700 (LWP 44629)]
[New Thread 0x7fffdae47700 (LWP 44630)]
[New Thread 0x7fffda646700 (LWP 44631)]
[New Thread 0x7fffd9e45700 (LWP 44632)]
[New Thread 0x7fffd9644700 (LWP 44633)]
[New Thread 0x7fffd8e43700 (LWP 44634)]
[New Thread 0x7fffd8642700 (LWP 44635)]
[New Thread 0x7fffd7e41700 (LWP 44636)]
[New Thread 0x7fffd7640700 (LWP 44637)]
[New Thread 0x7fffd6e3f700 (LWP 44638)]
[New Thread 0x7fffd663e700 (LWP 44639)]
[New Thread 0x7fffd5e3d700 (LWP 44640)]
[New Thread 0x7fffd563c700 (LWP 44641)]
[New Thread 0x7fffd4e3b700 (LWP 44642)]
[New Thread 0x7fffd463a700 (LWP 44643)]
[New Thread 0x7fffd3e39700 (LWP 44644)]
[New Thread 0x7fffd3638700 (LWP 44645)]
[New Thread 0x7fffd2e37700 (LWP 44646)]
[New Thread 0x7fffd2636700 (LWP 44647)]
[New Thread 0x7fffd1e35700 (LWP 44648)]
[New Thread 0x7fffd1634700 (LWP 44649)]
[New Thread 0x7fffd0e33700 (LWP 44650)]
[New Thread 0x7fffd0632700 (LWP 44651)]
[New Thread 0x7fffcfe31700 (LWP 44652)]
[New Thread 0x7fffcf630700 (LWP 44653)]
[New Thread 0x7fffcee2f700 (LWP 44654)]
[New Thread 0x7fffce62e700 (LWP 44655)]
[New Thread 0x7fffcde2d700 (LWP 44656)]
[New Thread 0x7fffcd62c700 (LWP 44657)]
[New Thread 0x7fffcce2b700 (LWP 44658)]
[New Thread 0x7fffcc62a700 (LWP 44659)]
[New Thread 0x7fffcbe29700 (LWP 44660)]
[New Thread 0x7fffcb628700 (LWP 44661)]
[New Thread 0x7fffcae27700 (LWP 44662)]
[New Thread 0x7fffca626700 (LWP 44663)]
[New Thread 0x7fffc9e25700 (LWP 44664)]
[New Thread 0x7fffc9624700 (LWP 44665)]
[New Thread 0x7fffc8e23700 (LWP 44666)]
[New Thread 0x7fffc8622700 (LWP 44667)]
[New Thread 0x7fffc7e21700 (LWP 44668)]
[New Thread 0x7fffc7620700 (LWP 44669)]
[New Thread 0x7fffc6e1f700 (LWP 44670)]
[New Thread 0x7fffc661e700 (LWP 44671)]
[New Thread 0x7fffc5e1d700 (LWP 44672)]
[New Thread 0x7fffc561c700 (LWP 44673)]
[New Thread 0x7fffc4e1b700 (LWP 44674)]
[New Thread 0x7fffc461a700 (LWP 44675)]
[New Thread 0x7fffc3e19700 (LWP 44676)]
[New Thread 0x7fffc3618700 (LWP 44677)]
[New Thread 0x7fffc2e17700 (LWP 44678)]
[New Thread 0x7fffc2616700 (LWP 44679)]
[New Thread 0x7fffc1e15700 (LWP 44680)]
[New Thread 0x7fffc1614700 (LWP 44681)]
[New Thread 0x7fffc0e13700 (LWP 44682)]
[New Thread 0x7fffc0612700 (LWP 44683)]
[New Thread 0x7fffbfe11700 (LWP 44684)]
[New Thread 0x7fffbf610700 (LWP 44685)]
[New Thread 0x7fffbee0f700 (LWP 44686)]
[New Thread 0x7fffbe60e700 (LWP 44687)]
[New Thread 0x7fffbde0d700 (LWP 44688)]
[New Thread 0x7fffbd60c700 (LWP 44689)]
[New Thread 0x7fffbce0b700 (LWP 44690)]
[New Thread 0x7fffbc60a700 (LWP 44691)]
[New Thread 0x7fffbbe09700 (LWP 44692)]
[New Thread 0x7fffbb608700 (LWP 44693)]
[New Thread 0x7fffbae07700 (LWP 44694)]
[New Thread 0x7fffba606700 (LWP 44695)]

Thread 33 "radiusd" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffdc64a700 (LWP 44627)]
0x00007ffff6c77269 in raise (sig=6) at ../sysdeps/unix/sysv/linux/pt-raise.c:35
35  ../sysdeps/unix/sysv/linux/pt-raise.c: No such file or directory.

alandekok commented 5 years ago

It should no longer crash in the VERIFY code.

8zero2 commented 5 years ago

Thanks man! Now runs fine in debug mode(earlier it was crashing in both)

But still crashing in threaded mode.

radiusd -fxxl stdout OUTPUT

(25) Executing: /usr/local/etc/raddb/support_scripts/date.sh 0: date: write error: Broken pipe Segmentation fault

backtrace

Starting program: /usr/local1/sbin/radiusd -fxx
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffebe69700 (LWP 37879)]
[New Thread 0x7fffeb668700 (LWP 37880)]
[New Thread 0x7fffeae67700 (LWP 37881)]
[New Thread 0x7fffea666700 (LWP 37882)]
[New Thread 0x7fffe9e65700 (LWP 37883)]
[New Thread 0x7fffe9664700 (LWP 37884)]
[New Thread 0x7fffe8e63700 (LWP 37885)]
[New Thread 0x7fffe8662700 (LWP 37886)]
[New Thread 0x7fffe7e61700 (LWP 37887)]
[New Thread 0x7fffe7660700 (LWP 37888)]
[New Thread 0x7fffe6e5f700 (LWP 37889)]
[New Thread 0x7fffe665e700 (LWP 37890)]
[New Thread 0x7fffe5e5d700 (LWP 37891)]
[New Thread 0x7fffe565c700 (LWP 37892)]
[New Thread 0x7fffe4e5b700 (LWP 37893)]
[New Thread 0x7fffe465a700 (LWP 37894)]
[New Thread 0x7fffe3e59700 (LWP 37895)]
[New Thread 0x7fffe3658700 (LWP 37896)]
[New Thread 0x7fffe2e57700 (LWP 37897)]
[New Thread 0x7fffe2656700 (LWP 37898)]
[New Thread 0x7fffe1e55700 (LWP 37899)]
[New Thread 0x7fffe1654700 (LWP 37900)]
[New Thread 0x7fffe0e53700 (LWP 37901)]
[New Thread 0x7fffe0652700 (LWP 37902)]
[New Thread 0x7fffdfe51700 (LWP 37903)]
[New Thread 0x7fffdf650700 (LWP 37904)]
[New Thread 0x7fffdee4f700 (LWP 37905)]
[New Thread 0x7fffde64e700 (LWP 37906)]
[New Thread 0x7fffdde4d700 (LWP 37907)]
[New Thread 0x7fffdd64c700 (LWP 37908)]
[New Thread 0x7fffdce4b700 (LWP 37909)]
[New Thread 0x7fffdc64a700 (LWP 37910)]
[New Thread 0x7fffdbe49700 (LWP 37911)]
[New Thread 0x7fffdb648700 (LWP 37912)]
[New Thread 0x7fffdae47700 (LWP 37913)]
[New Thread 0x7fffda646700 (LWP 37914)]
[New Thread 0x7fffd9e45700 (LWP 37915)]
[New Thread 0x7fffd9644700 (LWP 37916)]
[New Thread 0x7fffd8e43700 (LWP 37917)]
[New Thread 0x7fffd8642700 (LWP 37918)]
[New Thread 0x7fffd7e41700 (LWP 37919)]
[New Thread 0x7fffd7640700 (LWP 37920)]
[New Thread 0x7fffd6e3f700 (LWP 37921)]
[New Thread 0x7fffd663e700 (LWP 37922)]
[New Thread 0x7fffd5e3d700 (LWP 37923)]
[New Thread 0x7fffd563c700 (LWP 37924)]
[New Thread 0x7fffd4e3b700 (LWP 37925)]
[New Thread 0x7fffd463a700 (LWP 37926)]
[New Thread 0x7fffd3e39700 (LWP 37927)]
[New Thread 0x7fffd3638700 (LWP 37928)]
[New Thread 0x7fffd2e37700 (LWP 37929)]
[New Thread 0x7fffd2636700 (LWP 37930)]
[New Thread 0x7fffd1e35700 (LWP 37931)]
[New Thread 0x7fffd1634700 (LWP 37932)]
[New Thread 0x7fffd0e33700 (LWP 37933)]
[New Thread 0x7fffd0632700 (LWP 37934)]
[New Thread 0x7fffcfe31700 (LWP 37935)]
[New Thread 0x7fffcf630700 (LWP 37936)]
[New Thread 0x7fffcee2f700 (LWP 37937)]
[New Thread 0x7fffce62e700 (LWP 37938)]
[New Thread 0x7fffcde2d700 (LWP 37939)]
[New Thread 0x7fffcd62c700 (LWP 37940)]
[New Thread 0x7fffcce2b700 (LWP 37941)]
[New Thread 0x7fffcc62a700 (LWP 37942)]
[New Thread 0x7fffcbe29700 (LWP 37943)]
[New Thread 0x7fffcb628700 (LWP 37944)]
[New Thread 0x7fffcae27700 (LWP 37945)]
[New Thread 0x7fffca626700 (LWP 37946)]
[New Thread 0x7fffc9e25700 (LWP 37947)]
[New Thread 0x7fffc9624700 (LWP 37948)]
[New Thread 0x7fffc8e23700 (LWP 37949)]
[New Thread 0x7fffc8622700 (LWP 37950)]
[New Thread 0x7fffc7e21700 (LWP 37951)]
[New Thread 0x7fffc7620700 (LWP 37952)]
[New Thread 0x7fffc6e1f700 (LWP 37953)]
[New Thread 0x7fffc661e700 (LWP 37954)]
[New Thread 0x7fffc5e1d700 (LWP 37955)]
[New Thread 0x7fffc561c700 (LWP 37956)]
[New Thread 0x7fffc4e1b700 (LWP 37957)]
[New Thread 0x7fffc461a700 (LWP 37958)]
[New Thread 0x7fffc3e19700 (LWP 37959)]
[New Thread 0x7fffc3618700 (LWP 37960)]
[New Thread 0x7fffc2e17700 (LWP 37961)]
[New Thread 0x7fffc2616700 (LWP 37962)]
[New Thread 0x7fffc1e15700 (LWP 37963)]
[New Thread 0x7fffc1614700 (LWP 37964)]
[New Thread 0x7fffc0e13700 (LWP 37965)]
[New Thread 0x7fffc0612700 (LWP 37966)]
[New Thread 0x7fffbfe11700 (LWP 37967)]
[New Thread 0x7fffbf610700 (LWP 37968)]
[New Thread 0x7fffbee0f700 (LWP 37969)]
[New Thread 0x7fffbe60e700 (LWP 37970)]
[New Thread 0x7fffbde0d700 (LWP 37971)]
[New Thread 0x7fffbd60c700 (LWP 37972)]
[New Thread 0x7fffbce0b700 (LWP 37973)]
[New Thread 0x7fffbc60a700 (LWP 37974)]
[New Thread 0x7fffbbe09700 (LWP 37975)]
[New Thread 0x7fffbb608700 (LWP 37976)]
[New Thread 0x7fffbae07700 (LWP 37977)]
[New Thread 0x7fffba606700 (LWP 37978)]

Thread 80 "radiusd" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffc4e1b700 (LWP 37957)]
0x00007ffff6699428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.

alandekok commented 5 years ago

Please post the FULL back trace as suggested in doc/bugs. Posting 100 lines of new thread text isn't helpful/

8zero2 commented 5 years ago

Thread 27 died

Starting program: /usr/local1/sbin/radiusd -f
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffebe69700 (LWP 125193)]
[New Thread 0x7fffeb668700 (LWP 125194)]
[New Thread 0x7fffeae67700 (LWP 125195)]
[New Thread 0x7fffea666700 (LWP 125196)]
[New Thread 0x7fffe9e65700 (LWP 125197)]
[New Thread 0x7fffe9664700 (LWP 125198)]
[New Thread 0x7fffe8e63700 (LWP 125199)]
[New Thread 0x7fffe8662700 (LWP 125200)]
[New Thread 0x7fffe7e61700 (LWP 125201)]
[New Thread 0x7fffe7660700 (LWP 125202)]
[New Thread 0x7fffe6e5f700 (LWP 125203)]
[New Thread 0x7fffe665e700 (LWP 125204)]
[New Thread 0x7fffe5e5d700 (LWP 125205)]
[New Thread 0x7fffe565c700 (LWP 125206)]
[New Thread 0x7fffe4e5b700 (LWP 125207)]
[New Thread 0x7fffe465a700 (LWP 125208)]
[New Thread 0x7fffe3e59700 (LWP 125209)]
[New Thread 0x7fffe3658700 (LWP 125210)]
[New Thread 0x7fffe2e57700 (LWP 125211)]
[New Thread 0x7fffe2656700 (LWP 125212)]
[New Thread 0x7fffe1e55700 (LWP 125213)]
[New Thread 0x7fffe1654700 (LWP 125214)]
[New Thread 0x7fffe0e53700 (LWP 125215)]
[New Thread 0x7fffe0652700 (LWP 125216)]
[New Thread 0x7fffdfe51700 (LWP 125217)]
[New Thread 0x7fffdf650700 (LWP 125218)]
[New Thread 0x7fffdee4f700 (LWP 125219)]
[New Thread 0x7fffde64e700 (LWP 125220)]
[New Thread 0x7fffdde4d700 (LWP 125221)]
[New Thread 0x7fffdd64c700 (LWP 125222)]
[New Thread 0x7fffdce4b700 (LWP 125223)]
[New Thread 0x7fffdc64a700 (LWP 125224)]
[New Thread 0x7fffdbe49700 (LWP 125225)]
[New Thread 0x7fffdb648700 (LWP 125226)]
[New Thread 0x7fffdae47700 (LWP 125227)]
[New Thread 0x7fffda646700 (LWP 125228)]
[New Thread 0x7fffd9e45700 (LWP 125229)]
[New Thread 0x7fffd9644700 (LWP 125230)]
[New Thread 0x7fffd8e43700 (LWP 125231)]
[New Thread 0x7fffd8642700 (LWP 125232)]
[New Thread 0x7fffd7e41700 (LWP 125233)]
[New Thread 0x7fffd7640700 (LWP 125234)]
[New Thread 0x7fffd6e3f700 (LWP 125235)]
[New Thread 0x7fffd663e700 (LWP 125236)]
[New Thread 0x7fffd5e3d700 (LWP 125237)]
[New Thread 0x7fffd563c700 (LWP 125238)]
[New Thread 0x7fffd4e3b700 (LWP 125239)]
[New Thread 0x7fffd463a700 (LWP 125240)]
[New Thread 0x7fffd3e39700 (LWP 125241)]
[New Thread 0x7fffd3638700 (LWP 125242)]
[New Thread 0x7fffd2e37700 (LWP 125243)]
[New Thread 0x7fffd2636700 (LWP 125244)]
[New Thread 0x7fffd1e35700 (LWP 125245)]
[New Thread 0x7fffd1634700 (LWP 125246)]
[New Thread 0x7fffd0e33700 (LWP 125247)]
[New Thread 0x7fffd0632700 (LWP 125248)]
[New Thread 0x7fffcfe31700 (LWP 125249)]
[New Thread 0x7fffcf630700 (LWP 125250)]
[New Thread 0x7fffcee2f700 (LWP 125251)]
[New Thread 0x7fffce62e700 (LWP 125252)]
[New Thread 0x7fffcde2d700 (LWP 125253)]
[New Thread 0x7fffcd62c700 (LWP 125254)]
[New Thread 0x7fffcce2b700 (LWP 125255)]
[New Thread 0x7fffcc62a700 (LWP 125256)]
[New Thread 0x7fffcbe29700 (LWP 125257)]
[New Thread 0x7fffcb628700 (LWP 125258)]
[New Thread 0x7fffcae27700 (LWP 125259)]
[New Thread 0x7fffca626700 (LWP 125260)]
[New Thread 0x7fffc9e25700 (LWP 125261)]
[New Thread 0x7fffc9624700 (LWP 125262)]
[New Thread 0x7fffc8e23700 (LWP 125263)]
[New Thread 0x7fffc8622700 (LWP 125264)]
[New Thread 0x7fffc7e21700 (LWP 125265)]
[New Thread 0x7fffc7620700 (LWP 125266)]
[New Thread 0x7fffc6e1f700 (LWP 125267)]
[New Thread 0x7fffc661e700 (LWP 125268)]
[New Thread 0x7fffc5e1d700 (LWP 125269)]
[New Thread 0x7fffc561c700 (LWP 125270)]
[New Thread 0x7fffc4e1b700 (LWP 125271)]
[New Thread 0x7fffc461a700 (LWP 125272)]
[New Thread 0x7fffc3e19700 (LWP 125273)]
[New Thread 0x7fffc3618700 (LWP 125274)]
[New Thread 0x7fffc2e17700 (LWP 125275)]
[New Thread 0x7fffc2616700 (LWP 125276)]
[New Thread 0x7fffc1e15700 (LWP 125277)]
[New Thread 0x7fffc1614700 (LWP 125278)]
[New Thread 0x7fffc0e13700 (LWP 125279)]
[New Thread 0x7fffc0612700 (LWP 125280)]
[New Thread 0x7fffbfe11700 (LWP 125281)]
[New Thread 0x7fffbf610700 (LWP 125282)]
[New Thread 0x7fffbee0f700 (LWP 125283)]
[New Thread 0x7fffbe60e700 (LWP 125284)]
[New Thread 0x7fffbde0d700 (LWP 125285)]
[New Thread 0x7fffbd60c700 (LWP 125286)]
[New Thread 0x7fffbce0b700 (LWP 125287)]
[New Thread 0x7fffbc60a700 (LWP 125288)]
[New Thread 0x7fffbbe09700 (LWP 125289)]
[New Thread 0x7fffbb608700 (LWP 125290)]
[New Thread 0x7fffbae07700 (LWP 125291)]
[New Thread 0x7fffba606700 (LWP 125292)]

Thread 27 "radiusd" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffdf650700 (LWP 125218)]
0x00007ffff6699428 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/unix/sysv/linux/raise.c:54
54  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
  Id   Target Id         Frame 
  1    Thread 0x7ffff7fe9740 (LWP 125093) "radiusd" 0x00007ffff67615d3 in select
 () at ../sysdeps/unix/syscall-template.S:84
  2    Thread 0x7fffebe69700 (LWP 125193) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  3    Thread 0x7fffeb668700 (LWP 125194) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  4    Thread 0x7fffeae67700 (LWP 125195) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  5    Thread 0x7fffea666700 (LWP 125196) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  6    Thread 0x7fffe9e65700 (LWP 125197) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  7    Thread 0x7fffe9664700 (LWP 125198) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  8    Thread 0x7fffe8e63700 (LWP 125199) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  9    Thread 0x7fffe8662700 (LWP 125200) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  10   Thread 0x7fffe7e61700 (LWP 125201) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  11   Thread 0x7fffe7660700 (LWP 125202) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  12   Thread 0x7fffe6e5f700 (LWP 125203) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  13   Thread 0x7fffe665e700 (LWP 125204) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  14   Thread 0x7fffe5e5d700 (LWP 125205) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  15   Thread 0x7fffe565c700 (LWP 125206) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  16   Thread 0x7fffe4e5b700 (LWP 125207) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  17   Thread 0x7fffe465a700 (LWP 125208) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  18   Thread 0x7fffe3e59700 (LWP 125209) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  19   Thread 0x7fffe3658700 (LWP 125210) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  20   Thread 0x7fffe2e57700 (LWP 125211) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  21   Thread 0x7fffe2656700 (LWP 125212) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  22   Thread 0x7fffe1e55700 (LWP 125213) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  23   Thread 0x7fffe1654700 (LWP 125214) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  24   Thread 0x7fffe0e53700 (LWP 125215) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  25   Thread 0x7fffe0652700 (LWP 125216) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  26   Thread 0x7fffdfe51700 (LWP 125217) "radiusd" 0x00007ffff6c75827 in futex_
abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, 
    futex_word=0x6873e8 <thread_pool+168>)
    at ../sysdeps/unix/sysv/linux/futex-internal.h:205
* 27   Thread 0x7fffdf650700 (LWP 125218) "radiusd" 0x00007ffff6699428 in __GI_r
aise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54

Thread 27 (Thread 0x7fffdf650700 (LWP 125218)):
#0  0x00007ffff6699428 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/unix/sysv/linux/raise.c:54
        resultvar = 0
        pid = 125093
        selftid = 125218
#1  0x00007ffff669b02a in __GI_abort () at abort.c:89
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, 
          sa_mask = {__val = {0, 140737328914231, 3, 140736941306382, 2, 
              140737328915547, 1, 140737328919593, 3, 60, 140737328320509, 60, 
              140736941306416, 0, 140737328881280, 39}}, sa_flags = 0, 
          sa_restorer = 0x7fffdf650700}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x00007ffff7955650 in _fr_talloc_fault_simple (
    reason=0x7ffff70974f8 "Bad talloc magic value - unknown value")
    at src/lib/debug.c:782
No locals.
#3  0x00007ffff708d6fd in _talloc_zero ()
   from /usr/lib/x86_64-linux-gnu/libtalloc.so.2
No symbol table info available.
#4  0x00007ffff7963894 in fr_pair_alloc (ctx=0x7fff54003310)
    at src/lib/pair.c:52
        vp = 0xdf64bae0
#5  0x00007ffff7963934 in fr_pair_afrom_da (ctx=0x7fff54003310, da=0x738260)
    at src/lib/pair.c:88
        vp = 0x2def2f6e9d925200
#6  0x00007ffff7965d4f in fr_pair_make (ctx=0x7fff54003310, vps=0x0, 
    attribute=0x7fffdf64b9e0 "Dynamic-Qos-Param", 
    value=0x7fffdf64bae0 "pwfq-priority-group-rate 0 rate-absolute 1440", 
    op=T_OP_EQ) at src/lib/pair.c:1496
        da = 0x738260
        vp = 0x7fff54003b40
        tc = 0x7fffdf64bdd0 ""
        ts = 0x0
        tag = -128 '\200'
        found_tag = false
        buffer = "(\271d\337\377\177\000\000\000\000\000\000\n", '\000' <repeats 11 times>, "c\205\227\367\377\177", '\000' <repeats 18 times>, "`+\271\367\37
7\177\000\000\001\000\000\000\b\000\000\000p\271d\337\377\177\000\000X\271d\337\377\177\000\000P\271d\337\377\177'\000\000\000\000\000\006\000\000\000\016\273
d\337\377\177\000\000\302\277d\337\377\177\000\000\300+\271\367\377\177\000\000\000R\222\235n/\357- \271d\337\377\177\000\000\067\206\227\367\377\177\000\000\
240\271d\337\f\001\000\000\000p\230\367\b\000\000\000p\271d\337\377\177\000\000X\271d\337\377\177\000\000\200\271d\337\377\177\000\000\210e\226\367\377\177\00
0\000\340\271d\337\377\177\000\000"...
        attrname = 0x7fffdf64b9e0 "Dynamic-Qos-Param"
#7  0x00007ffff7966818 in fr_pair_list_afrom_str (ctx=0x7fff54003310, 
    buffer=0x7fffdf64bf80 "Dynamic-Qos-Param='pwfq-priority-group-rate 0 rate-absolute 1440',Dynamic-Qos-Param='pwfq-priority-group-rate 1 rate-absolute 1600'
,Dynamic-Qos-Param='pwfq-priority-group-rate 2 rate-absolute 1600',Dy"..., 
    list=0x7fffdf64cfd8) at src/lib/pair.c:1891
        vp = 0x7fff54003b40
        head = 0x0
        tail = 0x7fffdf64b9c0
        p = 0x7fffdf64bfc2 "Dynamic-Qos-Param='pwfq-priority-group-rate 1 rate-absolute 1600',Dynamic-Qos-Param='pwfq-priority-group-rate 2 rate-absolute 1600
',Dynamic-Qos-Param='pwfq-circuit-rate-max 1600',Dynamic-Qos-Param='po"...
        last_token = T_COMMA
        raw = {
          l_opand = "Dynamic-Qos-Param", '\000' <repeats 39 times>, "\020\000\000\000\060\000\000\000\200\300d\337\377\177\000\000\300\277d\337\377\177", '\00
0' <repeats 66 times>, "/usr/bin/php\000/var/www/html/radiusnew/rule_checker.php\000gh"..., 
          r_opand = "pwfq-priority-group-rate 0 rate-absolute 1440", '\000' <repeats 195 times>..., quote = T_SINGLE_QUOTED_STRING, op = T_OP_EQ}
#8  0x00007ffff7bab2c8 in radius_exec_program (ctx=0x7fff54003310, 
    out=0x7fffdf64d000 "", outlen=1024, output_pairs=0x7fffdf64cfd8, 
    request=0xe8e810, 
    cmd=0x9d9710 "/usr/bin/php /var/www/html/radiusnew/psrule_calculator.php '%{User-Name}' '%{control:Plan-Id}' '%{%{Acct-Multi-Session-Id}:-%{Acct-Session-I
D}}' '%{NAS-IP-Address}'", input_pairs=0xe8eac0, exec_wait=true, 
    shell_escape=true, timeout=10) at src/main/exec.c:606
        pid = 125517
        from_child = 117
        p = 0x7fffdf64c1e7 ""
        child_pid = 125513
        comma = 0
        status = 0
        ret = 0
        len = 615
        answer = "Dynamic-Qos-Param='pwfq-priority-group-rate 0 rate-absolute 1440',Dynamic-Qos-Param='pwfq-priority-group-rate 1 rate-absolute 1600',Dynamic-
Qos-Param='pwfq-priority-group-rate 2 rate-absolute 1600',Dy"...
#9  0x00007ffff5600815 in mod_exec_dispatch (instance=0x9d95e0, 
    request=0xe8e810) at src/modules/rlm_exec/rlm_exec.c:345
        inst = 0x9d95e0
        rcode = RLM_MODULE_OK
        status = 0
        input_pairs = 0xe8e4e8
        output_pairs = 0x7fff54003388
        answer = 0x0
        ctx = 0x7fff54003310
        out = "\000\356\350\000\000\000\000\000\240\001k\000\000\000\000\000\"\320d\337\001\000\000\000\020\350\350\000\000\000\000\000`\320d\337\306\000\000\
000\360\027\230\367\377\177\000\000\220\066\000T\377\177\000\000\032\000\000\000\000\000\000\000\031\000\000\000\000\000\000\000Fz\226\367\377\177\000\000\000
\ae\337\377\177\000\000\000R\222\235n/\357-\320\320d\337\377\177\000\000Fz\226\367\377\177\000\000\000\ae\337\377\177\000\000\320\355\350\000\000\000\000\000\
360\320d\337\265\001\000\000\354\201\230\367\377\177\000\000\031\000\000\000\000\000\000\000\240\001k\000\000\000\000\000\000\ae\337\376\000\000\000\360\027\2
30\367\377\177\000\000\031\000\000\000\000\000\000\000\320\355\350\000\000\000\000\000\000\ae\337\377\177\000\000\000"...
#10 0x00007ffff5600b56 in mod_accounting (instance=0x9d95e0, request=0xe8e810)
    at src/modules/rlm_exec/rlm_exec.c:439
        inst = 0x9d95e0
        status = 32767
        out = "@\325d\337\377\177\000\000P\363j\000\000\000\000\000\240\324d\337\377\177\000\000A\331\225\367\377\177\000\000@\325d\337\377\177\000\000Y\325d\
337}<\255\026\300\373s\000\000\000\000\000\360dj\000\000\000\000\000P\027\202\000\000\000\000\000\320\063k\000\000\000\000\000\320\324d\337\377\177\000\000\02
7\340\225\367\377\177\000\000@\325d\337\377\177\000\000\360dj\000\000\000\000\000P\325d\337h\265<\276h\005\000\000}<\255\026\020\325d\337\377\177\000\000\006\
341\225\367\377\177\000\000@\325d\337\377\177\000\000\360dj\000\000\000\000\000P\325d\337\377\177\000\000P\363j\000\000\000\000\000\320\063k\000\000\000\000\0
00\000R\222\235n/\357-\340\325d\337\377\177\000\000"...
        we_wait = false
        vp = 0x6af350
#11 0x000000000042b776 in call_modsingle (component=MOD_ACCOUNTING, 
    sp=0xdeb410, request=0xe8e810) at src/main/modcall.c:302
        blocked = 0
        indent = 10
#12 0x000000000042bec7 in modcall_recurse (request=0xe8e810, 
    component=MOD_ACCOUNTING, depth=4, entry=0x7fffdf64fb30, 
    do_next_sibling=true) at src/main/modcall.c:578
        sp = 0xdeb410
        if_taken = false
        was_if = false
        c = 0xdeb410
        priority = -1
        result = RLM_MODULE_OK
#13 0x000000000042b93e in modcall_child (request=0xe8e810, 
    component=MOD_ACCOUNTING, depth=4, entry=0x7fffdf64fb18, c=0xdeb250, 
    result=0x7fffdf64dfc0, do_next_sibling=true) at src/main/modcall.c:408
        next = 0x7fffdf64fb30
#14 0x000000000042c9b6 in modcall_recurse (request=0xe8e810, 
    component=MOD_ACCOUNTING, depth=3, entry=0x7fffdf64fb18, 
    do_next_sibling=true) at src/main/modcall.c:789
        g = 0xdea900
        if_taken = false
        was_if = false
        c = 0xdea900
        priority = -1
        result = RLM_MODULE_UNKNOWN
#15 0x000000000042b93e in modcall_child (request=0xe8e810, 
    component=MOD_ACCOUNTING, depth=3, entry=0x7fffdf64fb00, c=0xdea740, 
    result=0x7fffdf64e6a0, do_next_sibling=true) at src/main/modcall.c:408
        next = 0x7fffdf64fb18
#16 0x000000000042c9b6 in modcall_recurse (request=0xe8e810, 
    component=MOD_ACCOUNTING, depth=2, entry=0x7fffdf64fb00, 
    do_next_sibling=true) at src/main/modcall.c:789
        g = 0xdea640
        if_taken = true
        was_if = true
        c = 0xdea640
        priority = -1
        result = RLM_MODULE_UNKNOWN
#17 0x000000000042b93e in modcall_child (request=0xe8e810, 
    component=MOD_ACCOUNTING, depth=2, entry=0x7fffdf64fae8, c=0xde8920, 
    result=0x7fffdf64ed80, do_next_sibling=true) at src/main/modcall.c:408
        next = 0x7fffdf64fb00
#18 0x000000000042c9b6 in modcall_recurse (request=0xe8e810, 
    component=MOD_ACCOUNTING, depth=1, entry=0x7fffdf64fae8, 
    do_next_sibling=true) at src/main/modcall.c:789
        g = 0xde8820
        if_taken = true
        was_if = true
        c = 0xde8820
        priority = -1
        result = RLM_MODULE_OK
#19 0x000000000042b93e in modcall_child (request=0xe8e810, 
    component=MOD_ACCOUNTING, depth=1, entry=0x7fffdf64fad0, c=0xde2740, 
    result=0x7fffdf64f460, do_next_sibling=true) at src/main/modcall.c:408
        next = 0x7fffdf64fae8
#20 0x000000000042c9b6 in modcall_recurse (request=0xe8e810, 
    component=MOD_ACCOUNTING, depth=0, entry=0x7fffdf64fad0, 
    do_next_sibling=true) at src/main/modcall.c:789
        g = 0xde2640
        if_taken = false
        was_if = false
        c = 0xde2640
        priority = -1
        result = RLM_MODULE_UNKNOWN
#21 0x000000000042d828 in modcall (component=MOD_ACCOUNTING, c=0xde2640, 
    request=0xe8e810) at src/main/modcall.c:1134
        stack = {{result = RLM_MODULE_NOOP, priority = 0, unwind = 0, 
            c = 0xde2640}, {result = RLM_MODULE_OK, priority = 2, unwind = 0, 
            c = 0xde8820}, {result = RLM_MODULE_OK, priority = 0, unwind = 0, 
            c = 0xdea640}, {result = RLM_MODULE_OK, priority = 0, unwind = 0, 
            c = 0xdea900}, {result = RLM_MODULE_OK, priority = 2, unwind = 0, 
            c = 0xdeb410}, {result = RLM_MODULE_REJECT, priority = 0, 
            unwind = 0, c = 0x0} <repeats 27 times>}
#22 0x0000000000428c8a in indexed_modcall (comp=MOD_ACCOUNTING, idx=0, 
    request=0xe8e810) at src/main/modules.c:1031
        rcode = 1012
        list = 0xde2640
        server = 0xddc360
#23 0x000000000042b0ae in process_accounting (acct_type=0, request=0xe8e810)
    at src/main/modules.c:2189
No locals.
#24 0x000000000040f1bf in rad_accounting (request=0xe8e810)
    at src/main/acct.c:93
        vp = 0x0
        acct_type = 0
        result = 2
#25 0x000000000044121c in request_running (request=0xe8e810, action=1)
    at src/main/process.c:1571
        __FUNCTION__ = "request_running"
#26 0x000000000043ca0b in request_handler_thread (arg=0xe59a80)
    at src/main/threads.c:826
        self = 0xe59a80
#27 0x00007ffff6c6d6ba in start_thread (arg=0x7fffdf650700)
    at pthread_create.c:333
        __res = <optimized out>
        pd = 0x7fffdf650700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140736941328128, 
                -3694811335437480831, 0, 140737488348111, 140736941328832, 0, 
                3694880664809213057, 3694791345904520321}, 
              mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, 
            data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#28 0x00007ffff676b41d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
No locals.

8zero2 commented 5 years ago

Hi Alan,

The server is still crashing! , Did you get a chance to go through the last trace i sent you. Please help, still facing the same problem.

arr2036 commented 5 years ago

Can you provide a cut down test case for reproducing this?

8zero2 commented 5 years ago

no need to add home server for coa(has same behaviour with or without)

In post-auth or acct

update coa {
                User-Name = &User-Name
                        Acct-Session-Id="%{%{Acct-Multi-Session-Id}:-%{Acct-Session-ID}}"
        }
        coa_gqrs_attributes

coa_gqrs_attributes module

exec coa_gqrs_attributes {
        wait = yes
        input_pairs = request
        output_pairs = coa
        program = "/bin/cat /usr/local/etc/raddb/test"
        shell_escape = yes
        timeout = 10
}

contents of test file

cat test
Dynamic-Qos-Param="meter-class-burst OGERO 64000"

NOTE: Runs well when threading is off

8zero2 commented 5 years ago

Hi,

Did you get time to try this?

alandekok commented 5 years ago

I haven't been able to reproduce this here. How long does it take to crash when you run that test?

8zero2 commented 5 years ago

it happens within seconds or max few minutes. Apologies i wasnt available for few days, Just saw this.

alandekok commented 5 years ago

Probably related to #2304

mcnewton commented 5 years ago

@8zero2 Can you try latest v3.0.x branch please

mcnewton commented 5 years ago

We believe this is fixed now.

FreeRADIUS / freeradius-server

crash when running exec to output coa attributes #2328

Issue type

Defect

How to reproduce the issue

radiusd -fxxl stdout OUTPUT

radiusd -fxxl stdout OUTPUT

backtrace

Thread 27 died

no need to add home server for coa(has same behaviour with or without)

In post-auth or acct

coa_gqrs_attributes module

contents of test file