search

RNO-G / mattak

RNO-G dataformats
1 stars 1 forks source link

pyroot backend crashes on trying to use calibration #60

Closed CamphynR closed 1 week ago

CamphynR commented 3 months ago

When trying to iterate over a mattak.Dataset.Dataset in pyroot, given a calibration file, the program crashes and states: malloc(): invalid size (unsorted). Using the gdb debugger I managed to point to VoltageCalibration.cc line 1015 as the problem: TObject * obj = inputFile->Get(graphNameTitle); More specifically the line crashes on the second iteration of the loop. Commenting out lines 1045-1047 does allow the code to run. These are the upsampling lines

// Upsampling by a factor of 2
resid_volt[j][i*2+1] = resid_volt[j][i*2] + dV / 2;
resid_adc[j][i*2+1] = graph_residAve[j]->Eval(resid_volt[j][i*2+1]);

I tried to do something like this and this does solve the crash but makes the code crash in other places.

resid_volt[j][i*2+1] = resid_volt[j][i*2] + dV / 2;
TGraphErrors * temp = (TGraphErrors*) graph_residAve[j]->Clone();
resid_adc[j][i*2+1] = temp->Eval(resid_volt[j][i*2+1]);
free(temp);

I suspect the allocation of TObject * obj is causing the problem but I am definitely not an expert in C/C++ so any help would be appreciated.


Note that I changed line 88 in mattak/py/mattak/backends/pyroot/dataset.py to, because there was a small bug in there self.set_calibration(cal_file, cache_calibration=cache_calibration)

Minimal code to reproduce problem

import mattak.Dataset

ds = mattak.Dataset.Dataset(station = 0, run = 0, data_path  = /path/to/data/file/station/run,
                            verbose = True, backend = "pyroot", voltage_calibration = /path/to/vc.root)

for evtinfo, wf in ds.iterate(calibrated = True):
    print(wf) # or do something else

System on which crash occurs Python 3.10.12 ROOT 6.28/06 mattak: current main branch (19/06/2024)

Full gdb backtrace


(NuRadioMC_env) ruben@ruben-ThinkPad-X1-Carbon-Gen-11:~/Documents/projects/RNO-G_noise_study$ gdb --args python scripts/minimal_working_pyroot.py 
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...
(No debugging symbols found in python)
(gdb) run
Starting program: /home/ruben/Documents/envs/NuRadioMC_env/bin/python scripts/minimal_working_pyroot.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff3dff640 (LWP 62191)]
[New Thread 0x7ffff35fe640 (LWP 62192)]
[New Thread 0x7fffeedfd640 (LWP 62193)]
[New Thread 0x7fffee5fc640 (LWP 62194)]
[New Thread 0x7fffebdfb640 (LWP 62195)]
[New Thread 0x7fffe75fa640 (LWP 62196)]
[New Thread 0x7fffe4df9640 (LWP 62197)]
[New Thread 0x7fffe45f8640 (LWP 62198)]
[New Thread 0x7fffe1df7640 (LWP 62199)]
[New Thread 0x7fffdf5f6640 (LWP 62200)]
[New Thread 0x7fffdadf5640 (LWP 62201)]
[Detaching after vfork from child process 62202]
[Detaching after vfork from child process 62204]
[Detaching after vfork from child process 62208]
[Detaching after vfork from child process 62210]
[Detaching after vfork from child process 62223]
[New Thread 0x7fffc0bc2640 (LWP 62224)]
[Detaching after vfork from child process 62225]
mattak::Dataset::loadDir ( ~/Documents/data/RNO-G_data/station23/run1603,1) called
about to load waveforms 
 full dataset found 
about to load headers 
 success
about to load daqstatus 
 success
about to load pedestal 
 success
about to load runinfo 
 success
Looking for a calibration file
Found calibration file /home/ruben/Documents/data/RNO-G_data/calibration/station23/volCalConsts_pol9_s23_1683802481-1683803948.root
BAD FITTING WARNING: The average chi2/DOF over all samples of CH6 is greater than 6.0!!!
malloc(): invalid size (unsorted)

Thread 1 "python" received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352519680) at ./nptl/pthread_kill.c:44
44  ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352519680) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737352519680) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737352519680, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7c42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7c89676 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7ddbb77 "%s\n")
    at ../sysdeps/posix/libc_fatal.c:155
#6  0x00007ffff7ca0cfc in malloc_printerr (str=str@entry=0x7ffff7ddebc0 "malloc(): invalid size (unsorted)")
    at ./malloc/malloc.c:5664
#7  0x00007ffff7ca40dc in _int_malloc (av=av@entry=0x7ffff7e1ac80 <main_arena>, bytes=bytes@entry=56)
    at ./malloc/malloc.c:4002
#8  0x00007ffff7ca5139 in __GI___libc_malloc (bytes=56) at ./malloc/malloc.c:3329
#9  0x00007fffd72ae98c in operator new(unsigned long) () from /lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007fffd786e96d in TList::MakeIterator(bool) const ()
   from /home/ruben/Documents/software/root/root_install/lib/libCore.so
#11 0x00007fffd78663e6 in TIter::TIter(TIter const&) ()
   from /home/ruben/Documents/software/root/root_install/lib/libCore.so
#12 0x00007fffd6f3a9eb in TDirectoryFile::Get(char const*) ()
   from /home/ruben/Documents/software/root/root_install/lib/libRIO.so
#13 0x00007fffceeb9466 in mattak::VoltageCalibration::readFitCoeffsFromFile (this=<optimized out>, 
    inFile=<optimized out>, cache_tables=<optimized out>)
    at /home/ruben/Documents/software/mattak/src/VoltageCalibration.cc:1015
#14 0x00007fffd10dd064 in ?? ()
#15 0x00007fffffffc920 in ?? ()
#16 0x000055555c424370 in ?? ()
#17 0x00007fffffffc630 in ?? ()
#18 0x00000002ffffc920 in ?? ()
#19 0x0000000000000002 in ?? ()
#20 0x00007ffff4050a8a in WrapperCall(long, unsigned long, void*, void*, void*) ()
   from /home/ruben/Documents/software/root/root_install/lib/libcppyy_backend3_10.so
#21 0x00007fffd7da70cb in CPyCppyy::(anonymous namespace)::VoidExecutor::Execute(long, void*, CPyCppyy::CallContext*)
    () from /home/ruben/Documents/software/root/root_install/lib/libcppyy3_10.so
#22 0x00007fffd7d93646 in CPyCppyy::CPPMethod::ExecuteFast(void*, long, CPyCppyy::CallContext*) [clone .isra.0] ()
   from /home/ruben/Documents/software/root/root_install/lib/libcppyy3_10.so
#23 0x00007fffd7d961f8 in CPyCppyy::CPPMethod::ExecuteProtected(void*, long, CPyCppyy::CallContext*) ()
   from /home/ruben/Documents/software/root/root_install/lib/libcppyy3_10.so
#24 0x00007fffd7d9412e in CPyCppyy::CPPMethod::Execute(void*, long, CPyCppyy::CallContext*) ()
   from /home/ruben/Documents/software/root/root_install/lib/libcppyy3_10.so
#25 0x00007fffd7d94ea0 in CPyCppyy::CPPMethod::Call(CPyCppyy::CPPInstance*&, _object*, _object*, CPyCppyy::CallContext*)--Type <RET> for more, q to quit, c to continue without paging--c
 () from /home/ruben/Documents/software/root/root_install/lib/libcppyy3_10.so
#26 0x00007fffd7d99797 in CPyCppyy::(anonymous namespace)::mp_call(CPyCppyy::CPPOverload*, _object*, _object*) () from /home/ruben/Documents/software/root/root_install/lib/libcppyy3_10.so
#27 0x00005555556a4a7b in _PyObject_MakeTpCall ()
#28 0x000055555569e150 in _PyEval_EvalFrameDefault ()
#29 0x00005555556bc7f1 in ?? ()
#30 0x000055555569853c in _PyEval_EvalFrameDefault ()
#31 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#32 0x00005555556a3cbd in _PyObject_FastCallDictTstate ()
#33 0x00005555556b8a64 in ?? ()
#34 0x00005555556a4a1c in _PyObject_MakeTpCall ()
#35 0x000055555569e150 in _PyEval_EvalFrameDefault ()
#36 0x00005555556ae9fc in _PyFunction_Vectorcall ()
#37 0x000055555569853c in _PyEval_EvalFrameDefault ()
#38 0x00005555556939c6 in ?? ()
#39 0x0000555555789256 in PyEval_EvalCode ()
#40 0x00005555557b4108 in ?? ()
#41 0x00005555557ad9cb in ?? ()
#42 0x00005555557b3e55 in ?? ()
#43 0x00005555557b3338 in _PyRun_SimpleFileObject ()
#44 0x00005555557b2f83 in _PyRun_AnyFileObject ()
#45 0x00005555557a5a5e in Py_RunMain ()
#46 0x000055555577c02d in Py_BytesMain ()
#47 0x00007ffff7c29d90 in __libc_start_call_main (main=main@entry=0x55555577bff0, argc=argc@entry=2, argv=argv@entry=0x7fffffffd838) at ../sysdeps/nptl/libc_start_call_main.h:58
#48 0x00007ffff7c29e40 in __libc_start_main_impl (main=0x55555577bff0, argc=2, argv=0x7fffffffd838, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd828) at ../csu/libc-start.c:392
#49 0x000055555577bf25 in _start ()
cozzyd commented 3 months ago

At least one of the problems is that on line https://github.com/RNO-G/mattak/blob/main/src/VoltageCalibration.cc#L1036 and the following, the size is wrong (shouldn't be -1).

Probably time to resurrect my memory_fixes branch...

fschlueter commented 1 week ago

Is that fixed by now?

fschlueter commented 1 week ago

I assume this is fixed after merging #39 . Feel free to reopen this issue if the issue persist for you