search

openpreserve / jpylyzer

JP2 (JPEG 2000 Part 1) validator and properties extractor. Jpylyzer was specifically created to check that a JP2 file really conforms to the format's specifications. Additionally jpylyzer is able to extract technical characteristics.
http://jpylyzer.openpreservation.org/
Other
69 stars 28 forks source link

Valgrind reported error #90

Closed boxerab closed 2 months ago

boxerab commented 8 years ago

Dev Effort

1D

Description

I use jpylyzer in some of the unit tests for my jpeg 2000 codec.

I am getting "Conditional jump or move depends on uninitialised value(s)" errors:

http://my.cdash.org/viewDynamicAnalysisFile.php?id=3489290

Do you run valgrind against jpylyzer ?

bitsgalore commented 8 years ago

Hi Aaron,

Not using valgrind myself, but definitely looks like something that's worth checking out. Meanwhile, could you post a link to the image that is causing the error?

boxerab commented 8 years ago

My jpylyzer unit tests are only for encoding, so I will try sharing one of these encoded files. In the meantime, you might find this link useful:

http://stackoverflow.com/questions/20112989/how-to-use-valgrind-with-python#20126802

Aaron

boxerab commented 8 years ago

Looks like valgrind reports this error for every j2k file I run through jpylyzer.

Output looks like this:

UMC ==20523== Conditional jump or move depends on uninitialised value(s) ==20523== at 0x4C321A8: __stpcpy_sse2_unaligned (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==20523== by 0x4030C8: ??? (in /usr/bin/jpylyzer) ==20523== by 0x402BD3: ??? (in /usr/bin/jpylyzer) ==20523== by 0x5276A3F: (below main) (libc-start.c:289) ==20523== UMC ==20523== Conditional jump or move depends on uninitialised value(s) ==20523== at 0x4C2F0A8: strcpy (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==20523== by 0x402DF2: ??? (in /usr/bin/jpylyzer) ==20523== by 0x402BDE: ??? (in /usr/bin/jpylyzer) ==20523== by 0x5276A3F: (below main) (libc-start.c:289) ==20523== UMC ==20523== Conditional jump or move depends on uninitialised value(s) ==20523== at 0x4C2F0A8: strcpy (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==20523== by 0x4020A0: ??? (in /usr/bin/jpylyzer) ==20523== by 0x4020E8: ??? (in /usr/bin/jpylyzer) ==20523== by 0x402C3B: ??? (in /usr/bin/jpylyzer) ==20523== by 0x5276A3F: (below main) (libc-start.c:289) ==20523== UMR ==20534== Syscall param execve(filename) points to uninitialised byte(s) ==20534== at 0x5320FE7: execve (syscall-template.S:81) ==20534== by 0x53215FF: execvpe (execvpe.c:63) ==20534== by 0x404A1A: ??? (in /usr/bin/jpylyzer) ==20534== by 0x402D8C: ??? (in /usr/bin/jpylyzer) ==20534== by 0x5276A3F: (below main) (libc-start.c:289) ==20534== Address 0xffeffccb1 is on thread 1's stack

If jpylyzer or python was running with debug symbols, then we would see exactly where the problem is.

Valgrind is an absolutely awesome tool - I would highly recommend spending the time to figure out how to use valgrind to track this problem down.

boxerab commented 8 years ago

Once you install valgrind, on a Ubuntu system for example, it is as easy as invoking

$ valgrind jpylyzer SOME_FILE.j2k

to see the errors.

bitsgalore commented 6 years ago

Reminder for myself on compilation of Python for the analysis, based on:

Setting up Python from source

Get source distribution from here. Then following steps:

  1. Uncomment Py_USING_MEMORY_DEBUGGER line in 'Objects/obmalloc.c'

  2. Run following commands:

    ./configure --prefix=/home/johan/python27debug --without-pymalloc --with-pydebug --with-valgrind
make
make altinstall

    (using altinstall instead of install prevents the compiled version to replace any preinstalled version of Python; binaries are written to --prefix location).

Using valgrind

Then use valgrind like this:

valgrind --tool=memcheck --leak-check=full --suppressions=/home/johan/Python-2.7.13/Misc/valgrind-python.supp ~/python27debug/bin/python2.7 ~/jpylyzer/jpylyzer/jpylyzer.py ~/jpylyzer-test-files/aware.jp2 2> valgrind.txt

Full output here: valgrind.txt

Leak summary at end of file:

==9918== LEAK SUMMARY:
==9918==    definitely lost: 0 bytes in 0 blocks
==9918==    indirectly lost: 0 bytes in 0 blocks
==9918==      possibly lost: 1,190,172 bytes in 3,355 blocks
==9918==    still reachable: 1,825,553 bytes in 11,495 blocks
==9918==         suppressed: 32 bytes in 1 blocks
==9918== Reachable blocks (those to which a pointer was found) are not shown.
==9918== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9918== 
==9918== For counts of detected and suppressed errors, rerun with: -v
==9918== ERROR SUMMARY: 464 errors from 464 contexts (suppressed: 0 from 0)

From the Valgrind documentation:

"Possibly lost". This covers cases 5--8 (for the BBB blocks) above. This means that a chain of one or more pointers to the block has been found, but at least one of the pointers is an interior-pointer. This could just be a random value in memory that happens to point into a block, and so you shouldn't consider this ok unless you know you have interior-pointers.

Which means something's definitely not OK.

The actual errors are all of the following general form (in this case: repeated 464 times for different addresses):

==9918== 48 bytes in 1 blocks are possibly lost in loss record 42 of 2,248
==9918==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9918==    by 0x70225B9: prepare_s (_struct.c:1287)
==9918==    by 0x7022B87: s_init (_struct.c:1395)
==9918==    by 0x486CB1: type_call (typeobject.c:765)
==9918==    by 0x421452: PyObject_Call (abstract.c:2547)
==9918==    by 0x4221DF: PyObject_CallFunctionObjArgs (abstract.c:2774)
==9918==    by 0x7023C38: cache_struct (_struct.c:1830)
==9918==    by 0x7023D86: calcsize (_struct.c:1858)
==9918==    by 0x4DC5B5: call_function (ceval.c:4340)
==9918==    by 0x4D705E: PyEval_EvalFrameEx (ceval.c:2989)
==9918==    by 0x4D9F34: PyEval_EvalCodeEx (ceval.c:3584)
==9918==    by 0x4DCDCC: fast_function (ceval.c:4447)

Other example:

= 786,432 bytes in 1 blocks are possibly lost in loss record 2,248 of 2,248
==9966==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9966==    by 0x459121: dictresize (dictobject.c:643)
==9966==    by 0x459776: dict_set_item_by_hash_or_entry (dictobject.c:818)
==9966==    by 0x45986D: PyDict_SetItem (dictobject.c:848)
==9966==    by 0x48052A: PyString_InternInPlace (stringobject.c:4773)
==9966==    by 0x45E1F7: PyDict_SetItemString (dictobject.c:2468)
==9966==    by 0x504281: Py_InitModule4TraceRefs_64 (modsupport.c:86)
==9966==    by 0x826BE34: initbinascii (binascii.c:1512)
==9966==    by 0x4FECD2: _PyImport_LoadDynamicModule (importdl.c:53)
==9966==    by 0x4FAB78: load_module (import.c:1937)
==9966==    by 0x4FCDAA: import_submodule (import.c:2725)
==9966==    by 0x4FC34D: load_next (import.c:2539)

Even though these errors are more specific than those in the analysis by @boxerab, they're still not specific enough to link them to variables in the Python code. But maybe I'm missing something?

bitsgalore commented 6 years ago

Addition, this thread describes some alternative (and in this case probably better) ways to detect memory leaks in Python (ht @WillemJan):

https://stackoverflow.com/questions/1435415/python-memory-leaks

bitsgalore commented 4 years ago

Don't know if we can close this. Any ideas @carlwilson?

boxerab commented 4 years ago

Here is what i get from valgrind with latest version of jpylyser :

MemoryChecker.1096.log

carlwilson commented 4 years ago

I've not really got to grips with valgrind yet, my time on the last release was spent on other matters and some things had to give. I'd be interested to give this a look but it'll not be immediate. Leak hunting is rarely easy work and I've not profiled a Python app before so it'll take a little dedicated time. BTW @bitsgalore it appears we can't be closing this ;) Thanks for the extra info @boxerab

boxerab commented 2 months ago

Valgrind output from latest jpylyzer version 2.2.1

==81654== HEAP SUMMARY:
==81654==     in use at exit: 441,310 bytes in 94 blocks
==81654==   total heap usage: 3,905 allocs, 3,811 frees, 13,399,201 bytes allocated
==81654== 
==81654== LEAK SUMMARY:
==81654==    definitely lost: 0 bytes in 0 blocks
==81654==    indirectly lost: 0 bytes in 0 blocks
==81654==      possibly lost: 0 bytes in 0 blocks
==81654==    still reachable: 441,310 bytes in 94 blocks
==81654==         suppressed: 0 bytes in 0 blocks
==81654== Rerun with --leak-check=full to see details of leaked memory
==81654== 
==81654== For lists of detected and suppressed errors, rerun with: -s
==81654== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

So, I think it is safe to close this issue.

bitsgalore commented 2 months ago

@boxerab Thanks for the update. I have literally no idea what made the valgrind errors disappear (changes to the jpylyzer code or perhaps the Python interpreter?), but good to see we can close this after 8 years!