test_empty_size_pickles fails on some archs (MemoryError)

[jcfp]

Updated sabyenc in debian yesterday, but unfortunately test_empty_size_pickles results in a MemoryError on some architectures (so far: armel, mips and powerpc):

I: pybuild base:217: cd /<<PKGBUILDDIR>>/.pybuild/cpython2_2.7/build; python2.7 -m pytest --ignore=tests/yencfiles/
============================= test session starts ==============================
platform linux2 -- Python 2.7.14+, pytest-3.3.2, py-1.5.3, pluggy-0.6.0
rootdir: /<<PKGBUILDDIR>>, inifile: pytest.ini
collected 24 items

tests/test_decoder.py .............F..                                   [ 66%]
tests/test_inputvalidation.py ........                                   [100%]

=================================== FAILURES ===================================
___________________________ test_empty_size_pickles ____________________________

    def test_empty_size_pickles():
        # When article size is left empty, it should not result in segfaults!
        data_plain, data_chunks, data_bytes = read_pickle('tests/yencfiles/emptysize_67caae212')
        decoded_data, filename, crc_correct = sabyenc_wrapper(data_chunks, 0)
        assert filename == 'Jake.and.the.Never.Land.Pirates.S02E38.480p.hdtv.x264.r05'
        assert crc_correct == True
        assert len(decoded_data) == 384000

        # Or when it's an invalid number
>       decoded_data, filename, crc_correct = sabyenc_wrapper(data_chunks, -1)

tests/test_decoder.py:110: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data_chunks = ['222 0 <1377609106.98612.8@news.newshosting.com>\r\n\r\n=ybegin part=8 total=104 line=128 size=39936000 name=Jake.and...4\x81+>\xee\x0c~J.\x1907;jt@\xbf#\x9d\x99\x10\x9d\xccjp\xf8\xbai\xb4\x0eM \xcd=@\x16\x9f\xe5$\xc4\x8e=}\x83o\r\n", ...]
data_bytes = -1

    def sabyenc_wrapper(data_chunks, data_bytes):
        """ CRC's are """
>       decoded_data, filename, crc_calc, crc_yenc, crc_correct = sabyenc.decode_usenet_chunks(data_chunks, data_bytes)
E       MemoryError

tests/testsupport.py:33: MemoryError
===================== 1 failed, 23 passed in 6.17 seconds ======================

The previous version in there was 3.3.1 (3.3.2 and 3.3.3 were never in the official repos; ppa builds don't include tests) which predated this particular test. Overview here, click in the status column for the build logs.

Ideas?

[sanderjo]

FWIW: I can confirm the MemoryError on ARM 64bit

sander@nanopineo2:~/git/sabyenc$ pytest
============================================================================================= test session starts =============================================================================================
platform linux2 -- Python 2.7.12, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
rootdir: /home/sander/git/sabyenc, inifile: pytest.ini
collected 24 items

tests/test_decoder.py .............F..                                                                                                                                                                  [ 66%]
tests/test_inputvalidation.py ........                                                                                                                                                                  [100%]

================================================================================================== FAILURES ===================================================================================================
___________________________________________________________________________________________ test_empty_size_pickles ___________________________________________________________________________________________

    def test_empty_size_pickles():
        # When article size is left empty, it should not result in segfaults!
        data_plain, data_chunks, data_bytes = read_pickle('tests/yencfiles/emptysize_67caae212')
        decoded_data, filename, crc_correct = sabyenc_wrapper(data_chunks, 0)
        assert filename == 'Jake.and.the.Never.Land.Pirates.S02E38.480p.hdtv.x264.r05'
        assert crc_correct == True
        assert len(decoded_data) == 384000

        # Or when it's an invalid number
>       decoded_data, filename, crc_correct = sabyenc_wrapper(data_chunks, -1)

tests/test_decoder.py:110:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

data_chunks = ['222 0 <1377609106.98612.8@news.newshosting.com>\r\n\r\n=ybegin part=8 total=104 line=128 size=39936000 name=Jake.and...4\x81+>\xee\x0c~J.\x1907;jt@\xbf#\x9d\x99\x10\x9d\xccjp\xf8\xbai\xb4\x0eM \xcd=@\x16\x9f\xe5$\xc4\x8e=}\x83o\r\n", ...]
data_bytes = -1

    def sabyenc_wrapper(data_chunks, data_bytes):
        """ CRC's are """
>       decoded_data, filename, crc_calc, crc_yenc, crc_correct = sabyenc.decode_usenet_chunks(data_chunks, data_bytes)
E       MemoryError

tests/testsupport.py:33: MemoryError
==================================================================================== 1 failed, 23 passed in 14.50 seconds =====================================================================================
sander@nanopineo2:~/git/sabyenc$

sander@nanopineo2:~/git/sabyenc$ uname -a
Linux nanopineo2 4.11.8-sun50iw2 #10 SMP Mon Jul 3 16:09:59 CEST 2017 aarch64 aarch64 aarch64 GNU/Linux
sander@nanopineo2:~/git/sabyenc$

[jcfp]

FWIW: I can confirm the MemoryError on ARM 64bit

Interesting, for a second I thought failures would be restricted to 32bit systems.

[sanderjo]

Could it be caused by Endianness? Intel / X86 is little-endian, whereas others are not pure little-endian (but bi-endian ... ).

[jcfp]

Archs with test failure seem to be a mix of both: (iirc) mips and powerpc are big endian in debian whilst everything *-el (armel, mipsel) isn't.

[Safihre]

The problem was me being not very proficient in C. I was checking bytes <= 0 but bytes was of type unsigned int so it skips that test but still does malloc(-1) so that blows up.

[sanderjo]

FYI: On i386 linux gcc I had a core dump with pytest, which is gone with the branch test-mem (so git checkout test-mem)

sander@haring:~/git/sabyenc$ pytest
============================================================================================= test session starts =============================================================================================
platform linux2 -- Python 2.7.6, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
rootdir: /home/sander/git/sabyenc, inifile: pytest.ini
collected 24 items

tests/test_decoder.py .............Segmentation fault (core dumped)
sander@haring:~/git/sabyenc$

[jcfp]

Thanks for the quick fix. Expect to release 3.3.5 soonish? (otherwise I'll just backport the fix)

[Safihre]

Pushed!