miurahr / py7zr

7zip in python3 with ZStandard, PPMd, LZMA2, LZMA1, Delta, BCJ, BZip2, and Deflate compressions, and AES encryption.
https://pypi.org/project/py7zr/
GNU Lesser General Public License v2.1
461 stars 74 forks source link

`test_extract_high_compression_rate` fails in (Void Linux) chroot #478

Closed TinfoilSubmarine closed 2 years ago

TinfoilSubmarine commented 2 years ago

Describe the bug

test_extract_high_compression_rate fails in chroot.

To Reproduce

  1. Run tests in void-packages package builder chroot with tests enabled: ./xbps-src pkg python3-py7zr -Q
  2. See test failure in do_check (copied below)

Expected behavior

test_extract_high_compression_rate should pass.

Environment (please complete the following information):

Test data(please attach in the report):

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /builddir/py7zr-0.20.0, configfile: pyproject.toml, testpaths: tests
plugins: timeout-2.1.0
timeout: 480.0s
timeout method: signal
timeout func_only: False
collected 293 items

tests/test_archive.py ..............................................     [ 15%]
tests/test_basic.py .......................                              [ 23%]
tests/test_cli.py ......................                                 [ 31%]
tests/test_encryption.py ....................                            [ 37%]
tests/test_extra_codecs.py .......................                       [ 45%]
tests/test_extract.py .......................................s........   [ 62%]
tests/test_info.py ..................                                    [ 68%]
tests/test_misc.py .........F                                            [ 71%]
tests/test_unit.py ..................................................... [ 89%]
...............s..........                                               [ 98%]
tests/test_win32compat.py ...s                                           [100%]

=================================== FAILURES ===================================
______________________ test_extract_high_compression_rate ______________________

tmp_path = PosixPath('/tmp/pytest-of-joel/pytest-0/test_extract_high_compression_0')

    @pytest.mark.misc
    @pytest.mark.skipif(
        sys.platform.startswith("win"),
        reason="Only meaningful only on Unix-like OSes",
    )
    @pytest.mark.slow
    def test_extract_high_compression_rate(tmp_path):
        gen = Generator()
        with py7zr.SevenZipFile(tmp_path.joinpath("target.7z"), "w") as source:
            source.writef(gen, "source")
        limit = int(512e6)  # 0.5GB
        with limit_memory(limit):
            with py7zr.SevenZipFile(tmp_path.joinpath("target.7z"), "r") as target:
>               target.extractall(path=tmp_path)

/builddir/py7zr-0.20.0/tests/test_misc.py:168: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/builddir/py7zr-0.20.0/py7zr/py7zr.py:980: in extractall
    self._extract(path=path, return_dict=False, callback=callback)
/builddir/py7zr-0.20.0/py7zr/py7zr.py:641: in _extract
    self.worker.extract(
/builddir/py7zr-0.20.0/py7zr/py7zr.py:1209: in extract
    self.extract_single(
/builddir/py7zr-0.20.0/py7zr/py7zr.py:1293: in extract_single
    raise e
/builddir/py7zr-0.20.0/py7zr/py7zr.py:1290: in extract_single
    self._extract_single(fp, files, src_end, q, skip_notarget)
/builddir/py7zr-0.20.0/py7zr/py7zr.py:1350: in _extract_single
    crc32 = self.decompress(fp, f.folder, obfp, f.uncompressed, f.compressed, src_end)
/builddir/py7zr-0.20.0/py7zr/py7zr.py:1405: in decompress
    tmp = decompressor.decompress(fp, min(out_remaining, max_block_size))
/builddir/py7zr-0.20.0/py7zr/compressor.py:699: in decompress
    tmp = self._decompress(data, max_length)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <py7zr.compressor.SevenZipDecompressor object at 0x7f7306660760>
data = b"pMG\x0c\xd1\x91\x11\xaa\xad`\x1d\xba\xce\xb1'\x18\\Y\x86\xe9fRX\xbe\xe9v\xacY\xe4\xe5[\x05\x08\xf9\xc7\xda\xad\xfc\x...\xfc|\xde\xfb\x9a0.V\xc0\x8f\x85\xf3\x83\x81\xc0e\xc4%S\xf8\xf5\x9161\x05\xa5\xb0\xeeo\xc1pMG\x0c\xd1\x91\x11\xaa\xad`"
max_length = 79837184

    def _decompress(self, data, max_length: int):
        for i, decompressor in enumerate(self.chain):
            if self._unpacked[i] < self._unpacksizes[i]:
>               data = decompressor.decompress(data, max_length)
E               MemoryError: Unable to allocate output buffer.

/builddir/py7zr-0.20.0/py7zr/compressor.py:655: MemoryError
=============================== warnings summary ===============================
tests/test_extract.py::test_asyncio_executor
  /builddir/py7zr-0.20.0/tests/test_extract.py:428: DeprecationWarning: There is no current event loop
    loop = asyncio.get_event_loop()

tests/test_extract.py::test_asyncio_executor
  /builddir/py7zr-0.20.0/tests/test_extract.py:429: DeprecationWarning: There is no current event loop
    task = asyncio.ensure_future(aio7zr(tmp_path.joinpath("test_1.7z"), path=tmp_path))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/test_misc.py::test_extract_high_compression_rate - MemoryError: ...
============ 1 failed, 289 passed, 3 skipped, 2 warnings in 37.89s =============

Additional context

miurahr commented 2 years ago

The case is basically memory test. It limits allocation memory maximum 0.5GB and run test that will consume 0.4GB memory. If your build environment or pkg tool allocate less memory, it should be failed.

The part following does the thing.

        limit = int(512e6)  # 0.5GB
        with limit_memory(limit):

You have a option to skip slow and memory test that mark @pytest.mark.slow with option

-m "not slow"

https://docs.pytest.org/en/latest/example/markers.html?highlight=option+marker#marking-test-functions-and-selecting-them-for-a-run