miurahr
commented
4 years ago Do you try encryption?
Closed EdwardChamberlain closed 4 years ago
miurahr
commented
4 years ago Do you try encryption?
EdwardChamberlain
commented
4 years ago Do you try encryption?
Yes I'm using:
with py7zr.SevenZipFile('runtimefiles/backup.7z', 'w', password='secret') as archive:
archive.writeall('runtimefiles/backup', 'base’)I can successfully zip the file if I place a sample.txt file in the folder only but sending the database (database.db) results in this error.
miurahr
commented
4 years ago Could you provide exact sizes of target files when got error? Buffer length is depends on READ_BLOCKSIZE and padding size. Current implementation prepared a buffer which size is READ_BLOCKSIZE + CBC_BLOCKSIZE.
miurahr
commented
4 years ago @EdwardChamberlain Does #202 solve your problem? It changes buffer size to READ_BLOCKSIZE + CBC_BLOCKSIZE * 2
EdwardChamberlain
commented
4 years ago Doesnt seem to resolve it - still get the BufferOverflow error.
File is a called database_prod.db and is 16,920,576 bytes. In a folder called backup on its own.
Full code:
print("Copying files")
new_path = shutil.copy('database_prod.db', 'runtimefiles/backup')
print("New Path = ", new_path)
print("Done Copying")
print("Zipping")
with py7zr.SevenZipFile('runtimefiles/backup.7z', 'w', password='secret') as archive:
archive.writeall('runtimefiles/backup', 'base')
print("Done Zipping”)Does target file is a single database_prod.db file?
A READ_BLOCKSIZE = 1048576 bytes. A encryption loop would be 16 times and last 143360 bytes that is aligned CBC_BLOCKSIZE(=16) In this condition, a line where buffer overflow happened have never reached. Very strange.
miurahr
commented
4 years ago Another strange thing. You reported that python is 3.8 but error log point python3.7. A READ_BLOCKSIZE differs according to python versions. (because of library bug which exist in 3.7.4 and before)
if sys.version_info >= (3, 7, 5):
READ_BLOCKSIZE = 1048576
else:
READ_BLOCKSIZE = 32248Does py7zr v0.8.2 reproduce the error? An above part has been introduced in v0.8.3.
miurahr
commented
4 years ago I'd like to change READ_BLOCKSIZE for python 3.7 to 32768 bytes. 32248 is not 16bytes aligned. I've pushed a change to #202
EdwardChamberlain
commented
4 years ago Another strange thing. You reported that python is 3.8 but error log point python3.7.
Good catch - I am infact running python Python 3.7.3. Apologies!
My folder is a single folder called backup with a single database_prod.db file in it.
EdwardChamberlain
commented
4 years ago If I remove the password then I dont get the BufferOverflow but I do get a MemoryError (which I suspect is due to me using a raspberry pi)
miurahr
commented
4 years ago I cannot reproduce when running test with 16,920,576 bytes source file with v0.8.3 on Raspberry Pi 4B 4GB. All the test is passed on my RPi.
Could you run a test suite on your platform?
$ git clone http://github.com/miurahr/py7zr
$ cd py7zr
$ git checkout v0.8.3
$ pip install tox
$ tox -e py37Log here:
pi@raspberrypi:~/Documents/BenchyBot $ git clone http://github.com/miurahr/py7zr
Cloning into 'py7zr'...
warning: redirecting to https://github.com/miurahr/py7zr/
remote: Enumerating objects: 8, done.
remote: Counting objects: 100% (8/8), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 5274 (delta 2), reused 6 (delta 2), pack-reused 5266
Receiving objects: 100% (5274/5274), 28.05 MiB | 1.12 MiB/s, done.
Resolving deltas: 100% (3876/3876), done.
pi@raspberrypi:~/Documents/BenchyBot $ cd py7zr
pi@raspberrypi:~/Documents/BenchyBot/py7zr $ git checkout v0.8.3
Note: checking out 'v0.8.3'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at 492b925 Prepare for v0.8.3
pi@raspberrypi:~/Documents/BenchyBot/py7zr $ pip install tox
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting tox
Downloading tox-3.18.1-py2.py3-none-any.whl (82 kB)
|████████████████████████████████| 82 kB 243 kB/s
Collecting toml>=0.9.4
Downloading toml-0.10.1-py2.py3-none-any.whl (19 kB)
Collecting pluggy>=0.12.0
Downloading pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0
Downloading virtualenv-20.0.28-py2.py3-none-any.whl (4.9 MB)
|████████████████████████████████| 4.9 MB 12 kB/s
Requirement already satisfied: importlib-metadata<2,>=0.12; python_version < "3.8" in /home/pi/.local/lib/python3.7/site-packages (from tox) (1.6.0)
Collecting py>=1.4.17
Downloading py-1.9.0-py2.py3-none-any.whl (99 kB)
|████████████████████████████████| 99 kB 1.5 MB/s
Collecting filelock>=3.0.0
Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
Requirement already satisfied: packaging>=14 in /home/pi/.local/lib/python3.7/site-packages (from tox) (20.4)
Collecting six>=1.14.0
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting distlib<1,>=0.3.1
Downloading distlib-0.3.1-py2.py3-none-any.whl (335 kB)
|████████████████████████████████| 335 kB 3.6 MB/s
Collecting appdirs<2,>=1.4.3
Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Requirement already satisfied: zipp>=0.5 in /home/pi/.local/lib/python3.7/site-packages (from importlib-metadata<2,>=0.12; python_version < "3.8"->tox) (3.1.0)
Requirement already satisfied: pyparsing>=2.0.2 in /home/pi/.local/lib/python3.7/site-packages (from packaging>=14->tox) (2.4.7)
Installing collected packages: toml, pluggy, six, filelock, distlib, appdirs, virtualenv, py, tox
Successfully installed appdirs-1.4.4 distlib-0.3.1 filelock-3.0.12 pluggy-0.13.1 py-1.9.0 six-1.15.0 toml-0.10.1 tox-3.18.1 virtualenv-20.0.28
GLOB sdist-make: /home/pi/Documents/BenchyBot/py7zr/setup.py
py37 create: /home/pi/Documents/BenchyBot/py7zr/.tox/py37
py37 inst: /home/pi/Documents/BenchyBot/py7zr/.tox/.tmp/package/1/py7zr-0.8.3.zip
py37 installed: attrs==19.3.0,certifi==2020.6.20,cffi==1.14.1,chardet==3.0.4,codecov==2.1.8,coverage==5.2.1,coveralls==2.1.1,docopt==0.6.2,idna==2.10,importlib-metadata==1.7.0,iniconfig==1.0.0,more-itertools==8.4.0,mypy-extensions==0.4.3,packaging==20.4,pluggy==0.13.1,py==1.9.0,py-cpuinfo==7.0.0,py7zr @ file:///home/pi/Documents/BenchyBot/py7zr/.tox/.tmp/package/1/py7zr-0.8.3.zip,pyannotate==1.2.0,pycparser==2.20,pycryptodome==3.9.8,pyparsing==2.4.7,pytest==6.0.0,pytest-benchmark==3.2.3,pytest-cov==2.10.0,pytest-remotedata==0.3.2,pytest-timeout==1.4.2,requests==2.24.0,six==1.15.0,texttable==1.6.2,toml==0.10.1,urllib3==1.25.10,zipp==3.1.0,zstandard==0.14.0
py37 run-test-pre: PYTHONHASHSEED='3199819392'
py37 run-test: commands[0] | python -m pytest -vv
================================================================================= test session starts =================================================================================
platform linux -- Python 3.7.3, pytest-6.0.0, py-1.9.0, pluggy-0.13.1 -- /home/pi/Documents/BenchyBot/py7zr/.tox/py37/bin/python
cachedir: .tox/py37/.pytest_cache
benchmark: 3.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/pi/Documents/BenchyBot/py7zr, configfile: tox.ini
plugins: cov-2.10.0, remotedata-0.3.2, timeout-1.4.2, benchmark-3.2.3
timeout: 480.0s
timeout method: signal
timeout func_only: False
collected 246 items
tests/test_archive.py::test_compress_single_encoded_header PASSED [ 0%]
tests/test_archive.py::test_compress_directory_encoded_header PASSED [ 0%]
tests/test_archive.py::test_compress_files_encoded_header PASSED [ 1%]
tests/test_archive.py::test_compress_file_0 PASSED [ 1%]
tests/test_archive.py::test_compress_directory PASSED [ 2%]
tests/test_archive.py::test_compress_files_1 PASSED [ 2%]
tests/test_archive.py::test_register_archive_format PASSED [ 2%]
tests/test_archive.py::test_compress_with_simple_filter PASSED [ 3%]
tests/test_archive.py::test_compress_with_custom_filter PASSED [ 3%]
tests/test_archive.py::test_compress_files_2 PASSED [ 4%]
tests/test_archive.py::test_compress_files_3 PASSED [ 4%]
tests/test_archive.py::test_compress_symlink PASSED [ 4%]
tests/test_archive.py::test_compress_zerofile PASSED [ 5%]
tests/test_archive.py::test_compress_directories PASSED [ 5%]
tests/test_archive.py::test_compress_absolute_symlink_as_relative PASSED [ 6%]
tests/test_archive.py::test_compress_files_deref PASSED [ 6%]
tests/test_archive.py::test_compress_lzma2_bcj PASSED [ 6%]
tests/test_archive.py::test_compress_files_deref_loop PASSED [ 7%]
tests/test_archive.py::test_compress_copy SKIPPED [ 7%]
tests/test_archive.py::test_compress_multi_filter_delta PASSED [ 8%]
tests/test_archive.py::test_compress_deflate PASSED [ 8%]
tests/test_archive.py::test_compress_zstd PASSED [ 8%]
tests/test_archive.py::test_compress_arm64 PASSED [ 9%]
tests/test_archive.py::test_compress_ppc PASSED [ 9%]
tests/test_archive.py::test_compress_ia64 PASSED [ 10%]
tests/test_archive.py::test_compress_sparc PASSED [ 10%]
tests/test_archive.py::test_compress_armt PASSED [ 10%]
tests/test_archive.py::test_compress_small_files PASSED [ 11%]
tests/test_basic.py::test_basic_initinfo PASSED [ 11%]
tests/test_basic.py::test_cli_list_1 PASSED [ 12%]
tests/test_basic.py::test_cli_list_2 PASSED [ 12%]
tests/test_basic.py::test_basic_not_implemented_yet1 PASSED [ 13%]
tests/test_basic.py::test_basic_not_implemented_yet3 PASSED [ 13%]
tests/test_basic.py::test_basic_wrong_option_value PASSED [ 13%]
tests/test_basic.py::test_basic_extract_1 PASSED [ 14%]
tests/test_basic.py::test_basic_extract_2 PASSED [ 14%]
tests/test_basic.py::test_basic_decode_3 PASSED [ 15%]
tests/test_basic.py::test_py7zr_is_7zfile PASSED [ 15%]
tests/test_basic.py::test_py7zr_is_7zfile_fileish PASSED [ 15%]
tests/test_basic.py::test_py7zr_is_7zfile_path PASSED [ 16%]
tests/test_basic.py::test_py7zr_is_not_7zfile PASSED [ 16%]
tests/test_basic.py::test_cli_help PASSED [ 17%]
tests/test_basic.py::test_cli_no_subcommand PASSED [ 17%]
tests/test_basic.py::test_cli_list_verbose PASSED [ 17%]
tests/test_basic.py::test_cli_test PASSED [ 18%]
tests/test_basic.py::test_cli_info PASSED [ 18%]
tests/test_basic.py::test_cli_extract PASSED [ 19%]
tests/test_basic.py::test_cli_encrypted_extract PASSED [ 19%]
tests/test_basic.py::test_digests PASSED [ 19%]
tests/test_basic.py::test_digests_corrupted PASSED [ 20%]
tests/test_basic.py::test_non7z_ext PASSED [ 20%]
tests/test_basic.py::test_non7z_test PASSED [ 21%]
tests/test_basic.py::test_non7z_list PASSED [ 21%]
tests/test_basic.py::test_archive_creation PASSED [ 21%]
tests/test_basic.py::test_archive_already_exist PASSED [ 22%]
tests/test_basic.py::test_archive_without_extension PASSED [ 22%]
tests/test_basic.py::test_volume_creation PASSED [ 23%]
tests/test_basic.py::test_volume_creation_wrong_volume_unit PASSED [ 23%]
tests/test_basic.py::test_py7zr_write_mode PASSED [ 23%]
tests/test_basic.py::test_py7zr_writeall_single PASSED [ 24%]
tests/test_basic.py::test_py7zr_writeall_dir PASSED [ 24%]
tests/test_basic.py::test_py7zr_extract_specified_file PASSED [ 25%]
tests/test_basic.py::test_py7zr_extract_and_getnames PASSED [ 25%]
tests/test_basic.py::test_py7zr_extract_and_reset_iteration PASSED [ 26%]
tests/test_basic.py::test_context_manager_1 PASSED [ 26%]
tests/test_basic.py::test_context_manager_2 PASSED [ 26%]
tests/test_basic.py::test_extract_callback PASSED [ 27%]
tests/test_basic.py::test_py7zr_list_values PASSED [ 27%]
tests/test_benchmark.py::test_extract_benchmark[solid.7z-None] PASSED [ 28%]
tests/test_benchmark.py::test_extract_benchmark[mblock_1.7z-None] PASSED [ 28%]
tests/test_benchmark.py::test_extract_benchmark[encrypted_1.7z-secret] PASSED [ 28%]
tests/test_benchmark.py::test_benchmark_calculate_key1 PASSED [ 29%]
tests/test_benchmark.py::test_benchmark_calculate_key2 PASSED [ 29%]
tests/test_benchmark.py::test_benchmark_calculate_key3 PASSED [ 30%]
tests/test_concurrent.py::test_concurrent_futures SKIPPED [ 30%]
tests/test_concurrent.py::test_concurrent_extraction PASSED [ 30%]
tests/test_encryption.py::test_extract_encrypted_1 PASSED [ 31%]
tests/test_encryption.py::test_extract_encrypted_1_mem PASSED [ 31%]
tests/test_encryption.py::test_extract_encrypted_2 PASSED [ 32%]
tests/test_encryption.py::test_encrypt_file_0 PASSED [ 32%]
tests/test_encryption.py::test_encrypt_file_1 PASSED [ 32%]
tests/test_encryption.py::test_encrypt_file_2 PASSED [ 33%]
tests/test_encryption.py::test_encrypt_file_3 PASSED [ 33%]
tests/test_encryption.py::test_encrypt_file_4 SKIPPED [ 34%]
tests/test_extra_codecs.py::test_aes_cipher PASSED [ 34%]
tests/test_extra_codecs.py::test_deflate_compressor PASSED [ 34%]
tests/test_extra_codecs.py::test_deflate_compressor_flushed PASSED [ 35%]
tests/test_extra_codecs.py::test_deflate_decompressor PASSED [ 35%]
tests/test_extra_codecs.py::test_copy_compressor PASSED [ 36%]
tests/test_extra_codecs.py::test_copy_decompressor PASSED [ 36%]
tests/test_extra_codecs.py::test_zstd_compressor PASSED [ 36%]
tests/test_extra_codecs.py::test_zstd_decompressor PASSED [ 37%]
tests/test_extra_codecs.py::test_sevenzipcompressor_aes_lzma2 PASSED [ 37%]
tests/test_extra_codecs.py::test_aes_compressor PASSED [ 38%]
tests/test_extra_codecs.py::test_sevenzipcompressor_aes_only PASSED [ 38%]
tests/test_extra_codecs.py::test_aes_encrypt_data PASSED [ 39%]
tests/test_extra_codecs.py::test_aes_decrypt PASSED [ 39%]
tests/test_extra_codecs.py::test_extract_bzip2 PASSED [ 39%]
tests/test_extra_codecs.py::test_extract_ppmd PASSED [ 40%]
tests/test_extra_codecs.py::test_extract_deflate PASSED [ 40%]
tests/test_extra_codecs.py::test_extract_zstd PASSED [ 41%]
tests/test_extra_codecs.py::test_extract_p7zip_zstd SKIPPED [ 41%]
tests/test_extract.py::test_solid PASSED [ 41%]
tests/test_extract.py::test_solid_mem PASSED [ 42%]
tests/test_extract.py::test_empty PASSED [ 42%]
tests/test_extract.py::test_github_14 PASSED [ 43%]
tests/test_extract.py::test_github_14_mem PASSED [ 43%]
tests/test_extract.py::test_non_solid_umlaut PASSED [ 43%]
tests/test_extract.py::test_non_solid_umlaut_mem PASSED [ 44%]
tests/test_extract.py::test_solid_umlaut PASSED [ 44%]
tests/test_extract.py::test_solid_umlaut_mem PASSED [ 45%]
tests/test_extract.py::test_bugzilla_4 PASSED [ 45%]
tests/test_extract.py::test_extract_symlink PASSED [ 45%]
tests/test_extract.py::test_extract_symlink_mem PASSED [ 46%]
tests/test_extract.py::test_lzma2bcj PASSED [ 46%]
tests/test_extract.py::test_lzma2bcj_mem PASSED [ 47%]
tests/test_extract.py::test_lzma2bcj2 PASSED [ 47%]
tests/test_extract.py::test_extract_lzmabcj_archiveinfo PASSED [ 47%]
tests/test_extract.py::test_extract_lzmabcj_1 PASSED [ 48%]
tests/test_extract.py::test_extract_lzmabcj_2 PASSED [ 48%]
tests/test_extract.py::test_extract_lzmabcj_3 XFAIL [ 49%]
tests/test_extract.py::test_zerosize PASSED [ 49%]
tests/test_extract.py::test_zerosize_mem PASSED [ 50%]
tests/test_extract.py::test_register_unpack_archive PASSED [ 50%]
tests/test_extract.py::test_skip PASSED [ 50%]
tests/test_extract.py::test_github_14_multi PASSED [ 51%]
tests/test_extract.py::test_github_14_multi_mem PASSED [ 51%]
tests/test_extract.py::test_multiblock PASSED [ 52%]
tests/test_extract.py::test_multiblock_mem PASSED [ 52%]
tests/test_extract.py::test_multiblock_unlink PASSED [ 52%]
tests/test_extract.py::test_copy PASSED [ 53%]
tests/test_extract.py::test_copy_2 PASSED [ 53%]
tests/test_extract.py::test_close_unlink PASSED [ 54%]
tests/test_extract.py::test_asyncio_executor PASSED [ 54%]
tests/test_extract.py::test_no_main_streams PASSED [ 54%]
tests/test_extract.py::test_no_main_streams_mem PASSED [ 55%]
tests/test_extract.py::test_extract_symlink_with_relative_target_path PASSED [ 55%]
tests/test_extract.py::test_extract_emptystream_mix PASSED [ 56%]
tests/test_extract.py::test_extract_longpath_file PASSED [ 56%]
tests/test_extract.py::test_extract_symlink_overwrite PASSED [ 56%]
tests/test_extract.py::test_py7zr_extract_corrupted PASSED [ 57%]
tests/test_extract.py::test_extract_lzma2delta PASSED [ 57%]
tests/test_extract.py::test_lzma_raw_decompressor_lzmabcj XFAIL [ 58%]
tests/test_extract.py::test_decompress_small_files SKIPPED [ 58%]
tests/test_info.py::test_archiveinfo_deflate PASSED [ 58%]
tests/test_info.py::test_archiveinfo_deflate64 PASSED [ 59%]
tests/test_info.py::test_archiveinfo_lzma_bcj2 PASSED [ 59%]
tests/test_info.py::test_archiveinfo_lzma_bcj PASSED [ 60%]
tests/test_info.py::test_archiveinfo_lzma2_bcj PASSED [ 60%]
tests/test_info.py::test_archiveinfo_lzma_bcj_arm PASSED [ 60%]
tests/test_info.py::test_archiveinfo_lzma_bcj_armt PASSED [ 61%]
tests/test_info.py::test_archiveinfo_lzma_bcj_ia64 PASSED [ 61%]
tests/test_info.py::test_archiveinfo_lzma_bcj_ppc PASSED [ 62%]
tests/test_info.py::test_archiveinfo_lzma_bcj_sparc PASSED [ 62%]
tests/test_info.py::test_archiveinfo_7zaes_lzma PASSED [ 63%]
tests/test_info.py::test_archivetest_deflate PASSED [ 63%]
tests/test_info.py::test_archivetest_deflate64 PASSED [ 63%]
tests/test_info.py::test_archivetest_lzma_bcj2 PASSED [ 64%]
tests/test_info.py::test_archivetest_lzma_bcj XFAIL [ 64%]
tests/test_info.py::test_archivetest_lzma2_bcj PASSED [ 65%]
tests/test_info.py::test_archivetest_lzma_bcj_arm PASSED [ 65%]
tests/test_info.py::test_archivetest_lzma_bcj_armt PASSED [ 65%]
tests/test_info.py::test_archivetest_lzma_bcj_ia64 PASSED [ 66%]
tests/test_info.py::test_archivetest_lzma_bcj_ppc PASSED [ 66%]
tests/test_info.py::test_archivetest_lzma_bcj_sparc PASSED [ 67%]
tests/test_info.py::test_archivetest_7zaes PASSED [ 67%]
tests/test_info.py::test_list_filename_encryption PASSED [ 67%]
tests/test_unit.py::test_py7zr_signatureheader PASSED [ 68%]
tests/test_unit.py::test_py7zr_mainstreams PASSED [ 68%]
tests/test_unit.py::test_py7zr_folder_retrive PASSED [ 69%]
tests/test_unit.py::test_py7zr_folder_write PASSED [ 69%]
tests/test_unit.py::test_py7zr_unpack_info PASSED [ 69%]
tests/test_unit.py::test_py7zr_substreamsinfo PASSED [ 70%]
tests/test_unit.py::test_py7zr_substreamsinfo_write PASSED [ 70%]
tests/test_unit.py::test_py7zr_header PASSED [ 71%]
tests/test_unit.py::test_py7zr_encoded_header PASSED [ 71%]
tests/test_unit.py::test_py7zr_files_info_1 PASSED [ 71%]
tests/test_unit.py::test_py7zr_files_info_2 PASSED [ 72%]
tests/test_unit.py::test_lzma_lzma2_compressor PASSED [ 72%]
tests/test_unit.py::test_lzma_lzma2bcj_compressor PASSED [ 73%]
tests/test_unit.py::test_read_booleans[\xb4\x80-9-False-expected0] PASSED [ 73%]
tests/test_unit.py::test_read_booleans[\xff\xc0-10-False-expected1] PASSED [ 73%]
tests/test_unit.py::test_read_booleans[\xff\xff\x80-17-False-expected2] PASSED [ 74%]
tests/test_unit.py::test_write_booleans[booleans0-False-\xb4\x80] PASSED [ 74%]
tests/test_unit.py::test_write_booleans[booleans1-False-\xff\xc0] PASSED [ 75%]
tests/test_unit.py::test_write_booleans[booleans2-False-\xff\xff\x80] PASSED [ 75%]
tests/test_unit.py::test_write_booleans[booleans3-True-\x00\xb4\x80] PASSED [ 76%]
tests/test_unit.py::test_write_uint64[1-\x01] PASSED [ 76%]
tests/test_unit.py::test_write_uint64[127-\x7f] PASSED [ 76%]
tests/test_unit.py::test_write_uint64[128-\x80\x80] PASSED [ 77%]
tests/test_unit.py::test_write_uint64[65535-\xc0\xff\xff] PASSED [ 77%]
tests/test_unit.py::test_write_uint64[441-\x81\xb9] PASSED [ 78%]
tests/test_unit.py::test_write_uint64[16777087-\xe0\x7f\xff\xff] PASSED [ 78%]
tests/test_unit.py::test_write_uint64[234889472-\xee\x00!\x00] PASSED [ 78%]
tests/test_unit.py::test_write_uint64[4294967295-\xf0\xff\xff\xff\xff] PASSED [ 79%]
tests/test_unit.py::test_write_uint64[545766266495-\xf8\x7fV4\x12\x7f] PASSED [ 79%]
tests/test_unit.py::test_write_uint64[5124095575370701-\xfe\xcd\xab\x90xV4\x12] PASSED [ 80%]
tests/test_unit.py::test_write_uint64[14921046061426453453-\xff\xcd\xab\x90xV4\x12\xcf] PASSED [ 80%]
tests/test_unit.py::test_read_uint64[\x01-1] PASSED [ 80%]
tests/test_unit.py::test_read_uint64[\x7f-127] PASSED [ 81%]
tests/test_unit.py::test_read_uint64[\x80\x80-128] PASSED [ 81%]
tests/test_unit.py::test_read_uint64[\x81\xb9-441] PASSED [ 82%]
tests/test_unit.py::test_read_uint64[\xc0\xff\xff-65535] PASSED [ 82%]
tests/test_unit.py::test_read_uint64[\xe0\x7f\xff\xff-16777087] PASSED [ 82%]
tests/test_unit.py::test_read_uint64[\xee\x00!\x00-234889472] PASSED [ 83%]
tests/test_unit.py::test_read_uint64[\xf0\xff\xff\xff\xff-4294967295] PASSED [ 83%]
tests/test_unit.py::test_read_uint64[\xf8\x7fV4\x12\x7f-545766266495] PASSED [ 84%]
tests/test_unit.py::test_read_uint64[\xfe\xcd\xab\x90xV4\x12-5124095575370701] PASSED [ 84%]
tests/test_unit.py::test_read_uint64[\xff\xcd\xab\x90xV4\x12\xcf-14921046061426453453] PASSED [ 84%]
tests/test_unit.py::test_read_utf16[t\x00e\x00s\x00t\x00\x00\x00-test] PASSED [ 85%]
tests/test_unit.py::test_write_utf16[test-t\x00e\x00s\x00t\x00\x00\x00] PASSED [ 85%]
tests/test_unit.py::test_write_packinfo PASSED [ 86%]
tests/test_unit.py::test_utc PASSED [ 86%]
tests/test_unit.py::test_utcoffset PASSED [ 86%]
tests/test_unit.py::test_dst PASSED [ 87%]
tests/test_unit.py::test_localtime_tzname PASSED [ 87%]
tests/test_unit.py::test_read_crcs PASSED [ 88%]
tests/test_unit.py::test_fileinfo_st_fmt PASSED [ 88%]
tests/test_unit.py::test_wrong_mode PASSED [ 89%]
tests/test_unit.py::test_calculate_crc32 PASSED [ 89%]
tests/test_unit.py::test_calculate_crc32_blocksize PASSED [ 89%]
tests/test_unit.py::test_startheader_calccrc PASSED [ 90%]
tests/test_unit.py::test_write_signature_header PASSED [ 90%]
tests/test_unit.py::test_make_file_info1 PASSED [ 91%]
tests/test_unit.py::test_make_file_info2 PASSED [ 91%]
tests/test_unit.py::test_simple_compress_and_decompress PASSED [ 91%]
tests/test_unit.py::test_sevenzipcompressor_default PASSED [ 92%]
tests/test_unit.py::test_calculate_key1[secret^&-63-i@#ri#Ildajfdk-i@#ri#Ildajfdks\x00e\x00c\x00r\x00e\x00t\x00^\x00&\x00\x00\x00] PASSED [ 92%]
tests/test_unit.py::test_calculate_key1[secret-5--(6\xa7\xf0\xcc"\t\x9dod\xe2\x8d\xd2\xe9\x08^s\x9c\x99-\xa3N\x08\x10\x9e\~\x89<\x1cR\xc2] PASSED [ 93%]
tests/test_unit.py::test_calculate_key2[secret^&-63-i@#ri#Ildajfdk-i@#ri#Ildajfdks\x00e\x00c\x00r\x00e\x00t\x00^\x00&\x00\x00\x00] PASSED [ 93%]
tests/test_unit.py::test_calculate_key3[secret^&-63-i@#ri#Ildajfdk-i@#ri#Ildajfdks\x00e\x00c\x00r\x00e\x00t\x00^\x00&\x00\x00\x00] PASSED [ 93%]
tests/test_unit.py::test_calculate_key3[secret-5--(6\xa7\xf0\xcc"\t\x9dod\xe2\x8d\xd2\xe9\x08^s\x9c\x99-\xa3N\x08\x10\x9e\~\x89<\x1cR\xc2] PASSED [ 94%]
tests/test_unit.py::test_calculate_key1_nohash PASSED [ 94%]
tests/test_unit.py::test_calculate_key2_nohash PASSED [ 95%]
tests/test_unit.py::test_calculate_key3_nohash PASSED [ 95%]
tests/test_unit.py::test_helpers_readlink_dirfd SKIPPED [ 95%]
tests/test_unit.py::test_compressor_lzma2bcj PASSED [ 96%]
tests/test_unit.py::test_lzmadecompressor_lzmabcj PASSED [ 96%]
tests/test_unit.py::test_unit_buffer PASSED [ 97%]
tests/test_unit.py::test_supported_method_is_crypto PASSED [ 97%]
tests/test_unit.py::test_supported_method_is_crypto_id PASSED [ 97%]
tests/test_unit.py::test_supported_method_get_method_id PASSED [ 98%]
tests/test_win32compat.py::test_symlink_readlink_absolute PASSED [ 98%]
tests/test_win32compat.py::test_symlink_readlink_relative PASSED [ 99%]
tests/test_win32compat.py::test_hardlink_readlink PASSED [ 99%]
tests/test_win32compat.py::test_junction_readlink SKIPPED [100%]
================================================================================== warnings summary ===================================================================================
.tox/py37/lib/python3.7/site-packages/_pytest/config/__init__.py:1148
/home/pi/Documents/BenchyBot/py7zr/.tox/py37/lib/python3.7/site-packages/_pytest/config/__init__.py:1148: PytestConfigWarning: Unknown config ini key: basepython
self._warn_or_fail_if_strict("Unknown config ini key: {}\n".format(key))
-- Docs: https://docs.pytest.org/en/stable/warnings.html
----------- coverage: platform linux, python 3.7.3-final-0 -----------
Name Stmts Miss Branch BrPart Cover Missing
------------------------------------------------------------------
py7zr/__main__.py 4 4 0 0 0% 19-25
py7zr/archiveinfo.py 767 24 272 36 94% 101->106, 160->162, 190->195, 230->240, 237->236, 259->258, 398->399, 399, 400->403, 401->400, 403, 413->414, 414, 530->532, 540->541, 541-542, 597->600, 600->603, 611->613, 613->615, 615->617, 687->699, 699-700, 721-726, 733->732, 734->732, 738->739, 739, 744->747, 747, 750-751, 755->758, 765->764, 769->exit, 780->784, 784, 786->787, 787, 793->792, 808->809, 809, 865->866, 866, 877->878, 878, 920->921, 921-922, 928->930, 930->932, 945->948, 996->997, 997
py7zr/cli.py 278 41 68 10 84% 47-51, 54, 57-58, 61-69, 72, 75, 89->90, 90, 160->162, 162->164, 187->190, 190, 199->203, 203-204, 234->235, 235, 251->255, 255-256, 269-271, 274-276, 279->280, 280-281, 283->286, 286-292, 326->329, 329
py7zr/compressor.py 374 25 120 19 91% 42-43, 138->156, 156, 209->210, 210-214, 242->243, 243, 254->255, 255, 276->277, 277, 285->286, 286, 288->289, 289, 299->300, 300, 301->302, 302, 335->338, 338, 363->364, 364, 373->383, 374->381, 381-383, 399, 459->460, 460, 469->473, 473, 560->561, 561, 579->580, 580, 586->587, 587, 593->594, 594
py7zr/helpers.py 240 31 64 8 82% 121->122, 122-125, 136->139, 139, 159->162, 162, 179->182, 182, 219, 248->251, 251-254, 266->267, 267-270, 271->280, 274->275, 275, 280-282, 294-297, 300, 303, 335-338, 341, 344, 347, 351, 354
py7zr/properties.py 108 2 4 2 96% 31->32, 32, 33->34, 34
py7zr/py7zr.py 853 89 346 48 86% 45->46, 46-47, 52->53, 53, 80->85, 127->128, 128, 219->220, 220, 221->222, 222, 285->287, 287-292, 303->306, 306, 308->314, 314, 339->342, 342, 365->366, 366, 371->372, 372, 375->376, 376, 416->418, 418->419, 419, 444-446, 448->452, 452, 472->475, 475, 491-492, 536->537, 537-558, 583->589, 614->616, 632, 639->640, 640, 656-660, 666->667, 667, 678->677, 685->686, 686-688, 693->697, 698->699, 699, 710-712, 714->715, 715-716, 723-729, 752->753, 753-758, 763->765, 765->771, 771->772, 772-773, 780-781, 791->793, 793-794, 803->804, 804, 805->808, 808, 816->823, 823, 825->827, 827-830, 836->839, 839, 853->854, 854, 861->exit, 864->exit, 879->883, 880->881, 881, 913->918, 918, 983->984, 984, 999->1009, 1049->1050, 1050, 1059->1060, 1060, 1129->1132
py7zr/win32compat.py 69 63 14 1 8% 7->8, 8-174
------------------------------------------------------------------
TOTAL 2722 279 888 124 87%
3 files skipped due to complete coverage.
---------------------------------------------------------------------------------------------------------- benchmark: 6 tests ---------------------------------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_extract_benchmark[solid.7z-None] 21.6024 (1.0) 69.8307 (1.0) 34.8310 (1.0) 17.0474 (1.24) 29.0324 (1.0) 19.7210 (1.0) 1;0 28.7101 (1.0) 8 1
test_extract_benchmark[mblock_1.7z-None] 808.7668 (37.44) 863.8507 (12.37) 843.2438 (24.21) 22.8072 (1.66) 855.8137 (29.48) 32.4009 (1.64) 1;0 1.1859 (0.04) 5 1
test_benchmark_calculate_key3 7,068.2028 (327.20) 7,101.6372 (101.70) 7,084.1866 (203.39) 13.7538 (1.0) 7,087.9988 (244.14) 22.1144 (1.12) 2;0 0.1412 (0.00) 5 1
test_extract_benchmark[encrypted_1.7z-secret] 7,243.5860 (335.31) 10,343.8020 (148.13) 7,903.1772 (226.90) 1,365.2138 (99.26) 7,296.7671 (251.33) 856.6718 (43.44) 1;1 0.1265 (0.00) 5 1
test_benchmark_calculate_key2 10,797.3853 (499.82) 10,876.4598 (155.75) 10,843.5084 (311.32) 33.5127 (2.44) 10,852.9488 (373.82) 56.2524 (2.85) 1;0 0.0922 (0.00) 5 1
test_benchmark_calculate_key1 17,917.9713 (829.44) 18,052.5907 (258.52) 17,951.7850 (515.40) 56.8145 (4.13) 17,927.7970 (617.51) 44.5906 (2.26) 1;1 0.0557 (0.00) 5 1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
========================================================== 236 passed, 7 skipped, 3 xfailed, 1 warning in 594.47s (0:09:54) ===========================================================
_______________________________________________________________________________________ summary _______________________________________________________________________________________
py37: commands succeeded
congratulations :)
Uhmm, it is very strange, all the test is passed. Only your specific data cause a problem.
We need to make a minimum test data which reproduce buffer overflow and adding test case to the suite. Could you help to prepare test data?
miurahr
commented
4 years ago Update #202 PR to extend a buffer when more data come, and drop BufferOverflow exception. It does not generate buffer overflow anymore, but it potentially can consume a memory.
@EdwardChamberlain Could you try it?
miurahr
commented
4 years ago v0.8.5 released. Does it solve your problem?
ghost
commented
4 years ago It would be nice to provide a special version (or one or two .py files) to @EdwardChamberlain , that can print information about internal states to help diagnose the problem.
ghost
commented
4 years ago If I remove the password then I dont get the BufferOverflow but I do get a MemoryError (which I suspect is due to me using a raspberry pi)
IMO this exception looks like a program bug.
https://github.com/miurahr/py7zr/blob/v0.8.3/py7zr/helpers.py#L377-L378
EdwardChamberlain
commented
4 years ago @EdwardChamberlain Could you try it?
I will take a look sometime next week, I am away currently.
miurahr
commented
4 years ago V0.8.5 and v0.9.2 improve there and does not raise anymore.
https://github.com/miurahr/py7zr/blob/v0.8.5/py7zr/helpers.py#L370-L374
def add(self, data: Union[bytes, bytearray, memoryview]):
length = len(data)
self._buf[self._buflen:] = data
self._buflen += length
self.view = memoryview(self._buf[0:self._buflen])There is no further claim.
Describe the bug Zipping an SQLite .db file results in error:
file size: 16.9 mb
Expected behavior Creates a zipped file
Environment (please complete the following information):