packing-box / docker-packing-box

Docker image gathering packers and tools for making datasets of packed executables and training machine learning models for packing detection
GNU General Public License v3.0
49 stars 10 forks source link

`KeyError` on `dataset convert dataset-name` #80

Closed AlexVanMechelen closed 1 year ago

AlexVanMechelen commented 1 year ago

Description

KeyError: 'STANDARD_SECTION_NAMES' appears when running dataset convert dataset-name on the latest commit 0eb04cd.

Error debug output

┌──[user@packing-box]──[/mnt/share]────────                                                                                                                         ────[10.0.2.100]──[11:55:15]────
$ dataset convert test-pe-upx
00:00:00.452 [INFO] Converting to fileless dataset...
00:00:00.454 [INFO] Size of dataset:     49MB
00:00:00.455 [INFO] Loading features...
  0% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0/100 samples • 0:00:02 • -:--:--
Traceback (most recent call last):
  File "/home/user/.opt/tools/dataset", line 239, in <module>
    getattr(ds, args.command)(**vars(args))
  File "/home/user/.local/lib/python3.11/site-packages/pbox/helpers/items.py", line 308, in _wrapper
    return f(s, *a, **kw)
           ^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/pbox/core/dataset/__init__.py", line 368, in convert
    self._compute_all_features()
  File "/home/user/.local/lib/python3.11/site-packages/pbox/core/dataset/__init__.py", line 193, in _compute_all_features
    d = self._compute_features(exe)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/pbox/core/dataset/__init__.py", line 201, in _compute_features
    d.update(exe.data)        # be sure to include the features
             ^^^^^^^^
  File "/usr/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/pbox/core/executable/__init__.py", line 131, in data
    return Features(self)
           ^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/tinyscript/preimports/log.py", line 85, in _wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/pbox/core/executable/features.py", line 89, in __init__
    self._rawdata = Extractors(exe)
                    ^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/pbox/core/executable/extractors/__init__.py", line 60, in __init__
    self[name] = func(exe)
                 ^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/pbox/core/executable/extractors/__common__.py", line 38, in _wrapper
    return f(parse(str(target)), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/pbox/core/executable/extractors/__common__.py", line 59, in <lambda>
    standard_sections         = parse_binary(lambda exe: [s.name for s in exe.sections if s.name in \
                                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/lib/python3.11/site-packages/pbox/core/executable/extractors/__common__.py", line 60, in <listcomp>
    get_data(exe.format.name)['STANDARD_SECTION_NAMES']])
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'STANDARD_SECTION_NAMES'

Steps to reproduce

$ git clone git@github.com:packing-box/docker-packing-box.git
$ cd docker-packing-box/
$ docker build -t dhondta/packing-box .
$ docker run -it -h packing-box -v `pwd`:/mnt/share dhondta/packing-box
$ dataset make test-pe-upx -n 100 --format PE --packer upx
$ dataset convert test-pe-upx
dhondta commented 1 year ago

Hi @AlexVanMechelen ! You can retry your last command in verbose mode (dataset -v convert test-pe-upx)and check the latest lines of debug information before the traceback of the error. I expect you to see the path to the target executable for which information extraction fails. You will have a line of this type : pefeats '/home/user/.packing-box/datasets/test-pe-upx/files/[sha256] You can then execute the following command to see the file type and post it on this issue ; file /home/user/.packing-box/datasets/test-pe-upx/files/[sha256]

AlexVanMechelen commented 1 year ago

file /home/user/.packing-box/datasets/test-pe-upx/files/[sha256] output:

PE32+ executable (DLL) (console) x86-64, for MS Windows, 19 sections
dhondta commented 1 year ago

@AlexVanMechelen Please send me the failing sample via email.