search

packing-box / docker-packing-box

Docker image gathering packers and tools for making datasets of packed executables and training machine learning models for packing detection
GNU General Public License v3.0
49 stars 10 forks source link

`detector` in binary mode gives false result #63

Closed RomainJennes closed 1 year ago

RomainJennes commented 1 year ago

Detectors that allow multiclass detection can't be used in binary mode. The result is that they classify all executables as packed.

Preparation steps

$ dataset make baseline -n 100 -f PE -p upx

Issue

PEiD, for example, gives correct results in multiclass mode:

$ detector baseline -d PEiD
<<snipped>>
Detection results:
  ────────  ─────────  ──────  ─────────
  Accuracy  Precision  Recall  F-Measure
  83.00%    100.00%    63.04%  77.33%
  ────────  ─────────  ──────  ─────────

But fails in binary mode because classes everything as packed:

$ detector baseline -b -d PEiD
<<snipped>>
Detection results:
  ────────  ─────────  ───────  ─────────
  Accuracy  Precision  Recall   F-Measure
  29.00%    29.00%     100.00%  44.96%
  ────────  ─────────  ───────  ─────────

In contrary, detectors that can only work in binary mode give good results:

$ detector baseline -b -d REMINDer
<<snipped>>
 Detection results:
  ────────  ─────────  ───────  ─────────
  Accuracy  Precision  Recall   F-Measure
  97.00%    93.88%     100.00%  96.84%
  ────────  ─────────  ───────  ─────────