search

elastic / ember

Elastic Malware Benchmark for Empowering Researchers
Other
953 stars 279 forks source link

Unable to run python classify_binaries.py -m [/path/to/model] BINARIES #35

Closed KathiBrown closed 3 years ago

KathiBrown commented 4 years ago

Hi, i already run ember_train.py which gives me the following output: Training LightGBM model /usr/local/lib/python3.6/dist-packages/lightgbm-2.3.1-py3.6-linux-x86_64.egg/lightgbm/engine.py:148: UserWarning: Found num_iterations in params. Will use it instead of argument warnings.warn("Found {} in params. Will use it instead of argument".format(alias)) [LightGBM] [Warning] objective is set=binary, application=binary will be ignored. Current value: objective=binary [LightGBM] [Warning] objective is set=binary, application=binary will be ignored. Current value: objective=binary [LightGBM] [Warning] Starting from the 2.1.2 version, default value for the "boost_from_average" parameter in "binary" objective is true. This may cause significantly different results comparing to the previous versions of LightGBM. Try to set boost_from_average=false, if your old models produce bad results [LightGBM] [Info] Number of positive: 300000, number of negative: 300000 [LightGBM] [Info] Total Bins 212045 Killed

Seems like something went wrong. Shouldn't it give a model.txt file as an output on which i could run the classify_binaries.py module?

I hope you find some time to help me. Best regards Kathi

mrphilroth commented 4 years ago

Hi Kathi. I get the same warnings when I trained the EMBER model that I shipped. But the line specifying the Total Bins is a bit different:

[LightGBM] [Info] Number of positive: 300000, number of negative: 300000
[LightGBM] [Info] Total Bins 213649
[LightGBM] [Info] Number of data: 600000, number of used features: 2327
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf

And it continues training from there.

I tried with LightGBM version 2.2.3 and 2.3.0 (the latest I could get from conda). So I guess I can't reproduce your error.

KathiBrown commented 4 years ago

Hi Kathi. I get the same warnings when I trained the EMBER model that I shipped. But the line specifying the Total Bins is a bit different:

[LightGBM] [Info] Number of positive: 300000, number of negative: 300000
[LightGBM] [Info] Total Bins 213649
[LightGBM] [Info] Number of data: 600000, number of used features: 2327
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf

And it continues training from there.

I tried with LightGBM version 2.2.3 and 2.3.0 (the latest I could get from conda). So I guess I can't reproduce your error.

Thx anyway:)