search

GaloisInc / BESSPIN-Tool-Suite

The core tool of the BESSPIN Framework.
Other
5 stars 2 forks source link

Scoring Computation Error #1106

Closed njshanahan closed 3 years ago

njshanahan commented 3 years ago

I created a configuration that excluded PPAC from vulClasses because PPAC-3 produces a CALL-ERR score. This prevented the computeBesspinScale score from being calculated. The error messages are below.

With PPAC included.

(Error)~  computeBesspinScale: CWE-PPAC_3 score is CALL-ERR. Cannot compute the scale for scores not in [HIGH,MED,LOW,NONE,DETECTED,NoImpl]

Without PPAC included.

(Warning)~  computeNaiveCWEsTally: The overall score won't be computed since <S(PPAC)> is missing.
...
(Warning)~  computeBesspinScale: The BESSPIN Scale won't be computed since <S(PPAC)> is missing.

I imagine correcting the PPAC-3 CALL-ERR will resolve the issue. The log file is attached below.

test_PPAC_3.log

rtadros125 commented 3 years ago

Interesting; Thanks I want to say 4 things:

  1. The decision that the BESSPIN Scale won't be computed unless all needed score are provided, is to make sure the figure does not get skewed. So to make sure we compare apples to apples, we are going to stick with this error.
  2. You can always work around that by using the self-assessment feature. For the class that you don't run, turn useSelfAssessment ON, and fill the values in the corresponding ini file, and the tool will load these results instead of running the tests. Also, every time you run any class, an .ini file gets generated in workDir/cwesEvaluation/vulClass so you can copy/paste into your configSecurity .ini if you prefer to.
  3. The error happens because kern.log is not found. This is a bug. I will open a PR to fix this soon (and the score will be thus HIGH). [Opened #1107]
  4. This error should not have happened, but because you removed rsyslog as @podhrmic has asked, it did. My suggestion would be, for LMCO to score NONE on that test as it should: please build two debian images, one with rsyslog and one without. Add them to the same directory (or send them to me), will name them debian.elf and debian_noRsyslog.elf. When you do so, I will modify the cyberphys default config to use a custom OS Image, and adjust the path to debian_noRsyslog.elf.

Please lmk if something is unclear or if you suggest an alternative courses of action.

rtadros125 commented 3 years ago

The small fix is merged. Won't close this ticket till we have the two debian binaries.

njshanahan commented 3 years ago

I uploaded the two Debian images in GitLab issue 79114.

rtadros125 commented 3 years ago

I uploaded the two Debian images in GitLab issue 79114.

We need to think about a better way to transfer binaries in the future. We don't want to exceed our git limits and such, especially that keeping them there is not needed. Let's discuss this when we need some files in the future.