Warnings about non-registered check systems confuse SCAP Workbench

[jan-cerny]

Warnings about non-registered check systems introduced in https://github.com/OpenSCAP/openscap/pull/1055 cannot be parsed by SCAP Workbench, which then produces bunch of warnings:

10:46:24 
info     
SCAP Workbench 1.2.0, compiled with Qt 5.9.6, using OpenSCAP 1.3.0

10:46:26 
info     
Opened file '/usr/share/xml/scap/ssg/content/ssg-fedora-ds.xml'.

10:46:32 
info     
Querying capabilities...

10:46:32 
info     
Creating temporary files...

10:46:32 
info     
Starting the oscap process...

10:46:32 
info     
Processing...

10:46:36 
warning  
Error when parsing scan progress output from stdout of the 'oscap' process. ':' encountered while not reading rule ID, newline and/or rule result are missing! Read buffer is 'WARNING'.

10:46:36 
warning  
Error when parsing scan progress output from stdout of the 'oscap' process. ':' encountered while not reading rule ID, newline and/or rule result are missing! Read buffer is ' Skipping rule that requires an unregistered check system or incorrect content reference to evaluate. Please consider providing a valid SCAP/OVAL instead of http'.

10:46:36 
warning  
Error when parsing scan progress output from stdout of the 'oscap' process. Newline encountered while reading rule ID, rule result and/or ':' are missing! Read buffer is 'notchecked'.

10:46:36 
warning  
Error when parsing scan progress output from stdout of the 'oscap' process. ':' encountered while not reading rule ID, newline and/or rule result are missing! Read buffer is 'WARNING'.

10:46:36 
warning  
Error when parsing scan progress output from stdout of the 'oscap' process. ':' encountered while not reading rule ID, newline and/or rule result are missing! Read buffer is ' Skipping rule that requires an unregistered check system or incorrect content reference to evaluate. Please consider providing a valid SCAP/OVAL instead of http'.

10:46:36 
warning  
Error when parsing scan progress output from stdout of the 'oscap' process. Newline encountered while reading rule ID, rule result and/or ':' are missing! Read buffer is 'notchecked'.

[yuumasato]

Is this caused by the rule security_patches_up_to_date?

[jan-cerny]

@yuumasato No, it is not.

In my case it is because xccdf_org.ssgproject.content_rule_chronyd_or_ntpd_specify_remote_server and xccdf_org.ssgproject.content_rule_service_chronyd_or_ntpd_enabled in Fedora benchmark in scap-security-guide-0.1.39-2.fc27.noarch reference only OCIL, they don't reference OVALs.

The output of evaluating Standard Fedora Profile by new oscap:

oscap xccdf eval --profile standard /usr/share/xml/scap/ssg/content/ssg-fedora-ds.xml 
Title   Specify a Remote NTP Server                 
Rule    xccdf_org.ssgproject.content_rule_chronyd_or_ntpd_specify_remote_server                          
WARNING: Skipping rule that requires an unregistered check system or incorrect content reference to evaluate. Please consider providing a valid SCAP/OVAL instead of http://scap.nist.gov/schema/ocil/2
Result  notchecked                                  

Title   Enable the NTP Daemon                       
Rule    xccdf_org.ssgproject.content_rule_service_chronyd_or_ntpd_enabled                                
WARNING: Skipping rule that requires an unregistered check system or incorrect content reference to evaluate. Please consider providing a valid SCAP/OVAL instead of http://scap.nist.gov/schema/ocil/2
Result  notchecked                                  

Title   Disable SSH Access via Empty Passwords      
Rule    xccdf_org.ssgproject.content_rule_sshd_disable_empty_passwords                                   
Result  error                                       

Those lines starting with WARNING: aren't supported by SCAP Workbench.

[mpreisler]

@cipherboy I think we shoudl whitelist OCIL as well to prevent this from happening. And SCAP Workbench needs to learn about these messages and show them in a nicer way.

[jan-cerny]

There has been a change in OpenSCAP 1.3.2 introduced in https://github.com/OpenSCAP/openscap/pull/1376. This PR moves the message "Skipping rule ..." from stdout to stderr. As a result, SCAP Workbench behaves differently. The message is presented as error, it doesn't complain about parsing problems, and the "notchecked" result is shown in main Workbench window.

This can be seen as an improvement, because the error message goes straight to the point, and the result is not missing in the main SCAP Workbench Window.

However, there are multiple problems:

• Missing OVAL is not probably an error, it's a valid situation, allowed by the XCCDF standard, it's perfectly OK to have XCCDF rules without OVALs.
• SCAP Workbench considers everything that oscap writes on stderr as an error.
• The worst problem still remains: SCAP Workbench can be easily confused by anything that oscap process writes on stdout and stderr. The parsing is very fragile. As can be seen from the first comment, it's broken by presence of a colon on stdout. On OpenSCAP codebase there are many different error messages and it isn't in our power to test every of them with SCAP Workbench.

I suggest addressing it by re-thinking this integration and implementing something less fragile, maybe not dependent on usual stdout and stderr, but on some nice output format instead. Any ideas?