Validate mandatory tests

[cgi1]

Pre: #10

Please also add CLI arg to skip mandatory tests, because the user could also fix the input in Secvisogram.

Desc

Only the relevant parts are written in the specification, the full examples are written here. They are failing (at least) for the specific cases.

The naming of the examples is encoded like this:

OASIS_CSAF_TC-CSAF_2_0-2021-6-1-05-01.json

6.1.05 here means, that this is a test file for Mandatory test 6.1.5

--> All mandatory tests have already been implemented by secvisogram already. More detailed here

[cgi1]

@tschmidtb51: We understand, that it will be perfect to execute the mandatory tests - implemented in secvisogram - as part of the Github Actions chain.

Are you aware on how to call the validation functions from the CLI? Then we will start with implementing it into Github Actions to be compliant with the mandatory tests right from the start.

Furthermore, can we assume that secvisogram is completly validating all mandatory tests here?

[tschmidtb51]

@tschmidtb51: We understand, that it will be perfect to execute the mandatory tests - implemented in secvisogram - as part of the Github Actions chain.

The standard requires that a CVRF CSAF converter satisfies the "CSAF producer" conformance profile. The CSAF producer produces output in the CSAF format, according to the conformance profile "CSAF document" which states that a CSAF document does not fail any mandatory test defined in section 6.1. Therefore, the CVRF CSAF converter SHALL check (as the last step) whether the generated CSAF output is a valid CSAF document. The tests have been implemented in Secvisogram. However, just running them as GH-Actions would not be enough as the mandatory test should be an part of the CVRF CSAF converter.

Are you aware on how to call the validation functions from the CLI? Then we will start with implementing it into Github Actions to be compliant with the mandatory tests right from the start.

Unfortunately, not. I think that is not implemented at the moment.

Furthermore, can we assume that secvisogram is completly validating all mandatory tests here?

No. Not all test have been implemented yet. Watch secvisogram/secvisogram#36: Once it closes all test have been implemented. The entry point is here - some tests are already extracted into separate files.

[tschmidtb51]

Maybe existing projects, like sthagen/csaf-lint could help in the implementation?

Flagging @sthagen for attention.

[cgi1]

Thank you for the explanation @tschmidtb51!

So of course it would be great to use a common libary in order to execute the mandatory tests in both cases - Continous Testing as part of Github Actions as well as on the converted file into JSON as part of cvrf2csaf.py. For further maintainance aspects, I would also suggest to not re-implement all the mandatory checks at different places.

From the code I see, that sthagen/csaf-lint already comes with XML and JSON schema validation.

Invoking csaf-lint would therefore very much fullfill this requirement.

Sadly, I did not found any Validation Test implementations in sthagen/csaf-lint, which checks the Mandatory Tests.

The question arises at this point, if we should invoke calling the JS secvisogram checks here*, or if there will be a native python implementation of the checks (e.g. in csaf-lint or another project)?

*(e.g. using Js2Py. But this can cause code translating issues, so if available, we would prefer a python implementation.

[sthagen]

Luckily, I might provide the mandatory "business level" tests within short notice as a native python implementation.

[sustefil]

Luckily, I might provide the mandatory "business level" tests within short notice as a native python implementation.

Hi @sthagen , any luck with the mandatory tests implementation?

[sthagen]

@sustefil, @tschmidtb51 ... my promise is still not broken :wink:

[cgi1]

There is a new project by @sthagen which targets mandatory tests in python: turvallisuusneuvonta

Right now, only part of the mandatory tests are implemented yet.

We agreed with @tschmidtb51, that for now, we will integrate the execution of these tests into cvrf2csaf.py.

[zpevma]

Mandatory tests from @sthagen will be integrated in #55.