Policy check VALID, even if all fields are N/A

[pjotrek-b]

I've tried checking a completely invalid/broken AVI file. The implementation checker of course reports "NOT VALID".

But: The policy report result in the overview is misleading, as it displays a comforting, green "VALID". Yet all values are listed as "N/A".

I think it would be better to have it fail as "NOT VALID" in such a case?

[JeromeMartinez]

I guess you test a policy which has no constraints about the file format. From my point of view, a file with no test applied (all test "N/A") is as applying a policies set with 0 policy. So this is more a philosophical question: what is the result of a policy check with 0 policy? From my point of view, it is "VALID" and if we want to force a "NOT VALID", we need to add a test e.g. "must have at least 1 video stream".

I propose a 3 status: "NOT VALID" (at least 1 check fails) "VALID" (0 check fails and at least 1 check passes) "N/A" (0 check fails and 0 check passes) @dericed your thoughts?

[dericed]

np, I'll implement this for the January release

[ablwr]

+1

XSL is a fan of delivering false positive results much more often than other languages.

[t-rapp]

I also was confused when successfully testing a file with no audio stream against a policy which includes tests like Audio Format == PCM and Audio Channels >= 2.

Maybe there could be some validators added like "exists and equals (===)" to allow for stricter checks where "N/A" is treated as "fail"?

[JeromeMartinez]

@dericed Another remark related to this "valid" stuff: "valid" is too much, should be "passed" (=all tests we have made passed)

[JeromeMartinez]

Maybe there could be some validators added like "exists and equals (===)" to allow for stricter checks where "N/A" is treated as "fail"?

I am not in favor of this, because I expect that the test is per stream: if the stream does not exist, this is OK (no test to do on non existing streams, in other words it is "for each audio stream Audio format == PCM" so if no stream it is a pass) and tests passed. For the issue on non existing stream, this would be a specific test (e.g. "count of audio streams must be 1")

[t-rapp]

OK, this makes sense. Is there already some way to enter a rule like "count of audio streams must be 1" in the MediaConch GUI? I tried but could not find out how...

[JeromeMartinez]

Is there already some way to enter a rule like "count of audio streams must be 1" in the MediaConch GUI?

Not yet. On our ToDo-list. cc @dericed

[dericed]

I think on the operator level we could distinguish and have 'equals' and 'equals if exists'.

[JeromeMartinez]

I think on the operator level we could distinguish and have 'equals' and 'equals if exists'.

I think it is uselessly complex (you double the count of operators for just a test on the existence because it is not only about ==, it is for all operators). There are 2 different tests (existence and match of a value), so I prefer 2 distinct tests ("stream exists" and "when exists, format is PCM").

Note also that with 2 operators we'll be quickly limited (e.g. not possible to say "if count of audio is 2 audio 1 must be AAC and audio 2 must be AC-3, if count of audio is 8 all audio must be PCM") due to the mix of 2 different tests in a single operator so we can not expand them easily.

[t-rapp]

I think on the operator level we could distinguish and have 'equals' and 'equals if exists'.

As far as I understand from @JeromeMartinez' comment the current 'equals' already behaves as 'equals if exists'?

[pjotrek-b]

Daily build "16.09.20161030": Policy check shows "N/A" instead of pass now :smile:

I'd consider this closed.