Missing DMARC-policy due to no DMARC at all is not counted as 'fail'

[robinbzk]

DMARC-policy shows a warning in the scan result details of the dashboard if DMARC-existence is false. As the higher test failed, the lower test DMARC-policy should also fail.

The logic should be the same as DNSSEC existence and DNSSEC validity.

Now you get strange graphs where DMARC-existence has lower percentage than DMARC-policy.

Here DMARC-policy should show a failure symbol, and be counted as a fail.

[robinbzk]

Something similar happens with DANE rollover scheme. Probably the solution is to count warnings and info signs as fails where diagrams are created.

[stitch]

This is a representation of the score: only failures are counted in the statistics: all warnings and info do not cause a "point deduction" in the internet.nl scans, and that logic has been ported to the dashboard.

The big question is: how would we represent situations where there can be failure, warning, info and good? What should be subtracted from the bar and what not? Should the bar only denote success (and ignore not-tested/not-applicable)?

[robinbzk]

In my opinion there are only 4 possible statuses (PASS, FAIL, not_applicable, not_testable) that can have an effect on the statistics in the dashboard (not talking about the details).

The 'severity-rating' we give this in the Public Internet.nl interface should not be used in creating statistics.

Therefore WARNING or INFO is essentially the same as a FAIL for the statistics part.

In the specific case of DMARC policy, I think there is an error in the logic, because in the public Internet.nl interface there is no warning icon for DMARC policy if you fail.

[stitch]

The dmarc policy will be updated in the api v2 update end of next week.

I can alter the statistics to take into account "warning" and "info". The good_not_tested and not_tested will still be ignored.

Does that work for you?

[robinbzk]

I can alter the statistics to take into account "warning" and "info". The good_not_tested and not_tested will still be ignored.

Does that work for you?

Yes. E.g. the following results (for a random test) would result in 2 out of 3 passed = 66,6% adoption rate:

Domain -- Result domain1.xyz -- not_tested domain2.xyz -- good_not_tested domain3.xyz -- warning domain4.xyz -- passed domain15.xyz -- passed

^Unless… the test for domain1.xyz wasn't performed because a pre-conditional test already failed. In that case, _nottested would logically be the same as failed. The result would then be 2 out 4 of passed = 50% adoption rate.

Example: https://internet.nl/site/www.uwv.nl/867998/# Because the test IPv6 addresses for web server has failed, test for IPv6 reachability of web server and Same website on IPv6 and IPv4 are not performed (for efficiency). But the result should be counted as a failure in the aggregate statistics of the dashboard.

@baknu are you aware of this?

[stitch]

Because the test IPv6 addresses for web server has failed, test for IPv6 reachability of web server and Same website on IPv6 and IPv4 are not performed (for efficiency). But the result should be counted as a failure in the aggregate statistics of the dashboard.

Which makes "not tested" values relevant and dependent on other values. This means not processing the API values as face value, which i assume will also be relevant in many other situations.

[stitch]

Maybe i made a mistake: the graphs currently only count "passed". So all not_tested, failed, good_not_tested is all ignored. If we only count passed, and ignore the rest: would that work?

[robinbzk]

If we only count passed, and keep the total population the same, then that would probably work regarding IPv6 specifically.

In general, regarding graphs, perhaps the most neutral solution is to simply count all categories. So we would get stacked bar charts containing passed, failed, not testable and not applicable.

We are also working on some text explaining how we think the 'not_tested' results should be interpreted. In general the following rule seems to apply: in most cases where there is a 'not_tested' result, it is because of a implicit parent-child relationship with another test (supertest?). Therefore a not_tested results inherits the result of the parent test.

Maybe best to discuss this in a call before we make a choice and implement anything.

[stitch]

It makes sense. We have to come up with an idea how that works in comparisons: color / shape wise. Perhaps the same colors in the chart but another border color per report.

Op ma 18 mei 2020 om 16:06 schreef robinbzk notifications@github.com

Bart and I discussed this last Friday, perhaps the most neutral solution is to simply count all categories. So we would get stacked bar charts containing passed, failed, not tested and not applicable. Maybe best to discuss this in a call before we make a choice and implement anything.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/internetstandards/Internet.nl-dashboard/issues/147#issuecomment-630205672, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACRQFTOQOX4UGHEE7DBNC3RSE6GNANCNFSM4LBRGISQ .

-- Elger Jonker Creator, Builder, Maker, Hacker.

+31 (0)6 1342 5622 (+31 61 dial mac)

http://www.sha2017.org - enormous international hacker gathering http://www.awesomeretro.org - retro gaming foundation http://www.awesomespace.nl - workplace of retrogaming awesomeness http://www.elgerjonker.nl - curriculum vitae / weblog https://www.hack42.nl - hackerspace arnhem http://www.raveradio.nl - housemusic revival show http://greenpoint.space - dutch 1992 greenpoint mobile network http://twilight-cd.com - overview of classic bootleg software collection

This mail was sent using Lotus Notes 4.0.1

[robinbzk]

See https://github.com/internetstandards/Internet.nl-dashboard/issues/197 for desired solution.