policy_evaluated statuses

[jkrehm]

Is there a particular reason the statuses from policy_evaluated are not imported into the record? I'm not a DMARC expert (just figuring things out as I go) but that's usually the field that tells me something's wrong.

For example:

  <record>
    <row>
      <source_ip>104.47.37.209</source_ip>
      <count>1</count>
      <policy_evaluated>
        <disposition>none</disposition>
        <dkim>pass</dkim>
        <spf>fail</spf>
      </policy_evaluated>
    </row>
    <identifiers>
      <header_from>...</header_from>
    </identifiers>
    <auth_results>
      <dkim>
        <domain>...</domain>
        <result>pass</result>
      </dkim>
      <spf>
        <domain>nam02-cy1-obe.outbound.protection.outlook.com</domain>
        <result>pass</result>
      </spf>
    </auth_results>
  </record>

SPF passed, but DMARC failed. When I imported this particular file, the database has nothing about it failing, only passing. Is it considered useless information?

If it would be useful to the project (because I think it's useful) I can submit a pull request so it's stored in the database.

[solarissmoke]

It's not useless at all, and all the information in that field is imported here into the rptrecord table. This contains dkim_result and spf_result fields that tell you whether each of these passed.

Where in the database are you looking?

[jkrehm]

dkim_result and spf_result are coming from record > auth_results > dkim/spf > result. I'm referring to record > policy_evaluated > dkim/spf. I don't see where that information is stored, or even referenced.

The record > policy_evaluated > disposition and record > policy_evaluated > reason > type fields are stored, but not the pass/fail status of DKIM & SPF.

[jkrehm]

From http://www.ietf.org/mail-archive/web/dmarc/current/msg00632.html

The relevance of the underlying results to a DMARC policy evaluation; this is the policy_evaluated section of an aggregate report. Any time the SPF domain is unaligned a fail will appear here, even if SPF passed (and therefore shows a pass in the auth_results section).

That's what I've found. It's the most useful part of the report to me, because it tells me if someone is using our domain to send emails.

[jkrehm]

Perhaps some new fields, e.g. dkim_dmarc and spf_dmarc? If you're interested, I can send a pull request adding those fields in. The data's all there, just isn't getting stored right now.

[solarissmoke]

Thanks for that - I understand now. Yes, it makes sense to store this. From reading the list archive you linked to, it seems that the spf/dkim result that is currently being stored is not particularly useful - in fact there can be multiple such entries which the current code would fail to handle, as it is expecting only one of each.

I'm tempted to suggest using the existing fields (dkim_result, spf_result) to store this information, and stop storing the auth_results, unless you or someone else has a use for them. This would avoid a schema change. PR welcome.

[jkrehm]

The domain from auth_results is definitely useful to me, but from their example there I think you're correct that the current code will break (or not work as expected) if there's duplicate nodes in the auth_results node.

Have you come across a standard for any of this stuff? I'm beginning to wonder if it's all kind of half-baked and not worth my time at this point. Something as basic as how many nodes you should expect really ought to be stated somewhere. Or perhaps we assume there could be one or more and change the code to function that way? The current key structure wouldn't like that, though.

[solarissmoke]

The formal specification is here, and if you look at the schema definition at the end (assuming you're into that kind of torture ;-)) then it does indeed permit any number (minimum 1) of spf and dkim elements inside auth_results. We would need another table to store that information properly.