PSD DMARC registry list not embedded

[dvzrv]

When packaging 0.13.0 I had a closer look at how to devendor/provide the different lists. While we do package publicsuffix-list on Arch Linux (and I can actually just symlink our version of it in place), I realized, that although setup.py implies, that there is an embedded version of the PSD DMARC registry list, there actually is none in authheaders/psddmarc.csv.

Looking at https://psddmarc.org it's also very intransparent where this list gets generated (if at all), how often it will change, where its upstream is and who is actually responsible for it.

I plan on adding it to python-authheaders for now, but maybe there's a better way and you know more about the list's background?

[kitterma]

The list is based on what's described in this Internet Draft (hopefully eventually RFC):

https://datatracker.ietf.org/doc/draft-ietf-dmarc-psd/

If you don't generate the psddmarc.csv, then authheaders falls back to doing an RBL style lookup. Either will give the same result. It's a trade off between the extra DNS lookup and the risk the downloaded csv will get out of date. Personally, I think it's a small risk since the list changes rarely (It's changed once in a year and a half).

[dvzrv]

@kitterma thanks for the clarification!

Is there some git repository or similar from which the .csv is generated? Would be great to include a fixed version of the file (if possible).

[kitterma]

You can see a link to the csv here: http://www.psddmarc.org/registry.html

You can download the file via setup.py using:

python3 setup.py psddmarc

That does not, however, cause it to be installed when calling build/install. You'd need to do that as part of your build system.

The list isn't in git because it's only 5 lines long and rarely changes.

[niftylettuce]

Is this the reason why I get this error in travis (when using a build source)?

pip3 install git+https://github.com/forwardemail/authentication-headers.git

Later...

Traceback (most recent call last):␊
      File "/home/travis/.local/lib/python3.5/site-packages/authheaders/dmarc_lookup.py", line 137, in get_suffix_list_file_name␊
        from authheaders.findpsl import location␊
    ImportError: No module named 'authheaders.findpsl'␊
    ␊
    During handling of the above exception, another exception occurred:␊
    ␊
    Traceback (most recent call last):␊
      File "/home/travis/build/forwardemail/authheaders/scripts/authenticate-message.py", line 33, in <module>␊
        main()␊
      File "/home/travis/build/forwardemail/authheaders/scripts/authenticate-message.py", line 24, in main␊
        header = authheaders.authenticate_message(msg=message, authserv_id=authservId, ip=ip, mail_from=mailFrom, helo=helo, spf=True, dkim=True, arc=True)␊
      File "/home/travis/.local/lib/python3.5/site-packages/authheaders/__init__.py", line 291, in authenticate_message␊
        dmarc_result = check_dmarc(msg, spf_result, dkim_result, dnsfunc=dnsfunc, psddmarc=psddmarc)␊
      File "/home/travis/.local/lib/python3.5/site-packages/authheaders/__init__.py", line 241, in check_dmarc␊
        result, result_comment, from_domain, policy = dmarc_per_from(from_domain, spf_result, dkim_result, dnsfunc, psddmarc)␊
      File "/home/travis/.local/lib/python3.5/site-packages/authheaders/__init__.py", line 204, in dmarc_per_from␊
        elif adkim == "r" and get_org_domain(from_domain) == get_org_domain(dkim_result.header_d):␊
      File "/home/travis/.local/lib/python3.5/site-packages/authheaders/dmarc_lookup.py", line 115, in get_org_domain␊
        fn = get_suffix_list_file_name()␊
      File "/home/travis/.local/lib/python3.5/site-packages/authheaders/dmarc_lookup.py", line 138, in get_suffix_list_file_name␊
        except ModuleNotFoundError:␊
    NameError: name 'ModuleNotFoundError' is not defined

[niftylettuce]

Actually, I just saw that the version is v3.5 and not v3.8+, will try again.

[kitterma]

Note: This is RFC 9091 now. I'm going to close this since the build system provides a way to include the registry list.