lookups not beeing populated

[DBorGe]

Hi @aholzel , can you clarify me something please. I have my app setup and running, however i have some issues.

• The lookups (dmarc_email_domains.csv and spf_resolving_filters.csv) are not beeing populated.
• All my SPF alligments in the overview dashboard are unaligned.

steps I did: I've enabled the I've downloaded SA-CIM and added my domain to the lookup cim_corporate_email_domains.csv Now, running the following: |dmarc_domains("domain, spf_lookup_script")` I have table with my domain, and spf_lookup_script=false. The dmarc_spf_check.py is running but it seems not doing nothing, as I only have this output on the logs: 2020-08-20 17:40:00,297 loglevel=DEBUG file=dmarc_spf_check.py line=63 message="Splunk SessionKey provided by stdin" 2020-08-20 17:45:00,335 loglevel=DEBUG file=dmarc_spf_check.py line=63 message="Splunk SessionKey provided by stdin" 2020-08-20 17:50:00,310 loglevel=DEBUG file=dmarc_spf_check.py line=63 message="Splunk SessionKey provided by stdin" 2020-08-20 17:55:00,298 loglevel=DEBUG file=dmarc_spf_check.py line=63 message="Splunk SessionKey provided by stdin" 2020-08-20 18:00:00,270 loglevel=DEBUG file=dmarc_spf_check.py line=63 message="Splunk SessionKey provided by stdin"`

Can you tell me what I might be doing wrong?

[aholzel]

hello @DBorGe

I just did a clean install of the app and added a "random" domain to the "dmarc_email_domains.csv" (no need for the SA-CIM that lookup is mainly there for people that use ES so they don't have to fill 2 lookups). I needed to do a couple of things before it worked:

1. Please make sure nslookup is available on your server (new Centos installs don't have it installed by default anymore) I will put this on my todo list to get rid of this dependency.
2. To let the script fill the "spf_mailservers.csv" lookup you need to set the "spf_lookup_script" to true in the "dmarc_email_domains.csv" lookup.
3. By default the "dmarc_spf_check.py" script is disabled so that needs to be enabled
4. By default the "dmarc_spf_check.py" only runs at midnight.

I put the domain "ns.nl" in the first lookup and set the "spf_lookup_script" to true and adjusted the "dmarc_spf_check.py" parameters to run a bit sooner :-) and got the expected results:

nslookup -q=txt ns.nl
Server:  Firewall
Address:  192.168.1.1

Non-authoritative answer:
<<SNIP>>
        "v=spf1 ip4:145.15.111.0/24 ip4:159.135.138.110/32 ip4:37.188.118.32/32 ip4:213.206.247.116/32 ip4:145.15.108.0/24 ip4:52.210.124.125/32 ip4:52.48.44.248/32 include:spf.allsolutions.nl include:spf1.ns.nl -all"

the lookup content:

prt ip  mail_server_group
SPF_CIDR-199.127.232.0/22   199.127.232.0/22    ns.nl
SPF_CIDR-2a01:4180:4051:0800::/64   2a01:4180:4051:0800::/64    ns.nl
SPF_CIDR-148.105.8.0/21 148.105.8.0/21  ns.nl
SPF_PTR-mailer15.allsolutions.nl    83.219.86.200   ns.nl
SPF_PTR-mailer16.allsolutions.nl    83.219.86.201   ns.nl
SPF_PTR-mailer17.allsolutions.nl    83.219.86.202   ns.nl
<<SNIP>>

[DBorGe]

Hi @aholzel thank you for your reply. I have added 2 domains to the dmarc_email_domains.csv and set the value to true. The script is running, and I see in the SA-dmarc.log file: 2020-08-24 16:35:00,347 loglevel=DEBUG file=dmarc_spf_check.py line=63 message="Splunk SessionKey provided by stdin" 2020-08-24 16:40:00,335 loglevel=DEBUG file=dmarc_spf_check.py line=63 message="Splunk SessionKey provided by stdin" 2020-08-24 16:45:00,404 loglevel=DEBUG file=dmarc_spf_check.py line=63 message="Splunk SessionKey provided by stdin" 2020-08-24 16:50:00,453 loglevel=DEBUG file=dmarc_spf_check.py line=63 message="Splunk SessionKey provided by stdin"

however, my spf_mailservers.csv is still empty. Please check the image bellow: https://imgur.com/AqQqi0m - you see my nslookup result, my inputs config, and dmarc_email_domains and spf_mailservers csvs

[DBorGe]

Hi @aholzel I've just noticed this on my SA-dmarc.log:

2020-08-24 17:15:00,304 loglevel=DEBUG file=dmarc_spf_check.py line=63 message="Splunk SessionKey provided by stdin" 2020-08-24 17:15:00,423 loglevel=ERROR file=dmarc_spf_check.py line=301 message="Found unknown record: 194.X.X.11" 2020-08-24 17:15:00,423 loglevel=ERROR file=dmarc_spf_check.py line=301 message="Found unknown record: 155.X.X.243" 2020-08-24 17:20:00,283 loglevel=DEBUG file=dmarc_spf_check.py line=63 message="Splunk SessionKey provided by stdin" 2020-08-24 17:20:00,379 loglevel=ERROR file=dmarc_spf_check.py line=301 message="Found unknown record: 155.X.X.243" 2020-08-24 17:20:00,380 loglevel=ERROR file=dmarc_spf_check.py line=301 message="Found unknown record: 194.X.X.11"

i translated the ips and it translate to the domain, of the client, but not exactly the same. For example, my domain dmarc_email_domains as domain "xpto.pt" however, the ip above translates to: mail.company-xpto.pt and mail2.company-xpto.pt. However, the domain in the RUA report is the xpto.pt.

[aholzel]

@DBorGe I will have a look at is tonight, I think it might be the MX record that isn't resolved correct

[DBorGe]

thanks a lot @aholzel learning a lot of dmarc, spf and dkim with your app 👍

[aholzel]

@DBorGe Good to hear that the app is helping you! thanks for that feedback.

I created a new release that hopefully resolves the resolving problems. SA-dmarc v3.7.5

[DBorGe]

Hi @aholzel thanks a lot. now its creating the spf_mailservers.csv. However I still got some questions:

1. Now in Dmarc Overview, I see the "PTR of source IP" like this: "SPF_CIDR-..../32" is this supposed? Because before I saw the actual domain which was better I think.

2. All the spf for my domain are unaligned, that happens because if I hit "nslookup -q=txt " it fails, but if i hit "nslookup -q=txt 8.8.8.8". I suppose that if i change the script to do bellow instead, it should be fine right? "output = run_process("nslookup -q=" + str(record_type) + " " + str(domain) + " 8.8.8.8", resolve_timeout)"

[aholzel]

Hello @DBorGe

I have just create 2 release one that just fixes another problem with the csv writer and on that is not dependent on nslookup anymore and has probably fixed some of the problems you are facing.

1. That was on purpose (I am just not sure why anymore.. :confounded:) I changed it in the beta release to the ptr.

2. The spf is probably unaligned because of problem number 1. If the nslookup fails a "nslookup -q=txt <>" and it works with "nslookup -q=txt <> 8.8.8.8" than that means that the DNS server that is set systemwide can not resolve the query you are running and Google can. This can be caused by a split DNS or DNS view that is created on your dns server.

Can you test the beta release and see if that resolves your problems? changes are that I need to fix some excepions but I will work on that tomorrow or the day after that.

thanks in advance

[DBorGe]

Hi @aholzel thanks for the great feedback. Regarding the spf record, yes we have split dns, so I've configured the spf txt record inside and that was solved, however, i then started to have the following error: 2020-08-27 15:05:00,492 loglevel=ERROR file=dmarc_spf_check.py line=168 message="No ptr record found for MYIP" This is normal, since my internal dns dont have a rever ptr for my public IP.

Either way, i then updated the app for the beta version, but now, nothing is happening. on my dmarc_email_domains.csv I have 2 domains, and value for the script is true. I see the script is running: 2020-08-27 15:50:00,428 loglevel=DEBUG file=dmarc_spf_check.py line=70 message="Splunk SessionKey provided by stdin" 2020-08-27 15:55:00,242 loglevel=DEBUG file=dmarc_spf_check.py line=70 message="Splunk SessionKey provided by stdin" but it's not populating anything: $ cat spf_mailservers.csv ptr,ip,mail_server_group