Please add option to show api "abuseConfidenceScore" result in the postfix logs.

[ksuuk]

Hi,

I installed, fixed script permissions (644>755), but how can I determine from postfix logs that the script is running? Should I see "action=dunno" or something in the postfix logs for senders below the abuse score limit?

If there is no such message, add it and also add an option to always show for example the api result "abuseConfidenceScore" in the postfix logs.

[monsoft]

Hi,

As written in the README.md file, you will see in the postfix log that the host has a bad reputation:

Feb 5 01:10:18 mail postfix/smtpd[11151]: NOQUEUE: reject: RCPT from unknown[193.222.96.88]: 521 5.7.1 <unknown[193.222.96.88]>: Client host rejected: Bad host reputation.; from=<spameri@tiscali.it> to=<spameri@tiscali.it> proto=ESMTP helo=<WIN-4TTI4DH7SGH>

I can add an option to display AbuseIPDB score information in the response message that is logged in the postfix log if this help.

[ksuuk]

@monsoft

Thanks,

Yes, I read the manual, but this message only appears in the log when the recipient's IP address exceeds the specified score, but below that, nothing appears in the logs, so it's not clear if the script even works. And displaying the AbuseIPDB score information for each message would provide input as to whether the score needs to be changed, plus displaying this message indicates that the script is running.

[monsoft]

Hi,

It's not bad idea. I added this functionality and now in Postfix log you can see information with AbuseID score: Feb 8 06:39:49 mail abichecker[31362]: Email from host mta-1-75.mta.ometria.email[194.28.204.75] allowed. Abuse Score 50%.

[ksuuk]

@monsoft

Thanks, few notices:

Using: readonly SCRIPT_NAME="$(basename $0|tr -d '.sh')[$$]" results in "abicecker" in the logs, not "abichecker" :)

So I changed it to: SCRIPT_NAME="abichecker"

Then I noticed that all messages are whitelisted, for example: Feb 8 23:49:11 server abicecker[20162]: Host test.domain.ee whitelisted by domain.

cat hostname_domain_whitelist.txt
# Syntax:
#phx.paypal.com
#outbound.protection.outlook.com

Original code:

if [ ! -z ${client_name} ]; then
    if [ -f ${HOSTNAME_WHITELIST_DOMAINS} ]; then
        while IFS= read -r domain; do
            if [[ ${client_name} =~ ${domain} ]]; then
               echo "Host ${client_name} whitelisted by domain." | logger -p mail.info -t ${SCRIPT_NAME}
               email_allow
           fi
        done < ${HOSTNAME_WHITELIST_DOMAINS}
    fi
fi

ChatGPT found issues and fixed the code:

"The usage of [ ! -z ${client_name} ] without double quotes around ${client_name} could lead to unexpected behavior if the variable is empty or contains spaces. It's safer to use double quotes to prevent word splitting and pathname expansion. Similarly, it's better to quote the ${HOSTNAME_WHITELIST_DOMAINS} variable in [ -f ${HOSTNAME_WHITELIST_DOMAINS} ] to handle cases where the path contains spaces or special characters."

if [ ! -z "${client_name}" ]; then
    if [ -f "${HOSTNAME_WHITELIST_DOMAINS}" ]; then
        while IFS= read -r domain; do
            if [[ "${client_name}" =~ ${domain} ]]; then
                echo "Host ${client_name} whitelisted by domain." | logger -p mail.info -t "${SCRIPT_NAME}"
                email_allow
            fi
        done < "${HOSTNAME_WHITELIST_DOMAINS}"
    fi
fi

Now logs are much better: Feb 9 00:16:05 server abichecker: Email from host test.domain.ee[xxx.xxx.xxx.xxx] allowed. Abuse Score 70%.

[monsoft]

Hi,

Ahh, I seen this issue with Script_Name and I fixed in my local script and I forgot to fix it in abichecker source code with I commited to GitHub. You can use sed instead of tr “sed ‘s/.sh//m’”. This is how I fix it locally. I need to have a look to this whitelisting issue as I’m not experiencing it in my local running version.

Thanks for testing

On 8 Feb 2024, at 22:30, ksuuk @.***> wrote:

 @monsoft

Thanks, few notices:

Using: readonly SCRIPT_NAME="$(basename $0|tr -d '.sh')[$$]" results in "abicecker" in the logs, not "abichecker" :)

So I changed it to: SCRIPT_NAME="abichecker"

Then I noticed that all messages are whitelisted, for example: Feb 8 23:49:11 server abicecker[20162]: Host test.domain.ee whitelisted by domain.

cat hostname_domain_whitelist.txt

Syntax:

phx.paypal.com

outbound.protection.outlook.com

Original code: if [ ! -z ${client_name} ]; then if [ -f ${HOSTNAME_WHITELIST_DOMAINS} ]; then while IFS= read -r domain; do if [[ ${client_name} =~ ${domain} ]]; then echo "Host ${client_name} whitelisted by domain." | logger -p mail.info -t ${SCRIPT_NAME} email_allow fi done < ${HOSTNAME_WHITELIST_DOMAINS} fi fi

ChatGPT found issues and fixed the code:

"The usage of [ ! -z ${client_name} ] without double quotes around ${client_name} could lead to unexpected behavior if the variable is empty or contains spaces. It's safer to use double quotes to prevent word splitting and pathname expansion. Similarly, it's better to quote the ${HOSTNAME_WHITELIST_DOMAINS} variable in [ -f ${HOSTNAME_WHITELIST_DOMAINS} ] to handle cases where the path contains spaces or special characters."

if [ ! -z ${client_name} ]; then if [ -f ${HOSTNAME_WHITELIST_DOMAINS} ]; then while IFS= read -r domain; do if [[ ${client_name} =~ ${domain} ]]; then echo "Host ${client_name} whitelisted by domain." | logger -p mail.info -t ${SCRIPT_NAME} email_allow fi done < ${HOSTNAME_WHITELIST_DOMAINS} fi fi Now logs are much better: Feb 9 00:16:05 server abichecker: Email from host test.domain.ee[xxx.xxx.xxx.xxx] allowed. Abuse Score 70%.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[ksuuk]

@monsoft

BTW, the Abuse Score % in logs is false. Your example:

Feb 8 06:39:49 mail abichecker[31362]: Email from host mta-1-75.mta.ometria.email[194.28.204.75] allowed. Abuse Score 50%.

API call:

curl -s -G https://api.abuseipdb.com/api/v2/check --data-urlencode "ipAddress=194.28.204.75" -H "Key:XXX"
{"data":{"ipAddress":"194.28.204.75","isPublic":true,"ipVersion":4,"isWhitelisted":null,"abuseConfidenceScore":0,"countryCode":"GB","usageType":"Commercial","isp":"Ometria Ltd","domain":"ometria.email","hostnames":["mta-1-75.mta.ometria.email"],"isTor":false,"totalReports":0,"numDistinctUsers":0,"lastReportedAt":null}}

I also tested my example (Abuse Score 70%), and API call gives also: "abuseConfidenceScore":0,.

[monsoft]

Hi,

Yes, you're right. I used wrong variable in string which is logging to Postfix log. This has been corrected. I hope that processing whitelisted domain has been corrected as well with last source code push.

[ksuuk]

@monsoft

Thanks, everything seems to be working fine so far.

BTW, the manual says:

“This script use data and API provided by <a href="https://www.abuseipdb.com" target="_blank">AbuseIPDB</a> project.
API is free up to 3000 checks per day so if your SMTP server is quite busy, you will have to look for paid access.”

But AbuseIPDB https://www.abuseipdb.com/faq.html say’s:

"Are there any restrictions or limitations?

Due to limited resources, free accounts currently have 1,000 requests/day for both IP check and report actions through our free API. Verified webmasters are allowed 3,000 requests/day. Supports are allowed 5,000 requests/day. The web version of our service does not have any usage limitations.

APIv1: The rate limiter is a 24-hour sliding window. In simple terms, if you hit your limit of 1,000 requests, and your first request was made 23 hours ago, then in one hour you gain that one request back. So on and so forth for each nth request. All requests that exceeded the limit will be served will a 429 Too Many Requests response.

APIv2: The rate limiter resets at 00:00 UTC.”

So the manual gives the impression that anyone with an API key is allowed up to 3000 checks per day, but in reality only up to 1000 checks per day, as up to 3000 checks per day requires a webmaster account.

And since the limit is based on the API key, the solution would be to register multiple API keys and add a function to select a random API key in the script. :)

[monsoft]

Hi,

It's possible that "They" changed it slightly now (I will update Readme). In my opinion, if someone wanted to use AbuseIPDB with "something bigger", they should simply buy the appropriate license.

[monsoft]

I looked to my AbuseIPDB account, and my daily limit is 3k calls

Endpoint    Usage / Daily Limit Utilization Rate
check        18 / 3,000           0.6%

[ksuuk]

@monsoft

I didn't check the normal account limit, I just switched to webmaster and now I see the 3K daily limit as well.

[ksuuk]

@monsoft

Just in case anyone has a similar problem - I have one server, without jq and with an older shell version that doesn’t support the =~ operator. So I used grep and awk instead of jq and older shell compatible commands:

# Check if curl & jq are installed
check_commands curl
#check_commands jq
check_commands logger

#if [[ "${client_name}" =~ "${domain}" ]]; then
if echo "$client_name" | grep -q "$domain"; then

#if [[ ! "${REPORT_JSON}" =~ "ipAddress" ]]; then
if ! echo "$REPORT_JSON" | grep -q "ipAddress"; then

#ABUSE_CONFIDENCE_SCORE=$(echo "${REPORT_JSON}" | jq -r .data.abuseConfidenceScore)
ABUSE_CONFIDENCE_SCORE=$(echo "${REPORT_JSON}" | grep -o '"abuseConfidenceScore":[0-9]*' | awk -F: '{print $2}')

[monsoft]

You're right. There are still some older *nix systems which doesn't have (and never will be) jq. I will create additional version of abichecker for these systems.

Many Thanks