Cannot get reports automatically from mailbox

[elimae17]

I have configured an IMAP connection to our mailbox. According to the accessibility check everything should be fine. It recognizes how many emails are in the mailbox.

However, the reports from these emails are not getting added automatically to the tool. Over the last two days I got two new emails, and they still remain unseen. Am I missing something out? Do I have to configure something so that the scripts run regularly on their own?

I hope someone can help me!

[williamdes]

Hi @elimae17

Do I have to configure something so that the scripts run regularly on their own?

I would say yes, I use a cron with my docker version:

@hourly docker exec dmarc-srg.xxx.xxx.xxx php -f utils/fetch_reports.php

Let's let @liuch comment about this

[liuch]

Hi, There are several ways to process incoming DMARC reports by this tool. Perhaps the most convenient for you is to run the fetch_reports.php script periodically in a cron job, like this:

0 */6 * * * www-data cd /usr/local/share/dmarc-srg/www && php utils/fetch_reports.php source=email

See details in the script comments.

[elimae17]

Thank you both for your fast replies and explanations!

This would be the first time I use cron jobs, so excuse my amateur questions but I'm just getting used to the syntax.

0 */6 * * * would mean, that the script runs every 6 hours, is that right?

We are on a shared web host. So If I want to run this code once a day, would this code work?:

0 0 * * * /usr/bin/php /home/our-username/dmarc-srg-folder/utils/fetch_reports.php source=email

I am not familiar with commands but I would need the /usr/bin/php to run php scripts, right?

I'd appreciate your help!

[liuch]

0 */6 * * * would mean, that the script runs every 6 hours, is that right?

Yes, it is.

We are on a shared web host. So If I want to run this code once a day, would this code work?:

Yes. This code will be run once a day at 12 am. But I would have chosen a different time to run. Find out when you get most of these reports and add half or an hour. BTW: you may need to use cd command like in my example.

I am not familiar with commands but I would need the /usr/bin/php to run php scripts, right?

Yes, you are right. In this particular case, you need to run the command like this.

I hope this helps you.

[elimae17]

Thanks for the help!

I set up the following cron job for testing:

0 0,12 * * * /usr/bin/php /home/username/dmarc-check.mydomain.com/utils/fetch_reports.php source=email

It gets executed however I get the following email notification from my cron daemon:

Status: 500 Internal Server Error
X-Powered-By: PHP/8.0.28
Content-type: text/html; charset=UTF-8

The error logs contain the following:

[06-Jul-2023 12:00:05 UTC] PHP Warning:  require(init.php): Failed to open stream: No such file or directory in /home/username/dmarc-check.mydomain.com/utils/fetch_reports.php on line 55
[06-Jul-2023 12:00:05 UTC] PHP Fatal error:  Uncaught Error: Failed opening required 'init.php' (include_path='.:/opt/alt/php80/usr/share/pear:/opt/alt/php80/usr/share/php:/usr/share/pear:/usr/share/php') in /home/username/dmarc-check.mydomain.com/utils/fetch_reports.php:55
Stack trace:
#0 {main}
  thrown in /home/username/dmarc-check.mydomain.com/utils/fetch_reports.php on line 55

I hope you can help me further.

[williamdes]

Out of scope here, but I do not need to specify source=email in my cron and it works great. That's a bit weird reading the code, it seems to require it.

[liuch]

This is because "the current directory must be the one containing the classes directory". So you have to use cd command as I wrote in my example above. I also do not see the user name under which the script is supposed to run. In my case it is www-data.

[liuch]

@williamdes It's ok if you don't have other report sources set up.

[liuch]

@williamdes, source=directory is intended for cases where you cannot (or do not want to) provide access to the mailbox with reports, so you specify the directory on your server where your script or you put the files with reports to process them. I prefer to explicitly specify restrictions to avoid problems in the future, perhaps this has somehow manifested itself in my comments and code.

[elimae17]

I tried cd /home/username/dmarc-check.mydomain.com && php utils/fetch_reports.php source=email but unfortunately still got the 500 Internal Server error and the same Failed opening required 'init.php'

when I tried www-data cd /home/username/dmarc-check.mydomain.com && php utils/fetch_reports.php source=email i got this email:

/bin/sh: www-data: command not found

Regarding the username, as far as I understand it, since we are on a shared server, there is only one user and /home/ourusername/ is the only directory we have access to, all the files, including dmarc-srg, are below that.

[liuch]

but unfortunately still got the 500 Internal Server error and the same Failed opening required 'init.php'

Have you checked if the init.php file is available here and accessible? If you have access to the console, try these two commands one after anoter:

• cd /home/username/dmarc-check.mydomain.com
• php utils/fetch_reports.php source=email

Regarding the username, as far as I understand it, since we are on a shared server, there is only one user and /home/ourusername/ is the only directory we have access to, all the files, including dmarc-srg, are below that.

I forgot that you are using shared hosting. I'm sorry.

[elimae17]

Thank you for your continued support.

Unfortunately we don't have access to the console.

The init.php file should be where it belongs:

Is the comand maybe looking for it in the utils folder as it says _No such file or directory in /home/username/dmarc-check.mydomain.com/utils/fetchreports.php on line 55? (Sorry, I wouldn't know)

[liuch]

This happens because you don't specify the current directory or you specify the wrong current directory. Note: the current directory must be the one where the init.php is located, and it is not not utils/. Maybe your command with cd doesn't work?

[elimae17]

The init.php ist located in the directory called "dmarc-check.mydomain.com". I thought with the command cd /home/username/dmarc-check.mydomain.com I specified this directory and then with && php utils/fetch_reports.php source=email I execute the script?

I tried without the cd, but this

/usr/bin/php /home/username/dmarc-check.mydomain.com/utils/fetch_reports.php source=email

brought the same result.

Could our shared hosting with its limitations be the problem?

[liuch]

You did everything right. I have no idea why it doesn't work. What I do wonder is how you get the 500 error. It's an http server error. Most likely the problem is that the job is running on a webserver not in cli mode, and the cd command is ignored (or is performed in a separate process). I recommend that you ask who rent you the server how you should run php files with cron. I have not used shared hosting for a long time, sorry.

[elimae17]

Okay, I'll ask my hosting provider if they can help with it.

Thanks again for you help and patience!

[schleup]

maybe this could help (I know crontab can be quite annoying, especially for Newbies on a Un*x system). This is what I'm running in /etc/cron.d/dmarc on my Debian system:

# m h dom mon dow user command /5 www-data cd /var/www/dmarc-srg && php utils/fetch_reports.php > /dev/null 0 0 www-data cd /var/www/dmarc-srg && php utils/mailbox_cleaner.php > /dev/null 0 0 www-data cd /var/www/dmarc-srg && php utils/reports_cleaner.php > /dev/null 0 0 www-data cd /var/www/dmarc-srg && php utils/reportlog_cleaner.php > /dev/null

This starts all jobs as user www-data. Reports are fetched once in 5 minutes, all cleanup jobs run once per day at 00:00. I'm not using the email notifications.

Depending on your configuration it might not be necessary to mention the user running the commands (here: www-data)

[liuch]

@schleup, Only privileged users are allowed to run jobs under another user and it needs to have write access to /etc/. But in this case we are talking about shared hosting.

[elimae17]

My hosting provider told me there was indeed a technical problem on their side (I don't know which) and they fixed it. The cron job now works!

Thanks again for all your help!

[liuch]

I'm glad to hear that!

[elimae17]

For anyone who may have the same problem on a shared server, I had to change the cron job command as follows: cd /home/username/dmarc-srg-folder && /usr/local/bin/php utils/fetch_reports.php