Uncaught Error: Undefined constant "Liuch\DmarcSrg\Sources\SORTDATE"

[dwreski]

New user here. I've installed and configured dmarc-srg and trying to populate the database from an mbox with a history of DMARC emails and attachments. I've initialized the database, but when I run fetch_reports.php, there is an error about SORTDATE not being defined.

$ php utils/fetch_reports.php source=email
PHP Fatal error:  Uncaught Error: Undefined constant "Liuch\DmarcSrg\Sources\SORTDATE" in /var/www/orion.mydomain.com-443/html/dmarcreports/dmarc-srg/classes/Sources/MailboxSource.php:91
Stack trace:
#0 /var/www/orion.mydomain.com-443/html/dmarcreports/dmarc-srg/classes/Report/ReportFetcher.php(68): Liuch\DmarcSrg\Sources\MailboxSource->rewind()
#1 /var/www/orion.mydomain.com-443/html/dmarcreports/dmarc-srg/utils/fetch_reports.php(139): Liuch\DmarcSrg\Report\ReportFetcher->fetch()
#2 {main}
  thrown in /var/www/orion.mydomain.com-443/html/dmarcreports/dmarc-srg/classes/Sources/MailboxSource.php on line 91

What am I missing?

[liuch]

Hello! Do you have the php-imap module installed?

[dwreski]

Yes, php-pear-Net-IMAP on fedora36

php-pear-Net-IMAP.noarch : Provides an implementation of the IMAP protocol

[liuch]

I'm afraid php-pear-Net-IMAP provides a different API.

[liuch]

Maybe later I will implement support for php-net-imap, but not anytime soon. I'm sorry.

[dwreski]

I actually thought php-imap support was built into PHP itself? If not, which libraries do you support that I can install?

[liuch]

php-imap is one of PHP extensions: https://www.php.net/manual/en/book.imap.php but it usually put it in a separate package. Fedora do have this one with the same name, but as far as I am aware Fedora 34 dropped the php-imap package. Is this true for your version? I don't know. As for supporting imap libraries, currently I support only php-imap. To be honest, I did not expect that there could be any problems with the standard library. BTW, I use Debian.

[dwreski]

I located a php-imap package that works with fedora36. I made it a bit further, but not sure if this problem is still related. Which table in the database contains the actual data/files so I can investigate? During my experimentation, I did see it created a "failed" directory, but I deleted it before running this test.

The mbox with the DMARC files and attachments has about 1000 emails in it from the past month or two.

The dashboard showed no data after this run. It did produce a red pop-up in the upper-right corner that displayed "[-100] Unexpected token '<', "
<\b>"... is not valid JSON"

[alerts@orion dmarc-srg]$ php utils/database_admin.php drop
Message: Database tables have been dropped
[alerts@orion dmarc-srg]$ php utils/database_admin.php init
Message: The database has been initiated
[alerts@orion dmarc-srg]$ php utils/fetch_reports.php source=email
PHP Deprecated:  Return type of Liuch\DmarcSrg\DateTime::jsonSerialize() should either be compatible with JsonSerializable::jsonSerialize(): mixed, or the #[\ReturnTypeWillChange] attribute should 
be used to temporarily suppress the notice in /var/www/orion.guardiandigital.com-443/html/dmarcreports/dmarc-srg/classes/DateTime.php on line 39

Deprecated: Return type of Liuch\DmarcSrg\DateTime::jsonSerialize() should either be compatible with JsonSerializable::jsonSerialize(): mixed, or the #[\ReturnTypeWillChange] attribute should be us
ed to temporarily suppress the notice in /var/www/orion.guardiandigital.com-443/html/dmarcreports/dmarc-srg/classes/DateTime.php on line 39
Failed to get incoming report:
  Error message:
    - Failed to create a new mailbox
    - Incorrect message: Attachment count is not valid (0)
  Email message metadata:
    - From: abuse@126.com
    - Date: Mon, 11 Jun 2018 04:43:33 +0800

Failed to get incoming report:
  Error message:
    - Failed to create a new mailbox
    - Failed to add an incoming report: unknown domain
  Report ID: 1528678877.171759
  Email message metadata:
    - From: noreply@dmarc.yahoo.com
    - Date: Mon, 11 Jun 2018 01:19:53 -0700

There were another three or four similar sections of output from "failed to get incoming report". I do have debug=1, but it would be helpful if more debug output was provided.

[liuch]

Which table in the database contains the actual data/files so I can investigate?

reports and rptrecords.

Deprecated: Return type of Liuch\DmarcSrg\DateTime::jsonSerialize()...

It is on PHP8? I will fix it soon.

I did see it created a "failed" directory, but I deleted it before running this test.

The script creates the directory when it finds an email message with an incorrect report to move it there.

Failed to add an incoming report: unknown domain

There is no record in the database for the domain you have a report for. The first domain is added automatically, the second and subsequent ones are not. But you can change this in the settings.

Incorrect message: Attachment count is not valid (0)

The script found an email message that does not have any attachments. It cannot be a dmarc report. Perhaps you do not understand how it works. The script assumes that ALL messages in your mailbox are dmarc rua reports. All unread messages in the mailbox that are not reports are moved to the failed directory. You should either use a separate address for dmarc rua reports or set up an existing one so that reports are put into a separate folder in your INBOX.

I do have debug=1, but it would be helpful if more debug output was provided.

Yes, this is a problem. I'm already writing code that will allow you to get more information in debug mode.

I would recommend that you first try loading these reports from local files in the (by web UI) or the server directory (by web UI or console) to make it easier to understand the cause of the problem.

[liuch]

It did produce a red pop-up in the upper-right corner that displayed "[-100] Unexpected token '<', "<\b>"... is not valid JSON"

That is because of the "Deprecated:..." message. I have just fixed it.

[dwreski]

This appears to have (mostly) fixed the problem. It now successfully processes at least one message. However, my mbox contains DMARC reports that don't have attachments. It also contains a few spam messages that inadvertently got through. Is there any way to bypass processing these messages, instead of just failing altogether?

There is no record in the database for the domain you have a report for. The first domain is added automatically, the second and subsequent ones are not. But you can change this in the settings.

Do you mean the source domain? My domain? Or the domain sending the reports? Is this controlled through the allowed_domains parameter? Currently, I just have our domain listed there.

Thanks very much for your support.

[dwreski]

Also, why when I specified the name of the mbox with the DMARC reports in it, does it appear to also process all folders?

[liuch]

However, my mbox contains DMARC reports that don't have attachments.

How can it be? DMARC rua reporting emails must have one and only one attachment.

Is there any way to bypass processing these messages, instead of just failing altogether?

No, there is not. How will the script determine which messages in the mailbox it has already read and which have not? Currently after handling an email message the script either marks the message as read or moves it to the failed folder.

Do you mean the source domain? My domain? Or the domain sending the reports? Is this controlled through the allowed_domains parameter?

I mean your domain. You can add all your domain manually or set up the allowed_domains parameter so that domains are added automatically.

I just have our domain listed there.

The script did this itself for the first successfully processed report.

Also, why when I specified the name of the mbox with the DMARC reports in it, does it appear to also process all folders?

Are you sure? It does not process nested folders.

[dwreski]

However, my mbox contains DMARC reports that don't have attachments.

How can it be? DMARC rua reporting emails must have one and only one attachment.

I've attached one such email here. It may not be valid, but that didn't stop the sender from sending it :-) And our dmarc-admin address is public, of course, so anyone can email it. dmarc-report-fail.txt

These emails are processed by procmail, so perhaps I can write a rule to do what I need, but I'd prefer not to do that.

Is there any way to bypass processing these messages, instead of just failing altogether?

No, there is not. How will the script determine which messages in the mailbox it has already read and which have not? Currently after handling an email message the script either marks the message as read or moves it to the failed folder.

Perhaps if it doesn't contain a properly formatted attachment? Or even that the subject doesn't begin with "Report Domain:" ? Or perhaps it discards it if there's something in the body?

Also, why when I specified the name of the mbox with the DMARC reports in it, does it appear to also process all folders?

Are you sure? It does not process nested folders.

I am not 100% sure, but I do know that all mboxes were moved into the "failed" directory at one point, and I didn't put them there. It's possible the script was run prior to properly configuring the location and mailbox parameters.

[liuch]

It may not be valid, but that didn't stop the sender from sending it :-) And our dmarc-admin address is public, of course, so anyone can email it.

This report doesn't look like a valid RUA report. Who is stopping you from ignoring such "reports"? BTW, I saw the DMARC policy record on your domain, you indicated readiness to accept RUF reports to the same mail box. Perhaps some of your reports are RUF reports. I remind you: I have not implemented the processing of RUF reports.

Perhaps if it doesn't contain a properly formatted attachment?

But the next time the script goes to check the mail box, it will still have to look into this letter to find out if the message has an attachment. What if there are hundreds of letters? What if thousands?

Or even that the subject doesn't begin with "Report Domain:"

Recently, I was surprised to learn that some mail servers may indicate the incorrect extension of the report file. And you suggest checking the subject of the message. I don't think that.

[dwreski]

This report doesn't look like a valid RUA report. Who is stopping you from ignoring such "reports"? BTW, I saw the DMARC policy record on your domain, you indicated readiness to accept RUF reports to the same mail box. Perhaps some of your reports are RUF reports. I remind you: I have not implemented the processing of RUF reports.

The mailbox isn't monitored. We're hoping to automate this entire process. We can't control who emails us or what gets emailed to us. I was assuming some type of error checking of your input data.

I've changed the RUF address to be something different.

Perhaps if it doesn't contain a properly formatted attachment?

But the next time the script goes to check the mail box, it will still have to look into this letter to find out if the message has an attachment. What if there are hundreds of letters? What if thousands?

I was assuming that once the email in the mbox was processed, it would be deleted.

Recently, I was surprised to learn that some mail servers may indicate the incorrect extension of the report file. And you suggest checking the subject of the message. I don't think that.

I'm only suggesting you add some error checking of your input data.

The use case for me is to monitor the address that receives DMARC reports for our domains and generate some simple reports for us to more easily monitor DMARC issues

Now I'm 100% certain that it moved all of the mboxes in ~/mail into ~/mail/failed on its own, after running utils/fetch_reports.php

$mailboxes = [
    // Just for displaying in web-admin. Not necessary.
    'name'            => 'Dmarc-Rua',
    // Host of the email server. You can specify a port separated by a colon.
    'host'            => 'orion.example.com:993',
     // Connection encryption method. The valid values are:
     // 'none'     - without encryption (strongly not recommend).
     // 'ssl'      - SSL/TLS on a separate port, for IMAP it is usually port 993.
     // 'starttls' - STARTTLS method, usually on the standard IMAP port 143. Default value.
    'encryption'      => 'ssl',
    // Set true if you want to connect to the IMAP server without certificate validation
    'novalidate-cert' => false,
    // Mailbox user name.
    'username'        => 'dmarcuser',
    // Mailbox password.
    'password'        => 'password',
    // Mailbox name
    'mailbox'         => 'dmarcreports'
];

$directories = [
    // Just for displaying in web-admin. Not necessary.
    'name'     => 'Rua-Dir',
    // The directory location
    'location' => '/home/dmarcuser/mail'
];

$cleaner = [
    // It is used in utils/mailbox_cleaner.php
    'mailboxes' => [
        // Will remove messages older than (days)
        'days_old'       => 30,
        // How many messages will be removed at once maximum.
        'delete_maximum' => 50,
        // How many messages must be leave in the mailbox minimum.
        'leave_minimum'  => 100,
        // Cleaning up the mailbox that failed reports are moved to. This box is also subject
        // to the restrictions mentioned above. The valid values are:
        // 'none' - no action with it. Default value.
        // 'seen' - only seen messages will be removed
        // 'any'  - all messages will be removed
        'failed'         => 'none'
    ],
    // It is used in utils/reports_cleaner.php
    'reports' => [
        'days_old'       => 30,
        'delete_maximum' => 50,
        'leave_minimum'  => 100
    ],
    // It is used in utils/reportlog_cleaner.php
    'reportlog' => [
        'days_old'       => 30,
        'delete_maximum' => 50,
        'leave_minimum'  => 100
    ]
];

[liuch]

At first, let me explain to you the script's logic:

• The script assumes that the mailbox is ONLY for dmarc rua reports. In other words, there should not be other imal messages there.
• The script looks through ALL unread messages to extract reports from them.
• If the script succeeded to check the message, extract the report and save it to the database, the message is marked as read, otherwise the script moves the message to the failed folder. That is, all messages that could not be processed will get to this folder. The script will not mark them as read.

What the script does not do:

• The script does not look into nested folders.
• The script does not delete any emails.

Why is this done this way:

• For performance reasons. Part of the work of obtaining a list of messages that need to be processed has been shifted to a imap server. Thus, no matter how many letters there are in the mailbox, the script will not do unnecessary work.
• I may want to look onto the original report message for some correct incoming report.
• I may want to further configure my spam filter or see why some of the reports are not accepted, and for this I have the failed folder where the messages I have not seen are marked as unread. The script does not enter here, this place is just for me.

To summarize: After the script runs, all unread messages are either marked as read or moved to the failed folder. If you want to delete old mails in those mailboxes, use the mailbox_cleaner.php script. And this script also assumes that the mail box is only for dmarc rua reports.

[liuch]

I was assuming that once the email in the mbox was processed, it would be deleted.

No, this script (fetch_reports.php) does not delete any email messages. It either marks them or moves them.

I'm only suggesting you add some error checking of your input data.

What types of error do you mean?

Now I'm 100% certain that it moved all of the mboxes in ~/mail into ~/mail/failed on its own, after running utils/fetch_reports.php

Right, If those messages were unread and did not contain correct dmarc rua reports.

[dwreski]

What the script does not do:

• The script does not look into nested folders.
• The script does not delete any emails.

And what I am saying is that your script went beyond just the folders that's defined by "mailbox" and into the other folders under my IMAP account. Is that the intended behavior?

It did not just move the messages from the dmarcreports mbox that were not related to DMARC, but all the folders that were available under the account.

To summarize: After the script runs, all unread messages are either marked as read or moved to the failed folder. If you want to delete old mails in those mailboxes, use the mailbox_cleaner.php script. And this script also assumes that the mail box is only for dmarc rua reports.

I'm happy if your script moves the messages that are not related to DMARC from my dmarcreports folder into a "failed" folder, but not the other unrelated folders also in my account.

[liuch]

That is strange. I will definitely check it out. It seems I misunderstood you from the very beginning. I'm sorry for that.

[liuch]

Your dmarcreports folder is located not in INBOX. Did I understand you correctly?

[dwreski]

Your dmarcreports folder is located not in INBOX. Did I understand you correctly?

Yes, that's correct - see my previous comment that included the relevant conf.php sections. Thanks so much for your help.

[liuch]

So I have to set up my mail box in the same way to reproduce the issue.

[dwreski]

So I have to set up my mail box in the same way to reproduce the issue.

Message me offline if you want me to give you access to our system in some way.

[liuch]

Hello! Unfortunately, I still can't reproduce it using my mailbox. I've just added some code that allows you to get more detailed error messages in the debug mode. Perhaps the more detailed errors will help us to figure out what the problem is.

p.s. Could you message me via my email address I have in my profile? I have a few additional questions about your configuration.

[sts]

@liuch Same issue happened for me, since the imap extension wasn't loaded. Maybe adding a better error message would help in this case.

[liuch]

@sts That's a fair point. Fixed with https://github.com/liuch/dmarc-srg/commit/075cda7333b4657577ac50af7451c8580512c4b5