[Feature Request] Add support for Forwarded emails

[degan6]

Is your feature request related to a problem? Please describe. I've setup a separate Imap account for this integration and am having Gmail forward messages too it. I don't want to enable IMAP on my gmail.

The problem is: Gmail adds the Fwd: prefix to the emails though and looking through the code it seems that will break the subject search.

Describe the solution you'd like I think two fixed would help.

1. To lower all text before string matching. This will help ensure we don't get caught by fwd vs FWD.
2. Search for and remove common forward prefix's on subjects. The list isn't that long gmail uses Fwd: and I've seen FW:

Another approach for two would be to do a regex match on the subject after it has be switched to lower case.

Additional context How the email looks in my IMAP account.

Love this integration and am excited to get it working!

[degan6]

I looked through the code of the project and now understnad that my idea won't work as the python IMAP library simply don't work in the way my original comment expects.

What if we switched it, so during setup you asked for the forward prefix then added that to all searches?

[firstof9]

You can setup multiple emails for the integration to check so you don't need to forward emails.

[cnorick]

@firstof9, at least for me, the problem is that I'd rather not give access to my main email account. I like receiving USPS updates to my main account so that I get notified of them, but then I forward those to a separate email account that HomeAssistant has access to.

I'm hesitant to give HomeAssistant access to my main account for security concerns.

[cnorick]

@degan6 The thought I had was to make the email subject line and sender strings that are defined in const.py configurable in each user's configuration.yaml.

[firstof9]

@firstof9, at least for me, the problem is that I'd rather not give access to my main email account. I like receiving USPS updates to my main account so that I get notified of them, but then I forward those to a separate email account that HomeAssistant has access to.

I'm hesitant to give HomeAssistant access to my main account for security concerns.

I think @moralmunky does something similar, except he has it reversed. Email to a Home Assistant allowed email account, and fwd to his main.

[cnorick]

@firstof9, at least for me, the problem is that I'd rather not give access to my main email account. I like receiving USPS updates to my main account so that I get notified of them, but then I forward those to a separate email account that HomeAssistant has access to. I'm hesitant to give HomeAssistant access to my main account for security concerns.

I think @moralmunky does something similar, except he has it reversed. Email to a Home Assistant allowed email account, and fwd to his main.

That's a really good idea!! Thanks!

[moralmunky]

The reason the proposed fix wouldn't work is that we use a combination of the sender email address and subject line to identify shipping emails. This two term system helps in cases where the subject line term could be used for more than one shipper.

I use a dedicated email account in my home's domain for all things home server related to receive and send emails.

I would agree with @firstof9, forward from your other account if you still need the emails to show in your personal account. The only one that neither of these methods work for is Amazon. I don't see a way to have notification sent to an email address different from the account.

[degan6]

@moralmunky , Couldn't a prefix be added to the search term like:

(rv, data) = account.search( None, '(FROM "' + email + '" SUBJECT "' + prefix + subject + '" SENTON "' + today + '")', )

This prefix could be configured in the config flow or YAML and if blank it would not change the search result.

[moralmunky]

I'm not saying a search term couldn't be added. What you state would be perfectly adequate but would fail to return results.

The emails being forwarded are not from the carriers notification email address used in the search. In addition to adding the prefix, you would also have to change all of the email addresses from the list of addresses in the How it works section that are used to find emails to the personal email address that you are forwarding from to get matches.

This also decreases the confidence levels because it is 100% confident that, regardless of the subject line, mcinfo@ups.com is UPS. You run the risk of different carriers using identical subject lines which would then require additional search criteria in the body of the message.

In order to make accepting forwarded messages

• Add logic to remove the prefix or add the prefix for the subject line
• Allow overriding of all FROM addresses for the carriers to a single email address
• Add in a body text search to look for the carriers email address or name in the body of the forwarded message
• Support all of this in the functional code and in the front end integration settings
• Answer the question if it should be an over all setting or allow it for individual carriers

[cnorick]

Alternatively, if you were to make each of the list of addresses in How it works (as well as the subject lines and search terms for those addresses) configurable by the user, you could defer the risk of identical subject lines to those users who chose to override the default configuration.

That way, by default, users get the confidence that each carrier is correctly parsed from their corresponding email address. However, more advanced users are able to extend the component to behave in more customizable ways.

Users who forward emails would need to override the defaults. But those users may also mitigate the risk of getting matching subject lines by changing the subject of the email before they forward it. They could, for example, prefix 'ups' to the subject line of any email that came from mcinfo@ups.com.

One could imagine the following config where a user configures the component to correctly parse forwarded emails from UPS:

mail_and_packages:
  - carrier: ups
    email: me@mydomain.com
    delivering_subject_lines:
      - "FWD: UPS Update: Package Scheduled for Delivery Today"
      - "FWD: UPS Update: Follow Your Delivery on a Live Map"
   delivered_subject_lines:
      - "FWD: Your UPS Package was delivered"

This config lets the user forward messages that come from UPS, but keep the default behavior for all the other carriers. And I think it's fair to assume that if the user is willing (and knows how) to set up the config this way, then they'd also be willing to take on the risk of having to update the carrier configs themselves if the carriers were to change their email structure.

I'm, admittedly, not very familiar with the code base, but I imagine not much of the core logic would change. It would just require pulling values from the configuration and accepting the default if nothing is specified in the config.

Also, I just want to add a thank you for putting in the time on this project! It's super useful to me and many others, I'm sure.

[firstof9]

Technically you could just as easily modify the const.py file to make those adjustments.

[cnorick]

You certainly could. But to do that, you'd have to fork the project, and it would take a little extra work to keep the fork up to date with the main branch. My plan was to do something like that, but it seemed like others might benefit from having exposed that functionality as a feature through the config, so I thought I'd reply here first.

[firstof9]

The goal is the keep it simple really, and the integration uses config flow, no yaml config, and the honestly simplest path would be to have the email going to a 'Home Assistant' mailbox and then forwarded to your main email from there.

[cnorick]

Keeping it simple makes sense to me. I'm not sure forwarding from the Home Assistant mailbox completely fixes the problem though, due to Amazon only sending to the account email.

I think keeping it simple is a valid rule this change would break, so I agree for that reason that this change likely doesn't belong in this project.

[niharmehta]

Setting the shipper email to a HA email account works great for dedicated shippers. I do this already. (I then POP it over to my personal) . Where this breaks is Amazon which is the unique case of an identity provider, retail storefront, media services, and shipper all in one. Since it is not feasible to change the Amazon email account info, the choice comes down to not using Amazon tracking or opening up a potential security issue with personal account access. As Walmart and other heavy weights start to replicate this model, I imagine this will also be an issue for them as well.

The original email headers are still contained MIME header. So if it is an issue of reliable parsing, the original sender info is usually still there. I can see it adding complexity to the parser (check header if FWD, then check MIME), as well as issues if other email systems that may not keep the headers intact.

[sibbl]

I just came across this great integration and would really love to use it!

However, while I trust Home Assistant and its components in many ways, I don't want to give it access to my main mail account. At the same time, like already mentioned, I don't want to or cannot let all my shipping mails be delivered to secondary mail account (especially Amazon).

So this feature is kind of a deal breaker to me and I'll rather look for other solutions even if it means to have some providers in between which support me redirecting my mails to them.

I also wonder why the issue was closed so quickly. I agree that it's not an easy task, but it's technically doable. But having this closed on the day it was opened feels like this feature is not wanted...?