No text assigned (empty) from email body received with IMAP integration

Soleil01 commented 1 year ago

The problem

I was using the « IMAP email content integration » that was working fine with my application, but it is to be deprecated. In my case, I need to extract data from a big email (the text is UTF-8 HTML quoted printable of more than 81000 bytes) from my GMAIL account. I maybe able to extract a part of the body of the email to retain the part with the data I need (for example from 0 to 20000 bytes). In the documentation of the new version of the integration (IMAP) you say that we can obtain up to 30000 bytes of data, and so I did change the « max message size » to 29999 in the configuration. The email is effectively extracted, but the text part (trigger.event.data['text'] in the imap_content) is always blank (‘’). To access the data, I used the event trigger of type « imap_content » from your example in the documentation, as shown in the Example YAML snippet of this issue.

What version of Home Assistant Core has the issue?

2023.6.1

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Supervised

Integration causing the issue

IMAP

Link to integration documentation on our website

https://www.home-assistant.io/integrations/imap

Diagnostics information

No response

Example YAML snippet

- trigger:
      - platform: event
        event_type: "imap_content"
        id: "hydro_imap_event"
    sensor:
      - name: hydro_imap_content
        unique_id: "Hydro_imap_content"
        state: "{{ trigger.event.data['custom'] }}"
        attributes:
          Message: "{{ trigger.event.data['text'][0:20000] }}"
          Server: "{{ trigger.event.data['server'] }}"
          Username: "{{ trigger.event.data['username'] }}"
          Search: "{{ trigger.event.data['search'] }}"
          Folder: "{{ trigger.event.data['folder'] }}"
          Sender: "{{ trigger.event.data['sender'] }}"
          Date: "{{ trigger.event.data['date'] }}"
          Subject: "{{ trigger.event.data['subject'] }}"
          To: "{{ trigger.event.data['headers'].get('Delivered-To', ['n/a'])[0] }}"
          Return-Path: "{{ trigger.event.data['headers'].get('Return-Path',['n/a'])[0] }}"
          Received-first: "{{ trigger.event.data['headers'].get('Received',['n/a'])[0] }}"
          Received-last: "{{ trigger.event.data['headers'].get('Received',['n/a'])[-1] }}"

Anything in the logs that might be useful for us?

No response

Additional information

I have other issues with this integration mostly with special characters (french characters like « é », or simply a quote «‘») that are causing a validation error in the search parameters in the configuration dialog of the integration. The option parameter would need to be : « subject "Avis d’événement de pointe" ». Also the « Template to create custom event data » containing the value : {{ 'Avis d’événement de pointe' in subject }} is not working either.

home-assistant[bot] commented 1 year ago

Hey there @jbouwh, mind taking a look at this issue as it has been labeled with an integration (imap) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `imap` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign imap` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

imap documentation imap source _{^{(message by IssueLinks)}}

jbouwh commented 1 year ago

While that is not optimal to parse such big emails, the custom template feature might be of help. This template parses the email before the event is sent.

jbouwh commented 1 year ago

About the special characters, may be you can supply an email sample so I can investigate the encoding issue.

Soleil01 commented 1 year ago

Issue #94697

As you requested, here is an example of the email that is transmitted and treated with IMAP.

---------- Message transféré --------- De : Hydro-Québec @.> Date : sam. 25 févr. 2023 à 12:51 Objet : ❄️ Avis d’événement de pointe | Dimanche 26 février 2023 À : Daniel Galarneau @.>

Préparez-vous à réduire votre consommation d’énergie et à économiser. Si vous avez de la difficulté à voir ce message, consultez sa version Web http://ofsys.hydroquebec.com/T/OFSYS/SM3/375/2/S/F/8509/8347111/UV8uAIcY.html | English version http://ofsys.hydroquebec.com/T/OFSYS/SM2/375/2/S/F/8509/8347111/UV8uAIcY/747254.html [image: Hydro-Québec] https://ofsys.hydroquebec.com/T/OFC4/L2S/8509/B2743727/OPoo/747255/28096385/h90ZwN/1/8347111/UV8uAIcY/I/760288/RuqX7F.html?h=W-1hjH0fi3pF9OHWMtS1R4DSdgBPsoxoU1cV6gR-ej4 [image: Hydro-Québec] https://ofsys.hydroquebec.com/T/OFC4/L2S/8509/B2743727/OPoo/747255/28096385/h90ZwN/2/8347111/UV8uAIcY/I/760288/RuqX7F.html?h=JS-ULkSPO1bSmkkXD00dXgw4W8lgalA6thQH1o9Xw_E Avis | Tarification dynamique

25 février 2023

Bonjour.

Merci d’avoir choisi le tarif D avec l’option de crédit hivernal. Ce choix judicieux vous permet de contribuer à l’effort collectif lors des périodes de pointe.

Nous prévoyons que la demande d’électricité sera très élevée demain, ce qui sollicitera fortement l’ensemble du réseau électrique. Un ou des événements de pointe auront donc lieu :

26 février, de 6 h à 9 h, en matinée.

Nous vous invitons à prévoir la réduction de votre consommation d’énergie pour le ou les lieux suivants :

92 rue Marie-Chauvin https://www.google.com/maps/search/92+rue+Marie-Chauvin+%0A%09Boucherville+QC++J4B+4G5?entry=gmail&source=g

Boucherville QC J4B 4G5 https://www.google.com/maps/search/92+rue+Marie-Chauvin+%0A%09Boucherville+QC++J4B+4G5?entry=gmail&source=g

Meilleures salutations,

Les Services à la clientèle

https://ofsys.hydroquebec.com/T/OFC4/L2S/8509/B2743727/OPoo/747255/28096980/CAbO6P/1/8347111/UV8uAIcY/I/760288/RuqX7F.html?h=6EVk1hfEtych-vfwz6rirBxPMfI98SzZ8TVTizWv_Yw Un outil essentiel pour économiser au max !

Dès le lendemain d’un événement de pointe, accédez à l’Espace client et consultez votre tableau de suivi des crédits, un outil sur mesure qui vous permet de voir vos économies. Consulter le tableau https://ofsys.hydroquebec.com/T/OFC4/L2S/8509/B2743727/OPoo/747255/28096980/CAbO6P/2/8347111/UV8uAIcY/I/760288/RuqX7F.html?h=BI8idV4RDWQP7mlKmodVKH9Z0EK7uWxmVcxSFenwY7A

C’est l’heure du quiz !

Abaisser la température de quelques degrés ou décaler l’heure de la douche pourraient vous rapporter gros !

Testez vos connaissances et découvrez nos conseils pour minimiser votre consommation d’énergie et maximiser vos économies. Faire le quiz https://ofsys.hydroquebec.com/T/OFC4/L2S/8509/B2743727/OPoo/747255/28096399/qqKZMN/1/8347111/UV8uAIcY/I/760288/RuqX7F.html?h=6D9PUzvRYdrFW9oF5P6wvKpnRSniv3lCao30qZwd5NE

Veuillez ne pas répondre au présent courriel. Puisqu’il a été généré automatiquement, nous ne pouvons donner suite à vos questions ou commentaires. Si vous ne souhaitez plus recevoir d’avis d’événement de pointe, vous devez vous désinscrire de l’option de crédit hivernal https://ofsys.hydroquebec.com/T/OFC4/L2S/8509/B2743727/OPoo/747255/28096402/MmIQHf/1/8347111/UV8uAIcY/I/760288/RuqX7F.html?h=wbClUIO73xWp-zF6CvceT7Mt5qPbRfFhpZk5YDQZ2mo. Pour joindre les Services à la clientèle, veuillez composer le 1 877 234-6548, puis le code d’accès 8333.

Hydro-Québec ne vous demandera jamais des renseignements personnels par courriel. Consultez notre politique en matière de confidentialité https://ofsys.hydroquebec.com/T/OFC4/L2S/8509/B2743727/OPoo/747255/28096401/qfsO3v/2/8347111/UV8uAIcY/I/760288/RuqX7F.html?h=r1qRLMw1UFwwE0anUcShmhKbLf8hDik_JuUMGwfC3ZM .

Avis de confidentialité – Avertissement relatif à la Loi sur l’accès aux documents des organismes publics et sur la protection des renseignements personnels. L’information transmise avec ce courrier électronique est de nature privilégiée et confidentielle. Elle est destinée à l’usage exclusif de la personne visée. Si vous n’êtes pas cette personne, nous vous saurions gré d’en aviser l’expéditeur et de détruire le contenu sans le communiquer ou le reproduire.

Hydro-Québec

Accéder à mon Espace client https://ofsys.hydroquebec.com/T/OFC4/L2S/8509/B2743727/OPoo/747255/28096406/CNKene/1/8347111/UV8uAIcY/I/760288/RuqX7F.html?h=S-3EmX3OXGL06FMe5XNe9c_Nw4OeIOwDcusAGnFCc-M Mieux consommer https://ofsys.hydroquebec.com/T/OFC4/L2S/8509/B2743727/OPoo/747255/28096407/yGWuaI/1/8347111/UV8uAIcY/I/760288/RuqX7F.html?h=ihy5M_GidAVD-G2tejlOOKnA6WH6yhmc99VdlMXNbFk Conseils de sécurité https://ofsys.hydroquebec.com/T/OFC4/L2S/8509/B2743727/OPoo/747255/28096408/OBuZn5/1/8347111/UV8uAIcY/I/760288/RuqX7F.html?h=-MO2ERY9RUUKqwouV5MwKccvIsvhcwGxAulIZNZlcdo Me désabonner https://ofsys.hydroquebec.com/T/OFC4/L2S/8509/B2743727/OPoo/747255/28096409/MdVccX/1/8347111/UV8uAIcY/I/760288/RuqX7F.html?h=mKt6c3d3CdNiuBMs353Sc3mYffOfXLvDwzqpHdgzkzU

Ce courriel a été envoyé à votre adresse courriel @.***, car vous êtes inscrit ou inscrite au tarif D avec l’option de crédit hivernal. --

D. Galarneau 92 Marie Chauvin Boucherville, QC J4B 4G5 450-655-4327 @.** **@.***>

Soleil01 commented 1 year ago

Ok. I modified the custom template for : {{ text [0:20000] }}

Worked, but the message is only 9263 characters long. Why is it not 20000 characters, as my custom template extracted???

I had to eliminate the subject parameter of the search that did not seem to work as intended: … subject "nement de pointe"

Le ven. 16 juin 2023 à 01:33, Jan Bouwhuis @.***> a écrit :

About the special characters, may be you can supply an email sample so I can investigate the encoding issue.

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/94697#issuecomment-1594128620, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIKN3ZXK25FZ3CT7FO5NY6TXLPV3TANCNFSM6AAAAAAZIP62BQ . You are receiving this because you authored the thread.Message ID: @.***>

--

D. Galarneau 92 Marie Chauvin Boucherville, QC J4B 4G5 450-655-4327 @.** **@.***>

jbouwh commented 1 year ago

Emails often are larger because of the encoding

Soleil01 commented 1 year ago

Yes, it seems that the HTML is already interpreted in your integration. The HTML appeared entirely in the old version (IMAP content email integration) so it explains why it was 81575 characters long... And once the HTML is interpreted, this email seems just 9263 characters long.

This way, it may be simpler for me to extract the data I need…

But the question is: why is the text field empty while the custom field is having the full text???

Le ven. 16 juin 2023 à 13:08, Jan Bouwhuis @.***> a écrit :

Emails often are larger because of the encoding

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/94697#issuecomment-1594994862, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIKN3ZW2XQIIY6RLP7AVIIDXLSHH3ANCNFSM6AAAAAAZIP62BQ . You are receiving this because you authored the thread.Message ID: @.***>

--

D. Galarneau 92 Marie Chauvin Boucherville, QC J4B 4G5 450-655-4327 @.** **@.***>

jbouwh commented 1 year ago

The library to extract the content is the same as used in email_content_sensor. The algorithm is to prefer plain text, if that is not available, the HTML part will be returned. So may be the mail content was changed, and it includes a plain text part too. The only way to find you is to have the raw eml file. You can edit it to update private email addresses to something like example.com. Also update IP-adresses or hostnames if they are private. You can link it as a file or use a service like pastbin.com.

jbouwh commented 1 year ago

It seems enlarging the size finally worked and further action is not needed, if this is not correct, then feel free to reopen.

home-assistant / core