[ICS] Some ICS not interpreted

[chibenwa]

fetchUID.INBOX_100021.eml.txt

Idem from [Uploading - "Victor Berhault" victor@vberhault.fr - 2024-07-23 1618.eml.txt…]()

The Following message is well interpreted by THunderbird.

In TMail it do not get the calendar headers thus fails at getting the calndar banner

TODO

• Reproduce with a test in https://github.com/apache/james-project/blob/master/server/mailet/integration-testing/src/test/java/org/apache/james/transport/mailets/ICSAttachmentWorkflowTest.java
• Once diagnosed propose a fix.

[Arsnael]

In TMail it do not get the calendar headers thus fails at getting the calndar banner

Isn't that more of a problem on the front then, if it works with other mail clients?

Also, second eml upload link not working

[chibenwa]

Issue is we missed ics header: the back did not detect this was an event

[hungphan227]

I extracted ics part from the provided file, created ics file, attached to new mail and sent it to myself. As I see, it still work well:

[chibenwa]

@hungphan227 I asked you to work with the original mail. Not to craft a new mail. The original ICS is of course fine. That's the original EML that is crafted in such a way that we do not support it.

Please work with the original EML, this format NEED to be supported for interoperability purposes.

[hungphan227]

I created a test with the original mail as input. The mail that the recipient received contains these headers:

X-MEETING-METHOD: REQUEST X-MEETING-UID: 040000008200E00074C5B7101A82E00800000000A083969F65DCDA010000000000000000100000008FDBCDB265265C48AA18F5D73923C077 X-MEETING-SEQUENCE: 0 X-MEETING-DTSTAMP: 20240722T163547Z

Are these missed ics headers you mentioned?

[quantranhong1999]

@hungphan227 I asked the frontend team and they rely on the X-MEETING-UID header existence of the mail, before deciding should they call the CalendarEvent/parse request to display the blue bar.

the mailet to generate that header: ICALToHeader.

It is better to use the calendar parsing pipeline of tmail.linagora.com for easier reproducing IMO.

[hungphan227]

I created a test with the original mail as input. The mail that the recipient received contains these headers:

X-MEETING-METHOD: REQUEST X-MEETING-UID: 040000008200E00074C5B7101A82E00800000000A083969F65DCDA010000000000000000100000008FDBCDB265265C48AA18F5D73923C077 X-MEETING-SEQUENCE: 0 X-MEETING-DTSTAMP: 20240722T163547Z

Are these missed ics headers you mentioned?

The test in pr https://github.com/apache/james-project/pull/2375

[Arsnael]

Having a serious look at it... Maybe before pretending that tmail did not catch at all there was a calendar event, a little look at the logs first would have been better perhaps?

First of all, calendar pipeline in mailetcontainer.xml prod:

            <!-- ICAL pipeline -->
            <mailet match="All" class="StripAttachment">
                <mimeType>text/calendar</mimeType>
                <attribute>rawIcalendar</attribute>
                <onMailetException>ignore</onMailetException>
            </mailet>
            <mailet match="All" class="MimeDecodingMailet">
                <attribute>rawIcalendar</attribute>
                <onMailetException>ignore</onMailetException>
            </mailet>
            <mailet match="All" class="ICalendarParser">
                <sourceAttribute>rawIcalendar</sourceAttribute>
                <destinationAttribute>icalendar</destinationAttribute>
                <onMailetException>ignore</onMailetException>
            </mailet>
            <mailet match="All" class="ICALToHeader">
                <attribute>icalendar</attribute>
                <onMailetException>ignore</onMailetException>
            </mailet>
            <mailet match="All" class="ICALToJsonAttribute">
                <source>icalendar</source>
                <destination>icalendarAsJson</destination>
                <rawSource>rawIcalendar</rawSource>
                <onMailetException>ignore</onMailetException>
            </mailet>
            <mailet match="All" class="AmqpForwardAttribute">
                <uri>amqp://${env:OP_JAMES_AMQP_USERNAME}:${env:OP_JAMES_AMQP_PASSWORD}@${env:OP_JAMES_AMQP_HOST}:${env:OP_JAMES_AMQP_PORT}</uri>
                <exchange>james:events</exchange>
                <attribute>icalendarAsJson</attribute>
                <onMailetException>ignore</onMailetException>
            </mailet>
            <!-- End of ICAL pipeline -->

We can see that for every calendar step in the pipeline, if there is an exception, we ignore it and continue the mail processing. It's fair to think that maybe that's what happened right?

So checking the logs that day around that time, we can see that indeed there was an issue:

{"timestamp":"2024-07-23T10:21:20.201Z","level":"INFO","thread":"spooler-12512","mdc":{"protocol":"MAILET","mail":"Mail1721730079840-9rxC8RWev0qeZ6W0Hs3C","sender":"SRS0=It5W=OX=gmail.com=victor.berhault@ik2.com","recipients":"[btellier@linagora.com, ppereira@linagora.com]","host":"172.17.0.1","action":"MAILET","sessionId":"fbfc545b","state":"local-recipient","mailet":"ICalendarParser"},"logger":"org.apache.james.transport.mailets.ICalendarParser","message":"Error while parsing ICal object","context":"default","exception":"net.fortuna.ical4j.data.ParserException: Error at line 2:Unexpected end of file\n\tat net.fortuna.ical4j.data.CalendarParserImpl.nextToken(CalendarParserImpl.java:657)\n\tat net.fortuna.ical4j.data.CalendarParserImpl.nextToken(CalendarParserImpl.java:641)\n\tat net.fortuna.ical4j.data.CalendarParserImpl.assertToken(CalendarParserImpl.java:484)\n\tat net.fortuna.ical4j.data.CalendarParserImpl.skipNewLines(CalendarParserImpl.java:564)\n\tat net.fortuna.ical4j.data.CalendarParserImpl.assertToken(CalendarParserImpl.java:529)\n\tat net.fortuna.ical4j.data.CalendarParserImpl.parseCalendarList(CalendarParserImpl.java:192)\n\tat net.fortuna.ical4j.data.CalendarParserImpl.parse(CalendarParserImpl.java:163)\n\tat net.fortuna.ical4j.data.CalendarBuilder.build(CalendarBuilder.java:197)\n\tat net.fortuna.ical4j.data.CalendarBuilder.build(CalendarBuilder.java:185)\n\tat net.fortuna.ical4j.data.CalendarBuilder.build(CalendarBuilder.java:172)\n\tat org.apache.james.transport.mailets.ICalendarParser.createCalendar(ICalendarParser.java:143)\n\tat org.apache.james.transport.mailets.ICalendarParser.lambda$addCalendarsToAttribute$1(ICalendarParser.java:125)\n\tat java.base/java.util.stream.ReferencePipeline$7$1.accept(Unknown Source)\n\tat java.base/java.util.Iterator.forEachRemaining(Unknown Source)\n\tat java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)\n\tat java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)\n\tat java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)\n\tat org.apache.james.transport.mailets.ICalendarParser.addCalendarsToAttribute(ICalendarParser.java:126)\n\tat org.apache.james.transport.mailets.ICalendarParser.lambda$service$0(ICalendarParser.java:119)\n\tat java.base/java.util.Optional.ifPresent(Unknown Source)\n\tat org.apache.james.transport.mailets.ICalendarParser.service(ICalendarParser.java:119)\n\tat org.apache.james.mailetcontainer.impl.ProcessorImpl.process(ProcessorImpl.java:81)\n\tat com.github.fge.lambdas.consumers.ConsumerChainer.lambda$sneakyThrow$9(ConsumerChainer.java:73)\n\tat java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)\n\tat java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)\n\tat java.base/java.util.Collections$2.tryAdvance(Unknown Source)\n\tat java.base/java.util.Collections$2.forEachRemaining(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)\n\tat java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)\n\tat java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)\n\tat java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source)\n\tat org.apache.james.mailetcontainer.impl.MailetProcessorImpl.executeProcessingStep(MailetProcessorImpl.java:162)\n\tat org.apache.james.mailetcontainer.impl.MailetProcessorImpl.lambda$service$0(MailetProcessorImpl.java:130)\n\tat java.base/java.util.stream.ReduceOps$1ReducingSink.accept(Unknown Source)\n\tat java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)\n\tat java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)\n\tat java.base/java.util.stream.ReferencePipeline.reduce(Unknown Source)\n\tat org.apache.james.mailetcontainer.impl.MailetProcessorImpl.service(MailetProcessorImpl.java:128)\n\tat org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.handleWithProcessor(AbstractStateCompositeProcessor.java:98)\n\tat org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:80)\n\tat org.apache.james.mailetcontainer.lib.AbstractStateMailetProcessor.toProcessor(AbstractStateMailetProcessor.java:151)\n\tat org.apache.james.mailetcontainer.impl.MailetProcessorImpl.lambda$executeProcessingStep$7(MailetProcessorImpl.java:167)\n\tat com.github.fge.lambdas.consumers.ConsumerChainer.lambda$sneakyThrow$9(ConsumerChainer.java:73)\n\tat java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)\n\tat java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)\n\tat java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)\n\tat java.base/java.util.Collections$2.tryAdvance(Unknown Source)\n\tat java.base/java.util.Collections$2.forEachRemaining(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)\n\tat java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)\n\tat java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)\n\tat java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source)\n\tat org.apache.james.mailetcontainer.impl.MailetProcessorImpl.executeProcessingStep(MailetProcessorImpl.java:167)\n\tat org.apache.james.mailetcontainer.impl.MailetProcessorImpl.lambda$service$0(MailetProcessorImpl.java:130)\n\tat java.base/java.util.stream.ReduceOps$1ReducingSink.accept(Unknown Source)\n\tat java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)\n\tat java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)\n\tat java.base/java.util.stream.ReferencePipeline.reduce(Unknown Source)\n\tat org.apache.james.mailetcontainer.impl.MailetProcessorImpl.service(MailetProcessorImpl.java:128)\n\tat org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.handleWithProcessor(AbstractStateCompositeProcessor.java:98)\n\tat org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:80)\n\tat org.apache.james.mailetcontainer.lib.AbstractStateMailetProcessor.toProcessor(AbstractStateMailetProcessor.java:151)\n\tat org.apache.james.mailetcontainer.impl.MailetProcessorImpl.lambda$executeProcessingStep$7(MailetProcessorImpl.java:167)\n\tat com.github.fge.lambdas.consumers.ConsumerChainer.lambda$sneakyThrow$9(ConsumerChainer.java:73)\n\tat java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)\n\tat java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)\n\tat java.base/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)\n\tat java.base/java.util.Collections$2.tryAdvance(Unknown Source)\n\tat java.base/java.util.Collections$2.forEachRemaining(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)\n\tat java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)\n\tat java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)\n\tat java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source)\n\tat org.apache.james.mailetcontainer.impl.MailetProcessorImpl.executeProcessingStep(MailetProcessorImpl.java:167)\n\tat org.apache.james.mailetcontainer.impl.MailetProcessorImpl.lambda$service$0(MailetProcessorImpl.java:130)\n\tat java.base/java.util.stream.ReduceOps$1ReducingSink.accept(Unknown Source)\n\tat java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)\n\tat java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)\n\tat java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)\n\tat java.base/java.util.stream.ReferencePipeline.reduce(Unknown Source)\n\tat org.apache.james.mailetcontainer.impl.MailetProcessorImpl.service(MailetProcessorImpl.java:128)\n\tat org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.handleWithProcessor(AbstractStateCompositeProcessor.java:98)\n\tat org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:80)\n\tat org.apache.james.mailetcontainer.impl.JamesMailSpooler$Runner.performProcessMail(JamesMailSpooler.java:135)\n\tat org.apache.james.mailetcontainer.impl.JamesMailSpooler$Runner.lambda$processMail$4(JamesMailSpooler.java:127)\n\tat reactor.core.publisher.MonoRunnable.subscribe(MonoRunnable.java:49)\n\tat reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)\n\tat reactor.core.publisher.MonoUsing.subscribe(MonoUsing.java:109)\n\tat reactor.core.publisher.Mono.subscribe(Mono.java:4568)\n\tat reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:202)\n\tat reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)\n\tat reactor.core.publisher.Mono.subscribe(Mono.java:4552)\n\tat reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:126)\n\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)\n\tat reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)\n\tat java.base/java.util.concurrent.FutureTask.run(Unknown Source)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n"}

We can see that TMail did identify well that there was a calendar event attached to the email. It stripped it, did the mime decoding, but on the ICalendarParser step the library used ical4j returned the above error. More or less though this : net.fortuna.ical4j.data.ParserException: Error at line 2:Unexpected end of file

Which is quite cryptic, and @hungphan227 demonstrated by adding the provided eml in the ticket to the tests that ical4j seems to be able to parse it without issues (which I double checked)

So it seems that something went wrong in the lib ical4j at that particular moment, but as the eml seems valid when testing with it... some bug proper to the lib?

At this stage except maybe potentially reaching the ical4j community and opening an issue on their backlog at https://github.com/ical4j/ical4j , I don't see what much else we can do on our side @chibenwa ?

[chibenwa]

ics-bug.eml.txt

------=_Part_130_2015191872.1721730080113
BEGIN:VCALENDAR
...
END:VCALENDAR
Content-Type: text/calendar; method=REQUEST;
    charset="utf-8"
Content-Transfer-Encoding: 7bit
Content-Language: fr
Content-Disposition: attachment

------=_Part_130_2015191872.1721730080113--

The mime looks corrupted

The mime library then interprets the ICS content as headers

And it actually interpret ICS content after Content-Disposition: attachment so content is \r\n

Why the mime ends up like that? Casual suspect: TextCalendarBodyToAttachment

[chibenwa]

BINGO

    @Test
    public void serviceShouldChangeMessageContentTypeToMultipartWhenTextCalendarMessage() throws Exception {
        String messageContent = "Content-type: text/calendar; method=REPLY; charset=UTF-8\r\n" +
            "Content-transfer-encoding: 8BIT\r\n" +
            "\r\n" +
            "BEGIN:VCALENDAR";
        MimeMessage message = MimeMessageUtil.mimeMessageFromString(messageContent);

        Mail mail = FakeMail.builder()
            .name("name")
            .mimeMessage(message)
            .build();

        System.out.println(MimeMessageUtil.asString(mail.getMessage()));

        System.out.println("==============");

        mailet.service(mail);

        System.out.println(MimeMessageUtil.asString(mail.getMessage()));

        assertThat(mail.getMessage().isMimeType("multipart/*")).isTrue();
    }

=>

Content-type: text/calendar; method=REPLY; charset=UTF-8
Content-transfer-encoding: 8BIT

BEGIN:VCALENDAR
==============
Content-Type: multipart/mixed; 
    boundary="----=_Part_4_870759734.1724139960082"
MIME-Version: 1.0
Date: Tue, 20 Aug 2024 09:46:00 +0200 (CEST)
Message-ID: <1988962280.5.1724139960085@hp-HP-ProBook-440-G5>

------=_Part_4_870759734.1724139960082
BEGIN:VCALENDAR
Content-type: text/calendar; method=REPLY; charset=UTF-8
Content-transfer-encoding: 8BIT
Content-Disposition: attachment

------=_Part_4_870759734.1724139960082--

[hungphan227]

ics-bug.eml.txt --->where did you get this from?

[chibenwa]

ics-bug.eml.txt --->where did you get this from?

My inbox, using the log data from Rene.

I'm putting together a fix...

[chibenwa]

Here we go: https://github.com/apache/james-project/pull/2384