iCAL problem - All upcoming appointments dates wrong - Month value

[AWSW-de]

Hello there and merry Christmas to you,

I think I have a reproducible problem with the calendar module.

Platform: (Raspberry Pi 3, Mac, Recent Linux NOOBS).

Node Version: Recent version installed with your automatic install.

MagicMirror Version: Recent version installed with your automatic install.

Description:

The problem is, that the months are not fetched correctly. Today is the 26th of Dezember 2017 and it shows appointments for the next upcoming days that are in several weeks or months. Please see the attached picture. The fetched day seems to fit, but the months seem to be set to this month or mostly to January 2018.

My system is set to German language and time zone Europe/Berlin.

I already did a clean install of the recent MagicMorror version and just changed the URL in the config.js to my public ical calendar. The problem is then reproducible again.

Thanks for your help in advance. :)

Configuration: What does the used config.js file look like? Don't forget to remove any sensitive information!

Just changed the sample URL to my public iCal URL.

[AWSW-de]

Hello,

I think I found the reason for this. All not correctly shown appointments are recurring ones and they have 2 notifications: 1 for the day of the appointment itself and one 1 day before.

Removing the one of the day before of the appointment helped to remove most of the not correctly shown entries in MagicMirror. For the remaining ones I just needed to open and store the appointment again.

Thanks for your help and for this nice project. :)

Kind regards

[MichMich]

Ah. For now it would be great to add that to the documentation. It would be great if you could send a PR for this. I'll close this for now. Feel free to reopen!

[E3V3A]

@MichMich I don't see why you would want to close the issue if there has not been any PR fixing it? I just found a similar issue reported.

[MichMich]

@E3V3A Reopened.

[E3V3A]

FWIW:
I recall seeing an ical file that had some additional timezones and multiple entries. Perhaps have a close look at that to rule out bad or confusing ical files.

[pinsdorf]

Another aspect is that the calendar module cannot cope with iCal URLs from Microsoft Live/Hotmail/Outlook calendars. I tried this and got a parsing exception because of start/end times containing timezone information in the date/time string.

Those strings are apparently compliant to the iCal standard, yet, ical.js doesn't parse this: ical.js issue #76. IMHO, we should see if ical,js fixes the issue (there is even a PR #84) issue or try another iCal parser implementation. I'm happy to help.

[MichMich]

@pinsdorf A PR would be EXTREMELY appreciated.

[mbalfour]

If you're willing to dust it off, the fix should be in this PR: https://github.com/MichMich/MagicMirror/pull/375 . Unfortunately, it also contained an implementation of a monthly view, which is why this PR got rejected. But if you pull the monthly view parts out and focus more on the CalendarFetcher side of the changes, you should be able to get recurrences working. I don't have the time to do it myself right now, but you can also see functionally equivalent changes in the MMM-CalendarExt module if you need another example of what the changes look like. See these PRs: https://github.com/eouia/MMM-CalendarExt/pull/3 https://github.com/eouia/MMM-CalendarExt/pull/18

[mlbarrow]

Not sure if it is exactly related, but I recently deployed MM2 for home and have run into the calendar drama. Initially, appointments were missing with the base installation (2.7.1). After poking around a bit, I upgraded to the latest node-ical (0.9.2) and the missing appointments showed up. Unfortunately, they're showing up a day early. I also noticed birthday appointment showing up 2 months early. Are other folks still having significant issues with the iCal parsing? I can't say I'm surprised, calendaring is such a hard problem...

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mlbarrow]

Though the stale bot has marked this as stale, I continue to see this problem. It continued after the fixes to the ical code in 2.8.0 and persists today with the upgrade to 2.9.0. Would be willing to help diagnose this with someone with more clue than I have on the ical stuff.

[AWSW-de]

Though the stale bot has marked this as stale, I continue to see this problem. It continued after the fixes to the ical code in 2.8.0 and persists today with the upgrade to 2.9.0. Would be willing to help diagnose this with someone with more clue than I have on the ical stuff.

Hello, this helped me in the past and since then too:

Hello, I think I found the reason for this. All not correctly shown appointments are recurring ones and they have 2 notifications: 1 for the day of the appointment itself and one 1 day before. Removing the one of the day before of the appointment helped to remove most of the not > correctly shown entries in MagicMirror. For the remaining ones I just needed to open and store the appointment again.

HTH!

[nerdoug]

I've just (Oct 21,2019) created a MM system with latest s/w on PI-2, and am seeing the case where some Google calendar appointments are appearing one day early. I've seen something like this before, and offer a hypothesis: Perhaps the issue is related to the part of the day where the local date is different that the UTC date. I live near Toronto, Canada, and it is the late night events in my calendar that show up a day early. My UTC offset is currently 4 hours ( for a couple more weeks, then 5) and from 8PM to midnight my time, the UTC date is one day later than my local time. It's now about 10 PM here, and I'm going to watch to see if an evening reminder 2 days away corrects itself at midnight. I'll report back (even if only to confess I fell asleep)

[nerdoug]

While awaiting the stroke of midnight I kept poking around, and found another pattern in my admittedly skimpy data: The troublesome events have a different timestamp format in the iCal file than events that work fine in MM. In, particular, they have an explicit time zone, where the ones that work use UTC time.

Problematic: DTSTART;TZID=America/Toronto:20191023T200000

Working: DTSTART:20191024T010000Z

I've verified that the timezone info at the top of the iCal file is correct:

BEGIN:VTIMEZONE
TZID:America/Toronto
X-LIC-LOCATION:America/Toronto
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE

and there are other instances of events with explicit timezones in their iCal start time that display correctly in MM. However, none are in the 8PM to midnight interval where UTC/Local dates are different. I have some of captured iCal files if they're of interest to anyone.

[nerdoug]

20 minutes after midnight, and no change. The event I'll call ARES should be at 8 PM on Oct 23, but still shows as 9 PM on Oct 22 in MM. Being off by one hour feels related to Timezones and daylight saving time again. I'm wondering if the workaround mentioned by someone else where you delete and re-add a problematic entry effectively changes the event start time to use UTC in the iCal file. No time like the present... Yup, in about a minute (my fetchInterval = 60000) the new event appeared in the correct order, Oct 23 at 8PM. Curling the iCal shows a DTSTART ending in Z for Zulu time...

BEGIN:VEVENT
DTSTART:20191024T000000Z
DTEND:20191024T010000Z
DTSTAMP:20191022T042955Z
UID:68v89mvi7odqek44i0pa21991f@google.com
CREATED:20191022T042923Z
DESCRIPTION:
LAST-MODIFIED:20191022T042923Z
LOCATION:
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:ARES Net
TRANSP:OPAQUE
END:VEVENT

For comparison, here's the applicable snippet from an iCal curl before I did the delete / re-add...

BEGIN:VEVENT
DTSTART;TZID=America/Toronto:20191023T200000
DTEND;TZID=America/Toronto:20191023T210000
DTSTAMP:20191022T022919Z
UID:7aev6qf2bkubge5g86dingccdo@google.com
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=ACCEPTED;CN="pc: c
 anoe.eh@gmail.com";X-NUM-GUESTS=0:mailto:h9jqtvb5aj1uh4pu52mu2qgac4@group.c
 alendar.google.com
RECURRENCE-ID;TZID=America/Toronto:20191023T200000
CREATED:20171124T042837Z
DESCRIPTION:xx
LAST-MODIFIED:20191022T005034Z
LOCATION:
SEQUENCE:1
STATUS:CONFIRMED
SUMMARY:ARES Net
TRANSP:OPAQUE
END:VEVENT

enough detective work - I'm off to bed.

[nerdoug]

My incorrect MM calendar items have all be resolved by deleting and re-adding any that had an explicit time zone in the DTSTART field as shown in a curl of the iCal URL. (Once I weeded out all the obsolete expired entries in that file, which are being repeatedly downloaded.) I'm not sure how the problematic entries were created, but I'm guessing they were created or edited on my iPhone rather than in the google calendar website. (Resolving that by converting to Android in the Black Friday sales .) I'll watch to verify that the recurring events that had issues this week work properly next week.

[nerdoug]

my problem reoccurred the next time I had a recurring event in the time period when the Date in UTC is different than local time. An event at 8 PM local time is showing a day early in MM. We go to Daylight saving time tonight so it will be interesting to see if symptoms change. Come to think of it, the events that are showing a day early are set for Monday, and a week from Monday, so they're after the time change.

[nerdoug]

Can't resist experimenting - must be a character flaw. Defined a new event a day later than one described above, with no recurrence, and it worked fine. the iCal dates were in UTC, and didn't include Toronto time zone.

Then I defined one an hour later, with weekly recurrence. It appeared in the iCal curl, but the dates included explicit time zones. It hasn't shown up in MM at all, although events before and after the first occurrence do.

Then I went back and changed the first test event to be recurring weekly. It disappeared from my MM display, and the curl'd iCal shows its DTSTART and DTEND timestamps now have explicit time zones.

Last test is to change the 2nd test event, which was originally defined as recurring, to a one time event. Sure enough, it reappeared in MM, and its iCal data went back to UTC timestamps.

Did I say "Last" test? Just kidding. Now I want to do the same tests for an event that happens just before, and just after the period where my UTC date differs from my local date. Think I better take on some more coffee before I start that.

[nerdoug]

OK. To save typing I'll abbreviate "the period when the UTC date differs from my local date" as PWDD. I defined 2 events in the hour before the PWDD, one recurring, the other not. I also defined 2 events similarly in the hour following PWDD. The all appeared as expected in MM. I then modified all 4 events, changing the recurrence for each. They all worked as expected in MM. Did a curl and saw DTEVENT timestamps with/without explicit timestamps following same pattern as before.

Scanned the curl'd iCal file, and found instances of recurring events outside the PWDD which work normally in MM, and have iCAl dates with time zones. I could not find any non-recurring events that have timezones in iCal; they all use UTC in that case.

We ex-skydivers are pretty careful about jumping to false conclusions, but the testing summarized above seems to indicate that either the iCal parser or MM have a problem processing the specific case of recurring events that occur during the period when the UTC date is different from the local date. I should probably figure out how to report this to whoever supports the iCal parser - I'll have a run at that.

And of course, I'll be doing more experiments in a day or two after the Daylight Saving Time change, to see if the symptoms still occur only in the different PWDD.

I'm wondering if all the fixes that mbalfour identified in comments above and PR 375 got implemented? Michmich, if you're following my verbosity here, could you comment on that?

[nerdoug]

I noticed that this topic wasn't in the digest I received recently, although this issue (1105) seems to be open. Being new to this forum stuff, I'm not sure if I've done the right things to bring my experiment findings to the attention of the right people who can investigate and possible fix the problem. Should I be opening another issue for the specific circumstances I've been looking at? Is anyone looking at the info I'm adding to this issue (1105)?

[kellogg76]

I noticed that this topic wasn't in the digest I received recently, although this issue (1105) seems to be open. Being new to this forum stuff, I'm not sure if I've done the right things to bring my experiment findings to the attention of the right people who can investigate and possible fix the problem. Should I be opening another issue for the specific circumstances I've been looking at? Is anyone looking at the info I'm adding to this issue (1105)?

I also have the same issues, also in Canada if that's of any use. Having to remove and re-add entries is a pain if that's the only fix.

I also get some people's birthday's reported as being on the correct date every single month, but only 2 members of my family get the special birthday every month.

[mlbarrow]

I agree that remove/add is a poor solution. I’m not in Canada and have all these issues, including the more-than-annual birthday problem for some folks, too!

-- Michael L. Barrow michael at barrow dot me +1.541-600-2027

On Oct 31, 2019, at 05:46, kellogg76 notifications@github.com wrote:

I noticed that this topic wasn't in the digest I received recently, although this issue (1105) seems to be open. Being new to this forum stuff, I'm not sure if I've done the right things to bring my experiment findings to the attention of the right people who can investigate and possible fix the problem. Should I be opening another issue for the specific circumstances I've been looking at? Is anyone looking at the info I'm adding to this issue (1105)?

I also have the same issues, also in Canada if that's of any use. Having to remove and re-add entries is a pain if that's the only fix.

I also get some people's birthday's reported as being on the correct date every single month, but only 2 members of my family get the special birthday every month.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[nerdoug]

@kellog76, @mlbarrow. Curious on how similar our issues are. I've narrowed my symptoms down to events that are: -recurring -have a scheduled time that occurs in the period when the UTC date is different than the local date.

This period ends at local midnight, and begins at midnight minus the UTC offset for local time (i.e. at midnight UTC). I'm near Toronto and my offset is now 4 hours, so for me, the period is from 8PM to midnight local time. West of zero degrees longitude, the local date is one day earlier that UTC during this period. East of it, the local date is one day later that the UTC date.

For me, recurring MM calendar events in this period appear one day early, or sometimes disappear altogether. I wouldn't be surprised if they appeared one day later for someone West of Greenwich, England.

I think I probably should submit a new issue with these specific symptoms and test results. I have a flurry of Halloween related activity today, but I'll try to figure out how to do that in the next couple of days. If I get really keen, I'll see if I can find and figure out the source code. However, unless it's written in Fortran, an old geezer like me probably won't understand it (grin).

Thanks for responding. I was beginning to think my testing and reporting was pointless.

[secastles]

I'm having a similar problem. Last week (or maybe even a little earlier than that) some of my recurring appointments started showing up a day early on the calendar. I decided to just delete and re-add. However, upon re-adding they aren't even showing up at all. An example is below (taken from the iCal):

UID:blahblahblah@google.com
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;CN=support@yomomma.com;X-NUM-GUESTS=0:mailto:support@yomomma
CREATED:20191105T212504Z
DESCRIPTION:
LAST-MODIFIED:20191105T212611Z
LOCATION:
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:Test Event
TRANSP:OPAQUE
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20191107T150000
DTEND;TZID=America/New_York:20191107T200000
RRULE:FREQ=WEEKLY;BYDAY=TH
DTSTAMP:20191105T212838Z
ORGANIZER;CN=MagicMirror:mailto:whoever@group.calendar.g
 oogle.com

That event shows up in my Google Calendar but won't show up on the MagicMirror (to be fair I'm using MMM-CalendarWeek but it seems related to this issue). I tested with another event too, deleted and re-added. Not showing up. Other events are showing up though. Even some recurring ones. Here's one that shows up correctly (it's been fine since the problems started so I haven't deleted and re-added this one):

UID:blahblahblah@google.com
CREATED:20190731T154142Z
DESCRIPTION:This one works
LAST-MODIFIED:20190916T144401Z
LOCATION:somewhere
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:this is the summary
TRANSP:OPAQUE
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20190914T090000
DTEND;TZID=America/New_York:20190914T093000
RRULE:FREQ=WEEKLY;WKST=SU;UNTIL=20190928T035959Z;BYDAY=SA
DTSTAMP:20191105T212838Z

I'm happy to help dig and test if someone can provide me some direction.

[secastles]

Also, it doesn't seem to matter when during the day I add new test events (9AM, 10PM), none of them show up.

[nerdoug]

Interesting, we're in the same timezone. You might try my retest for our new offset from UTC: 1) define half hour events that recur weekly, about 2 days out, so they'll show up in MM calendar, for following local times: 18:30 19:30 23:30 00:30 (following day)

(These are chosen to bracket the boundaries of the period when the date in UTC time is different than our locale time. I did a similar test a week ago, while we were in DST, and it will be interesting to see if the results change.)

Note which of the appointments appear in the MM calendar 2) go back and change the events to be non-recurring. Wait a while and check to see which ones now appear in the MM calendar.

I'm off to do the same tests myself, and will report back shortly.

[secastles]

OK, I added the four test appointments. Format in iCal looks like this:

UID:blahblahblah@google.com
CREATED:20191105T234753Z
DESCRIPTION:
LAST-MODIFIED:20191105T234753Z
LOCATION:
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:Test4
TRANSP:OPAQUE
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20191107T233000
DTEND;TZID=America/New_York:20191108T000000
RRULE:FREQ=WEEKLY;BYDAY=TH
DTSTAMP:20191105T234846Z
UID:blahblahblah@google.com
CREATED:20191105T234730Z
DESCRIPTION:
LAST-MODIFIED:20191105T234730Z
LOCATION:
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:Test3
TRANSP:OPAQUE
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20191107T193000
DTEND;TZID=America/New_York:20191107T200000
RRULE:FREQ=WEEKLY;BYDAY=TH
DTSTAMP:20191105T234846Z
UID:blahblahblah@google.com
CREATED:20191105T234649Z
DESCRIPTION:
LAST-MODIFIED:20191105T234706Z
LOCATION:
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:Test2
TRANSP:OPAQUE
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20191107T183000
DTEND;TZID=America/New_York:20191107T190000
RRULE:FREQ=WEEKLY;BYDAY=TH
DTSTAMP:20191105T234846Z
UID:blahblahblah@google.com
CREATED:20191105T234631Z
DESCRIPTION:
LAST-MODIFIED:20191105T234658Z
LOCATION:
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:Test1
TRANSP:OPAQUE
END:VEVENT
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20191106T170000
DTEND;TZID=America/New_York:20191106T180000
RRULE:FREQ=WEEKLY;BYDAY=WE
DTSTAMP:20191105T234846Z

BTW: do event blocks start with UID? Just checking.

Results: The only one that shows up in MM is Test4, or the one that starts at 00:30.

I then converted them all to one-time events and restarted the MagicMirror service.

Results: All events show correctly.

So, based on this: it seems to only be affecting recurring appointments, but only some of them. Like you suggest, perhaps if the appointment is on the same day as UTC then it's fine, otherwise it disappears into the ether.

[secastles]

Something's odd about my particular situation. MM has been working fine for months and months up until last week, or maybe the week before. That's when the issue started happening. Before that all recurring events were fine.

[nerdoug]

I just created issue #1803 for the specific issue: Recurring Google Calendar events for times when UTC date differ from local date appear at wrong time in MM #1803

[nerdoug]

@tisnatch: I think the section in the iCal curl for a particular event starts with BEGIN:VEVENT and ends with END:VEVENT so some of your iCal snippets above are actually parts of two different events. Also, I'm surprised that your test1 didn't appear in MM. My event that was set up similarly did appear correctly, and your iCal stuff looks right to me.

OK, I'm definitely shifting over to #1803 which, unlike this general purpose issue, is specific to the circumstances I've been investigating.

[MichMich]

Please use https://github.com/MichMich/MagicMirror/issues/1798