Closed dastious closed 5 years ago
Can you please post the part of the response XML containing the string (GMT+01:00) Amsterdam, Berlin, Berne, Rome, Stockholm, Vienne
? See https://github.com/ecederstrand/exchangelib#troubleshooting
in fact the real error is not the warning :
File "test.py", line 50, in
for item in account_source.calendar.all().order_by('datetime_received'): File "/home/migrator/calmigrator/venv/lib/python3.5/site-packages/exchangelib/queryset.py", line 311, in iter for val in self._format_items(items=self._query(), return_format=self.return_format): File "/home/migrator/calmigrator/venv/lib/python3.5/site-packages/exchangelib/queryset.py", line 390, in _as_items for i in iterable: File "/home/migrator/calmigrator/venv/lib/python3.5/site-packages/exchangelib/account.py", line 629, in fetch shape=IdOnly, File "/home/migrator/calmigrator/venv/lib/python3.5/site-packages/exchangelib/services.py", line 597, in _pool_requests for elem in r.get(): File "/home/migrator/calmigrator/venv/lib/python3.5/site-packages/exchangelib/services.py", line 330, in _get_elements_in_response container_or_exc = self._get_element_container(message=msg, name=self.element_container_name) File "/home/migrator/calmigrator/venv/lib/python3.5/site-packages/exchangelib/services.py", line 303, in _get_element_container raise self._get_exception(code=response_code, text=msg_text, msg_xml=msg_xml) exchangelib.errors.ErrorTimeoutExpired: The request timed out.
It seems that this Timeout error is only happenning with calendar not with all other folder and things.
Is it ok if I only show you this part of the xml ? Since this calendar have private information. Well I can make a fake calendar if you really need to full xml. Tell me.
<t:Start>2010-06-23T00:00:00+02:00</t:Start>
<t:End>2010-06-24T00:00:00+02:00</t:End>
<t:OriginalStart>2010-06-23T00:00:00+02:00</t:OriginalStart>
</t:LastOccurrence>
<t:StartTimeZone
Name="(GMT+01:00) Amsterdam, Berlin, Berne, Rome, Stockholm, Vienne"
Id="(GMT+01:00) Amsterdam, Berlin, Berne, Rome, Stockholm, Vienne">
<t:Periods>
<t:Period Bias="-PT1H" Name="Standard"
Id="trule:Microsoft/Registry/(GMT+01:00) Amsterdam, Berlin, Berne, Rome, Stockholm, Vienne/1601-Standard"/>
<t:Period Bias="-PT2H" Name="Daylight"
Id="trule:Microsoft/Registry/(GMT+01:00) Amsterdam, Berlin, Berne, Rome, Stockholm, Vienne/1601-Daylight"/>
</t:Periods>
<t:TransitionsGroups>
<t:TransitionsGroup Id="0">
<t:RecurringDayTransition>
<t:To Kind="Period">trule:Microsoft/Registry/(GMT+01:00) Amsterdam, Berlin, Berne, Rome, Stockholm,
Vienne/1601-Daylight
</t:To>
<t:TimeOffset>PT3H</t:TimeOffset>
<t:Month>3</t:Month>
<t:DayOfWeek>Sunday</t:DayOfWeek>
<t:Occurrence>-1</t:Occurrence>
</t:RecurringDayTransition>
<t:RecurringDayTransition>
<t:To Kind="Period">trule:Microsoft/Registry/(GMT+01:00) Amsterdam, Berlin, Berne, Rome, Stockholm,
Vienne/1601-Standard
</t:To>
<t:TimeOffset>PT2H</t:TimeOffset>
<t:Month>10</t:Month>
<t:DayOfWeek>Sunday</t:DayOfWeek>
<t:Occurrence>-1</t:Occurrence>
</t:RecurringDayTransition>
</t:TransitionsGroup>
</t:TransitionsGroups>
<t:Transitions>
<t:Transition>
<t:To Kind="Group">0</t:To>
</t:Transition>
</t:Transitions>
</t:StartTimeZone>
<t:EndTimeZone Name="" Id="(GMT+01:00) Amsterdam, Berlin, Berne, Rome, Stockholm, Vienne">
<t:Periods>
<t:Period Bias="-PT1H" Name="Standard"
Id="trule:Microsoft/Registry/(GMT+01:00) Amsterdam, Berlin, Berne, Rome, Stockholm, Vienne/1601-Standard"/>
<t:Period Bias="-PT2H" Name="Daylight"
Id="trule:Microsoft/Registry/(GMT+01:00) Amsterdam, Berlin, Berne, Rome, Stockholm, Vienne/1601-Daylight"/>
</t:Periods>
<t:TransitionsGroups>
<t:TransitionsGroup Id="0">
<t:RecurringDayTransition>
<t:To Kind="Period">trule:Microsoft/Registry/(GMT+01:00) Amsterdam, Berlin, Berne, Rome, Stockholm,
Vienne/1601-Daylight
</t:To>
<t:TimeOffset>PT3H</t:TimeOffset>
<t:Month>3</t:Month>
<t:DayOfWeek>Sunday</t:DayOfWeek>
<t:Occurrence>-1</t:Occurrence>
</t:RecurringDayTransition>
<t:RecurringDayTransition>
<t:To Kind="Period">trule:Microsoft/Registry/(GMT+01:00) Amsterdam, Berlin, Berne, Rome, Stockholm,
Vienne/1601-Standard
</t:To>
<t:TimeOffset>PT2H</t:TimeOffset>
<t:Month>10</t:Month>
<t:DayOfWeek>Sunday</t:DayOfWeek>
<t:Occurrence>-1</t:Occurrence>
</t:RecurringDayTransition>
</t:TransitionsGroup>
</t:TransitionsGroups>
<t:Transitions>
<t:Transition>
<t:To Kind="Group">0</t:To>
</t:Transition>
</t:Transitions>
</t:EndTimeZone>
<t:ConferenceType>0</t:ConferenceType>
<t:AllowNewTimeProposal>true</t:AllowNewTimeProposal>
<t:IsOnlineMeeting>false</t:IsOnlineMeeting>
<t:NetShowUrl/>
The timezone IDs are really weird. They are not standard Microsoft or CLDR timezone values. You can add a translation to a reasonable pytz
timezone ID to exchangelib.winzone.MS_TIMEZONE_TO_PYTZ_MAP
at the top of your script.
Timeouts occur because your server is too loaded. You have a couple of options:
account.calendar.only('start', 'end', 'subject')
. See https://github.com/ecederstrand/exchangelib#searchingServiceAccount
to hide the timeout if they occur irregularly. See https://github.com/ecederstrand/exchangelib#setup-and-connectingJust to know that I'm already using ServiceAccount so that's why i tough this is not a normal behavior. But i will try with decreasing the number of items in a paged request.
(also the timezone come from exchange 2004 migrated to ex 2010 migrated to ex 2016... that's probably why there a weird)
Thanks a lot for the hints !
ErrorTimeoutExpired
is the server telling us to step back a bit because it's too busy. Newer versions of exchangelib try to back off gracefully, decrease the connection pool size etc, but we may still re-throw ErrorTimeoutExpired
if we could not scale back further, or if we got too impatient.
Closing because I don't think there's anything to change in exchangelib. Feel free to re-open if you think otherwise.
Hi, hitting the timezone error today
Cannot convert value '(UTC+01:00) Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna' on field '_start_timezone' to type <class 'exchangelib.ewsdatetime.EWSTimeZone'> (unknown timezone ID)
with exchangelib 4.9.0
any thought? thanks!
The suggestion is to add a custom entry for this time zone definition in exchangelib.winzone.MS_TIMEZONE_TO_IANA_MAP
Thanks for the fast answer @ecederstrand . Could it be that there is something wrong on the server side? Should I reach out to the Exchange server admin?
It's my understanding that this is a config issue on the Exchange / Windows server. The timezone ID should be e.g. W. Europe Standard Time
. Alternatively, we'll have to find out how the (GMT+01:00) Amsterdam, Berlin, Berne, Rome, Stockholm, Vienne
timezone ID came to be before we can discuss a possible solution for exchangelib.
I found it: the TZID in calendar entries comes from each individual calendar client or the sender of the calendar invitations, and not from the server. It's not standardized or canonicalized.
This particular one is that of the Microsoft OWA rich web client. The list of TZIDs from OWA is here https://gist.github.com/monperrus/12f852e9b629e7028494ffc92da52aeb
Thanks for the list! Where did you get it from? If we add this to exchangelib, I'd like a reference to official Microsoft docs. I've found this so far: https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/default-time-zones?view=windows-11 but it would be nice with something we can download and process automatically to test that whatever we import into exchangelib is up-to-date.
Also, that's really depressing behavior. I've seen my share of timezone craziness over the years. Allowing any timezone input without validation is a whole new level of crazy.
It gets even worse. The timezone ID (GMT+01:00) Amsterdam, Berlin, Berne, Rome, Stockholm, Vienne
that the OP saw is not the same as the (UTC+01:00) Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna
mentioned in the docs I just posted. I assume e.g. the Berne -> Bern
difference is due to localization. We can't possibly bring in the English version plus all localized versions of Microsoft's timezone names.
At this point, I don't have any good solutions on how to reliably translate the posted timezone definitions to official IANA timezones which is what the Python timezone library expects. We could try to somehow use the TransitionsGroups to match an IANA timezone, but the TransitionsGroups definition delivered in the XML is by no means complete compared to the full set of transitions that e.g. Europe/Amsterdam
defines.
I got the list by reverse-engineering the OWA JS code.
We can't possibly bring in the English version plus all localized versions of Microsoft's timezone names.
I agree.
I see that in those cases, the TZID is just an identifier between a VEVENT and a VTIMEZONE. So TZID can be any arbitrary string.
What about matching on the offset from the VTIMEZONE instead of the TZID if this is a reference to a VTIMEZONE?
Unfortunately, the offset is not sufficient to match with a timezone. There may be multiple timezones within a single offset with varying transitions between standard time and daylight saving time, and the offset will vary depending on whether the timestamp is within DST or not.
The more assumptions we make trying to match arbitrary timezone labels with actual IANA timezones, the more difficult-to-debug issues we create when those assumptions are wrong.
The more assumptions we make trying to match arbitrary timezone labels with actual IANA timezones, the more difficult-to-debug issues
agree
Unfortunately, the offset is not sufficient to match with a timezone.
we can do that as best effort, only if the TZID is not known.
it would be strictly backward compatible and better than the current error Cannot convert value
While I appreciate the suggestions, I really don't like guessing. Most people don't check warnings, and I prefer errors over difficult-to debug issues.
As an example, Iceland is GMT+0 and does not observe DST. Britain is also GMT+0 but does observe DST. If we were to guess Britain but the timezone was in fact Iceland, then we would get the time wrong if we move a meeting from September to November.
As an example, Iceland is GMT+0 and does not observe DST. Britain is also GMT+0 but does observe DST. If we were to guess Britain but the timezone was in fact Iceland, then we would get the time wrong if we move a meeting from September to November.
I see. That's probably the root cause of quite many subtle calendar bugs that we see on the Internet (and that I've experienced), resulting from a combination of wrong timezone identification combined with event movements over DST weekends.
I'm reading RFC 5545, "Internet Calendaring and Scheduling Core Object Specification (iCalendar)" and there is no obvious solution. It simply says "This document does not define a naming convention for time zone identifiers".
So coming back to a good timezone identification algorithm, here is a new proposal:
WDYT?
If you want to match on offset and DST, then for 90% of the places in the world where people actually live and create meetings in Exchange, there's going to be more than one candidate. GMT+1 will probably have at least 10.
That leaves us with guessing based on some sort of fuzzy string matching. Which means we'll often be right and sometimes wrong. Which I think still leaves exchangelib users in a worse situation than getting an error. I would prefer to simply improve the error message so users know exactly what to do if they get this error. After all, the solution is really simple:
from exchangelib.winzone import MS_TIMEZONE_TO_IANA_MAP as MS_TO_IANA
MS_TO_IANA["(GMT+01:00) Amsterdam, Berlin, Berne, Rome, Stockholm, Vienne"] = "Europe/Amsterdam"
# Your code here
Unless you have an unusually diverse set of accounts, the amount of custom entries will be low. You can even choose to catch the UnknownTimeZone
exception and do your own fuzzy string matching to populate MS_TIMEZONE_TO_IANA_MAP
automatically.
Which means we'll often be right and sometimes wrong
Agree
Unless you have an unusually diverse set of accounts, the amount of custom entries will be low.
Not really, because one does not control this field, it is set by the calendar client of all people sending invitations, so it's open-ended.
Maybe the algorithm above could be part of another library to be created: icaltz: deduce timezones from icalendar VTIMEZONE.
Reading more about timezone inference, the deduction algorithm would also have to take into account fields TZNAME
and X-LIC-LOCATION
.
Now searching for an existing implementation of a similar algorithm, not finding any in https://github.com/libical/libical/
I do this to get all email and then calendar, with email it's seems to work well. But if i list all calendar item like this :
The script crash with this message :
I tested without order_by('datetime_received') it produces the same kind of error but in a loop (not just 2 errors) :
I tested with the master branch same thing.