Unicode URLs break completely with DAV storage and Python 2

[aitzkora]

Hi, I tried to run a vdirsyncer sync and I get the following error

/usr/lib64/python2.7/urllib.py:1294: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal

return ''.join(map(quoter, s)) Unhandled exception occured for inria_calendar/Calendar. Traceback (most recent call last): File "build/bdist.linux-x86_64/egg/vdirsyncer/cli/tasks.py", line 77, in sync_collection force_delete=force_delete File "build/bdist.linux-x86_64/egg/vdirsyncer/sync.py", line 163, in sync b_info.prepare_idents(storage_a.read_only) File "build/bdist.linux-x86_64/egg/vdirsyncer/sync.py", line 111, in prepare_idents if prefetch else ()): File "build/bdist.linux-x86_64/egg/vdirsyncer/storage/dav.py", line 385, in get_multi href_xml.append('{}/D:href'.format(_encode_href(href))) File "build/bdist.linux-x86_64/egg/vdirsyncer/storage/dav.py", line 33, in _encode_href return utils.compat.urlquote(x, '/@') File "/usr/lib64/python2.7/urllib.py", line 1294, in quote return ''.join(map(quoter, s)) KeyError: u'\xc3'

I installed vdisyncer from souces and may be it is due to my urllib... I you have any ideas, i would be interested Bests regards, marc

[untitaker]

Hello @aitzkora, what is the config you're using? Could you send me the output of vdirsyncer -vdebug sync in private (per email)?

[untitaker]

Thanks for the output. I just noticed that the error message you're recieving is never supposed to happen in that way: URLs should never be unicode. Could you try running the testsuite from latest master? (I just pushed a commit).

virtualenv -p /path/to/vdirsyncers/python env
. env/bin/activate
make install-test
make test

[untitaker]

Nevermind, I've found the bug -- brainfart on my side.

[untitaker]

Alright, the unicode_urls branch should contain an appropriate fix for Python 2. Note that the stable version of vdirsyncer probably works flawlessly under Python 3.

pip install -U git+https://github.com/untitaker/vdirsyncer.git@unicode_urls

[aitzkora]

Hi, i run the command pip install -U git+https://github.com/untitaker/vdirsyncer.git@unicode_urls and now i get an different error. Thanks a lot for the job your have done. Bests regards, marc

Traceback (most recent call last): File "/home/fux/vdirsyncer/lib/python2.7/site-packages/vdirsyncer/cli/tasks.py", line 77, in sync_collection force_delete=force_delete File "/home/fux/vdirsyncer/lib/python2.7/site-packages/vdirsyncer/sync.py", line 175, in sync action(a_info, b_info, conflict_resolution) File "/home/fux/vdirsyncer/lib/python2.7/site-packages/vdirsyncer/sync.py", line 188, in inner .format(ident, dest.storage)) UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 30: ordinal not in range(128)

[untitaker]

Thanks, please try again.

[aitzkora]

i tried ~/vdirsyncer/bin/pip2.7 install -U git+https://github.com/untitaker/vdirsyncer.git@unicode_urls and when I run ~/vdirsyncer/bin/vdirsyncer sync I got

Syncing inria_calendar/Calendar error: Directory /home/fux/.calendars/Calendar does not exist. error: No collection Calendar found for storage inria_calendar_local. Should vdirsyncer attempt to create it? [y/N]: y Unhandled exception occured for inria_calendar/Calendar. Traceback (most recent call last): File "/home/fux/vdirsyncer/lib/python2.7/site-packages/vdirsyncer/cli/tasks.py", line 77, in sync_collection force_delete=force_delete File "/home/fux/vdirsyncer/lib/python2.7/site-packages/vdirsyncer/sync.py", line 163, in sync b_info.prepare_idents(storage_a.read_only) File "/home/fux/vdirsyncer/lib/python2.7/site-packages/vdirsyncer/sync.py", line 111, in prepare_idents if prefetch else ()): File "/home/fux/vdirsyncer/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 381, in get_multi if href != self._normalize_href(href): File "/home/fux/vdirsyncer/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 363, in _normalize_href return _normalize_href(self.session.url, _args, *_kwargs) File "/home/fux/vdirsyncer/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 27, in _normalize_href x = utils.compat.urlparse.urljoin(base, href) File "/usr/lib64/python2.7/urlparse.py", line 271, in urljoin params, query, fragment)) File "/usr/lib64/python2.7/urlparse.py", line 231, in urlunparse return urlunsplit((scheme, netloc, url, query, fragment)) File "/usr/lib64/python2.7/urlparse.py", line 242, in urlunsplit url = '//' + (netloc or '') + url UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 66: ordinal not in range(128)

but before at least, it was possible to download a list of .ics

[aitzkora]

may be it is due to my version of urllib, is it possible to use a downloaded version by virtualenv

[untitaker]

Please try again.

You can't upgrade urllib separately, it's part of Python. I think if you use Python 3 you have none of these problems.

[untitaker]

@aitzkora could you try again?

[aitzkora]

Victory! It works :-) When using ikhal i still have some errors in the console and strange name with some characters warning: This event will not be available in khal. Unknown exception happened. Traceback (most recent call last): File "/home/fux/vdirsyncer2/lib/python2.7/site-packages/khal/khalendar/khalendar.py", line 202, in _update_vevent self._dbtool.update(event.raw, href=href, etag=etag) File "/home/fux/vdirsyncer2/lib/python2.7/site-packages/khal/khalendar/backend.py", line 252, in update for vevent in sorted(vevents, key=sort_key): File "/home/fux/vdirsyncer2/lib/python2.7/site-packages/khal/khalendar/backend.py", line 236, in sort_key uid = str(vevent['UID']) UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 30: ordinal not in range(128) So I can close that issue ? Viele Dank! Thanks a million! marc

[geier]

Could you open an issue about this at khal? And please include the UID that causes theses problems.

[untitaker]

Excellent!