Date localization and text encoding errors

[hironemu]

Hello,

I found following error message in my environment.

[ERROR: alfred.workflow.action.script] execution error: Invalid date and time date 1/1/2000 of «script». (-30720)

Probably, I think that there is a problem in the following line.

https://github.com/idpaterson/alfred-wunderlist-workflow/blob/master/source/wunderlist/ui/lists.applescript#L274

set_value("listsUpdatedDate", date ("1/1/2000" as string) , "") of wf

My system settings (System Settings > Language & Region) is following:

It is working when I changed system setting following.

Could you support Japanese environment and others? Thanks

[idpaterson]

Thanks, that is a great find. Does the workflow fail as a result of that, or was it just something you noticed?

Since Yosemite completely broke the workflow, it is undergoing a major rewrite (see #49) that will soon be ready for testing. Would you be willing to help test the new version when it is available?

The new workflow uses the Python parsedatetime module to understand locale-aware natural language date formats. Unfortunately, it may be necessary for users to install an additional brew package and Python module in order to support locale-aware date formats. If we can streamline the process for that then dates can be entered in any regional format. For example Jan 2 would be represented as "1/2/2016" in the US, "2/1/2016" in the UK, and "2016/01/02" in Japan.

The parsedatetime module only natively supports English, Spanish, Dutch, German, and Australian date formats without the additional install of ICU and PyICU. While month names, date formats, and days of the week would be translated for all locales by ICU, only English, Dutch, and German support translated modifiers like "tomorrow" and "in 4 weeks". I just need to get all of this straight for documentation :smiley:

[hironemu]

Thank you for your response.

Does the workflow fail as a result of that, or was it just something you noticed?

The workflow was failed (I couldn't show list and couldn't add new task)

Would you be willing to help test the new version when it is available?

Yes, I will test it!

The new workflow uses the Python parsedatetime module to understand locale-aware natural language date formats. Unfortunately, it may be necessary for users to install an additional brew package and Python module in order to support locale-aware date formats. If we can streamline the process for that then dates can be entered in any regional format. For example Jan 2 would be represented as "1/2/2016" in the US, "2/1/2016" in the UK, and "2016/01/02" in Japan.

That's great! I will install additional package if it is available.

[idpaterson]

Great! I will update this issue in the next week or two when the new version is ready to test.

[hironemu]

I got it. Thank you!

[idpaterson]

Please download the alfredworkflow file attached to the first beta release.

As we discussed earlier, I do not think this will work out-of-the-box with your local date format. If it does, that's great! Otherwise, I will write up some instructions for installing the extra goodies that you need for local dates.

You can test with the command wl 1/2/2016 which should not but will probably show January 2, 2016 if it thinks you're in the US. Then try wl 2016/1/2 which should show January 2, 2016 if it gets your locale correct.

[hironemu]

Thank you for your great work!

I have found same problem #54 . I'll try it again when it is resolved.

Thanks!

[idpaterson]

@hironemu, go ahead and try again when you have time, the beta release was updated with a fix for that issue.

[hironemu]

@idpaterson

I have tried it and I was successfully authorized! Then I tried the following three patterns.

wl 2016/1/2

• Log

[INFO: alfred.workflow.input.scriptfilter] <?xml version="1.0" encoding="utf-8"?>
<items><item valid="yes"><title>New task in Inbox...</title><subtitle>2016/1/2</subtitle><arg> 2016/1/2</arg><icon>icons/dark/task.png</icon></item><item autocomplete=" : 2016/1/2" valid="no"><title>Change list</title><subtitle>Prefix the task, e.g. Automotive: 2016/1/2</subtitle><icon>icons/dark/list.png</icon></item><item autocomplete=" 2016/1/2 due " valid="no"><title>Set a due date</title><subtitle>"due" followed by any date-related phrase, e.g. due next Tuesday; due May 4</subtitle><icon>icons/dark/calendar.png</icon></item><item autocomplete=" 2016/1/2 every " valid="no"><title>Make it a recurring task</title><subtitle>"every" followed by a unit of time, e.g. every 2 months; every year; every 4w</subtitle><icon>icons/dark/recurrence.png</icon></item><item autocomplete=" 2016/1/2 *" valid="no"><title>Star</title><subtitle>End the task with * (asterisk)</subtitle><icon>icons/dark/star.png</icon></item><item autocomplete="" valid="no"><title>Main menu</title><subtitle /><icon>icons/dark/back.png</icon></item></items>

wl 1/2/2016

• Log

[ERROR: alfred.workflow.input.scriptfilter] XML Parse Error 'The operation couldn’t be completed. (NSXMLParserErrorDomain error 9.)'. Row (null), Col (null): 'xmlParseCharRef: invalid xmlChar value 55357' in XML:
<?xml version="1.0" encoding="utf-8"?>
<items><item valid="no"><title>New task in Inbox...</title><subtitle>&#55357;&#56517; Due Jan 02, 2016   Begin typing to add a new task</subtitle><arg> 1/2/2016</arg><icon>icons/dark/task.png</icon></item><item autocomplete=" : 1/2/2016" valid="no"><title>Change list</title><subtitle>Prefix the task, e.g. Automotive: </subtitle><icon>icons/dark/list.png</icon></item><item autocomplete=" due " valid="no"><title>Change the due date</title><subtitle>"due" followed by any date-related phrase, e.g. due next Tuesday; due May 4</subtitle><icon>icons/dark/calendar.png</icon></item><item autocomplete=" 1/2/2016 every " valid="no"><title>Make it a recurring task</title><subtitle>"every" followed by a unit of time, e.g. every 2 months; every year; every 4w</subtitle><icon>icons/dark/recurrence.png</icon></item><item autocomplete=" 1/2/2016 *" valid="no"><title>Star</title><subtitle>End the task with * (asterisk)</subtitle><icon>icons/dark/star.png</icon></item><item autocomplete="" valid="no"><title>Main menu</title><subtitle /><icon>icons/dark/back.png</icon></item></items>

wl test task next fri

This pattern has occured following error.

[ERROR: alfred.workflow.input.scriptfilter] XML Parse Error 'The operation couldn’t be completed. (NSXMLParserErrorDomain error 9.)'. Row (null), Col (null): 'xmlParseCharRef: invalid xmlChar value 55357' in XML:
<?xml version="1.0" encoding="utf-8"?>
<items><item valid="yes"><title>New task in Inbox...</title><subtitle>&#55357;&#56517; Due Fri, Mar 20   test task next</subtitle><arg> test task next fri</arg><icon>icons/dark/task.png</icon></item><item autocomplete=" : test task next fri" valid="no"><title>Change list</title><subtitle>Prefix the task, e.g. Automotive: test task next</subtitle><icon>icons/dark/list.png</icon></item><item autocomplete=" test task next due " valid="no"><title>Change the due date</title><subtitle>"due" followed by any date-related phrase, e.g. due next Tuesday; due May 4</subtitle><icon>icons/dark/calendar.png</icon></item><item autocomplete=" test task next fri every " valid="no"><title>Make it a recurring task</title><subtitle>"every" followed by a unit of time, e.g. every 2 months; every year; every 4w</subtitle><icon>icons/dark/recurrence.png</icon></item><item autocomplete=" test task next fri *" valid="no"><title>Star</title><subtitle>End the task with * (asterisk)</subtitle><icon>icons/dark/star.png</icon></item><item autocomplete="" valid="no"><title>Main menu</title><subtitle /><icon>icons/dark/back.png</icon></item></items>

[idpaterson]

So the character it is complaining about is: 📅

Do you see that character on this page as a red and white calendar icon? On my system that encodes to 📅 in the XML, whereas yours shows ��. I'll take a look at the encoding to see if there is a way to use the calendar character or if I should just avoid it.

Stay tuned, I'll let you know what I find. I'm going to rename this task to reflect the localization and encoding issues that it now represents.

[idpaterson]

Please download this build and try the following test commands:

1. wl:1
2. wl:2
3. wl:3
4. wl:4
5. wl:5
6. wl:6

The first test should match what is being done in the beta workflow, so it should fail. The others investigate possible fixes and test different characters. Let me know which of those error out and which, if any, show a result like this:

[hironemu]

Do you see that character on this page as a red and white calendar icon?

Yes, I do.

I have tried the test of wl:1 to wl:6 and captured screenshots and logs.

wl:1

[ERROR: alfred.workflow.input.scriptfilter] XML Parse Error 'The operation couldn’t be completed. (NSXMLParserErrorDomain error 9.)'. Row (null), Col (null): 'xmlParseCharRef: invalid xmlChar value 55357' in XML:
<?xml version="1.0" encoding="utf-8"?>
<items><item autocomplete="" valid="no"><title>Test 1</title><subtitle>&#55357;&#56517; due</subtitle><icon>icons/dark/back.png</icon></item></items>

wl:2

• same as wl:1

[ERROR: alfred.workflow.input.scriptfilter] XML Parse Error 'The operation couldn’t be completed. (NSXMLParserErrorDomain error 9.)'. Row (null), Col (null): 'xmlParseCharRef: invalid xmlChar value 55357' in XML:
<?xml version="1.0" encoding="utf-8"?>
<items><item autocomplete="" valid="no"><title>Test 2</title><subtitle>&#55357;&#56517; due</subtitle><icon>icons/dark/back.png</icon></item></items>

wl:3

[INFO: alfred.workflow.input.scriptfilter] <?xml version="1.0" encoding="utf-8"?>
<items><item autocomplete="" valid="no"><title>Test 3</title><subtitle>&#8635; due</subtitle><icon>icons/dark/back.png</icon></item></items>

wl:4

• same as wl:1

[ERROR: alfred.workflow.input.scriptfilter] XML Parse Error 'The operation couldn’t be completed. (NSXMLParserErrorDomain error 9.)'. Row (null), Col (null): 'xmlParseCharRef: invalid xmlChar value 55357' in XML:
<?xml version="1.0" encoding="utf-8"?>
<items><item autocomplete="" valid="no"><title>Test 4</title><subtitle>&#55357;&#56517; due</subtitle><icon>icons/dark/back.png</icon></item></items>

wl:5

• same as wl:1

[ERROR: alfred.workflow.input.scriptfilter] XML Parse Error 'The operation couldn’t be completed. (NSXMLParserErrorDomain error 9.)'. Row (null), Col (null): 'xmlParseCharRef: invalid xmlChar value 55357' in XML:
<?xml version="1.0" encoding="utf-8"?>
<items><item autocomplete="" valid="no"><title>Test 5</title><subtitle>&#55357;&#56517; due</subtitle><icon>icons/dark/back.png</icon></item></items>

wl:6

[INFO: alfred.workflow.input.scriptfilter] <?xml version="1.0" encoding="utf-8"?>
<items><item autocomplete="" valid="no"><title>Test 6</title><subtitle>&#8635; due</subtitle><icon>icons/dark/back.png</icon></item></items>

[idpaterson]

Thanks, I'll just avoid using that calendar icon.

[idpaterson]

Here are the instructions for installing PyICU. This may be something that I can automate in the workflow though I would certainly need an affirmative confirmation from the user before doing any global installs on their machine.

Install Homebrew

If you do not already have Homebrew, install it by pasting the following line in the terminal:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Use Homebrew to install ICU

Next, install the ICU C library

brew install icu4c

Install PyICU

Finally, install the python bindings for ICU. This command may require sudo depending on your system configuration. Feel free to use pip rather than easy_install if you have it set up.

sudo easy_install pyicu

Try it out!

When you add a task, the workflow should now automatically detect that PyICU is available and use it to better support your locale. That should in turn provide at least a few improvements to date parsing. Please give these a try and send a few examples of any that do not work if you would like me to troubleshoot.

1. Weekday names (e.g. miércoles)
2. Month names (e.g. junio)
3. Weekday and month abbreviations (e.g. lun, ene)
4. mm/dd/yyyy dd/mm/yyyy yyyy/mm/dd or other local variant

Note that some of the features will still only work in English. For example, repetition uses English words week, month, next, etc. for which PyICU would not provide translations.

[hironemu]

I have installed ice and pyicu. Then it have occurred following error.

$ sudo easy_install pyicu
Password:
Searching for pyicu
Reading http://pypi.python.org/simple/pyicu/
Best match: PyICU 1.8
Downloading https://pypi.python.org/packages/source/P/PyICU/PyICU-1.8.tar.gz#md5=00c8d40e5400f52c8474aa9480e8dbc1
Processing PyICU-1.8.tar.gz
Running PyICU-1.8/setup.py -q bdist_egg --dist-dir /tmp/easy_install-QuGxnX/PyICU-1.8/egg-dist-tmp-qZZ93j
In file included from _icu.cpp:27:
./common.h:86:10: fatal error: 'unicode/utypes.h' file not found
#include <unicode/utypes.h>
         ^
1 error generated.
error: Setup script exited with error: command 'cc' failed with exit status 1

But I have ran following command which is reported on following issue.

https://github.com/idpaterson/alfred-wunderlist-workflow/blob/develop/Wunderlist.alfredworkflow

$ sudo CFLAGS=-I/usr/local/opt/icu4c/include LDFLAGS=-L/usr/local/opt/icu4c/lib pip install pyicu

Then I downloaded the workflow from develop branch.

https://github.com/idpaterson/alfred-wunderlist-workflow/blob/develop/Wunderlist.alfredworkflow

I have test some commands and there are succeeded!!

• wl test next wed
• wl 部屋の掃除をする next fri
• wl test every fri
• wl test apr 14
• wl test 3/22

But I have found small issue. next was appended to added task.

And following date(yyyy/MM/dd) was not detected.(but I have not found any error message)

• wl test 2015/03/22

[STDERR: alfred.workflow.input.scriptfilter] 18:11:08 workflow.py:1972 DEBUG    Workflow version : 0.4.0-beta.2
18:11:08 workflow.py:1387 DEBUG    Reading settings from `/Users/hironemu/Library/Application Support/Alfred 2/Workflow Data/com.ipaterson.alfred.wunderlist/settings.json` ...
18:11:08 workflow.py:2233 DEBUG    Update check not due
18:11:08 workflow.py:2350 DEBUG    Got password : com.ipaterson.alfred.wunderlist:oauth_token
18:11:08 workflow.py:1496 DEBUG    No data stored for `prefs`
18:11:08 workflow.py:1511 DEBUG    Data `lists` stored in `cpickle` format
18:11:08 workflow.py:1526 DEBUG    Stored data loaded from : /Users/hironemu/Library/Application Support/Alfred 2/Workflow Data/com.ipaterson.alfred.wunderlist/lists.cpickle
18:11:08 workflow.py:2163 DEBUG    Set last run version : 0.4.0-beta.2
18:11:08 workflow.py:2008 DEBUG    Workflow finished in 0.304 seconds.
[INFO: alfred.workflow.input.scriptfilter] <?xml version="1.0" encoding="utf-8"?>
<items><item valid="yes"><title>Inbox &#8211; create a new task...</title><subtitle>test 2015/03/22</subtitle><arg>--stored-query</arg><icon>icons/dark/task.png</icon></item><item autocomplete=" : test 2015/03/22" valid="no"><title>Change list</title><subtitle>Prefix the task, e.g. Automotive: test 2015/03/22</subtitle><icon>icons/dark/list.png</icon></item><item autocomplete=" test 2015/03/22 due " valid="no"><title>Set a due date</title><subtitle>"due" followed by any date-related phrase, e.g. due next Tuesday; due May 4</subtitle><icon>icons/dark/calendar.png</icon></item><item autocomplete=" test 2015/03/22 every " valid="no"><title>Make it a recurring task</title><subtitle>"every" followed by a unit of time, e.g. every 2 months; every year; every 4w</subtitle><icon>icons/dark/recurrence.png</icon></item><item autocomplete=" test 2015/03/22 *" valid="no"><title>Star</title><subtitle>End the task with * (asterisk)</subtitle><icon>icons/dark/star.png</icon></item><item autocomplete="" valid="no"><title>Main menu</title><subtitle /><icon>icons/dark/back.png</icon></item></items>

[idpaterson]

Thank you, I knew there was another complication to the install. Thanks for posting the extra compiler flags, I had forgotten about that step.

I also noticed the problem with "next" followed by a day of the week and fixed it last night for inclusion in beta.3. Thanks for the details, here's more info if you're curious #58!

I don't think PyICU is working properly yet based on your observations. I remember there was an extra step required on my end to get it to work, maybe I forgot to do that. Once it is working it should actually allow you to use weekday names in your own language. Great details, I am very grateful for your help!

[idpaterson]

I was able to get my computer set up with the same locale as you and fix a bug in the workflow, but now parsedatetime is throwing some errors about non-ascii characters in its patterns, like 月 for month. Pretty cool that ICU has the word "month" translated, I should definitely use that for recurrence as well.

Anyway, it looks like it might take some more work to get this functional. I'll let you know when I have a preview build ready.

[idpaterson]

Since I don't know anything about Japanese, I think I've gotten somewhere but not very far. With a number of tweaks to parsedatetime the workflow is no longer crashing with non-ascii character errors and is parsing a few locale-specific dates. For example, 8月 works with parsedatetime to represent August but 八月 does not parse at all. Unfortunately, while 8月 works to represent August 1, 8月19日 is parsed as August 8, 2019 and 2015年8月19日 doesn't work at all.

Days of the week fare much better, 火曜日 is correctly interpreted as Tuesday (as is 火).

I also found that the parsedatetime code only supports mm/dd/yyyy or dd/mm/yyyy formats, it ignores yyyy/mm/dd so PyICU won't help here. The author has considered supporting it..

I think the most important observation so far is that if PyICU works properly it precludes the use of English text. This means you would be forced to use 火曜日 rather than Tuesday. If a few of the keywords like "next" and "last" still require English but the rest require Japanese this could be even more confusing.

A number of changes are needed to parsedatetime in order to support these things so I don't foresee this getting worked out before the 0.4 release of the workflow. It seems like you can work around it for now with US English dates, but it is important to get this working eventually.

[hironemu]

That's great! I never thought it was possible(8月 火曜日 and other pattern).

but 八月 does not parse at all

I think It's not problem. Because 八月 is rarely used at PC.

I also found that the parsedatetime code only supports mm/dd/yyyy or dd/mm/yyyy formats, it ignores yyyy/mm/dd so PyICU won't help here. The author has considered supporting it..

It is a bit unfortunate that we can not use the yyyy/mm/dd format. Because, in almost case, yyyy/mm/dd format is used in Japan.

If a few of the keywords like "next" and "last" still require English but the rest require Japanese this could be even more confusing.

It is a good that we can input next 火曜日. However, It is inconvenient that we can not input next Tuesday If only one of them will be done , I'd like to use next Tuesday. (Maybe most Japanese who are using such software understand "English day of week".)

[idpaterson]

That's all good news to me, thank you very much for your help. I will share your thoughts with the developer of parsedatetime to see how to proceed with those fixes. For the sake of the workflow I think that the best course of action for this release is to simply not instruct people to install PyICU.

You may remove it with the following:

wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py
sudo pip uninstall pyicu

[hironemu]

I got it. I will uninstall pyicu. Thanks!

[idpaterson]

@hironemu, here is an updated version of the test workflow that you tried last month. Can you please run through the wl:1, wl:2, wl:3, wl:4, wl:5, and wl:6 commands again to see whether they behave any differently with regard to those XML Parse Errors? If you haven't updated to Alfred 2.7 yet, please test both before and after updating. There were some changes in 2.7 and in the workflow that might possibly address this issue.

[hironemu]

@idpaterson

I have already updated Alfred 2.7.1. I have tested following pattern in Alfred 2.7.1. There are no parse error.

wl:1, wl:2, wl:4, wl:5

wl:3, wl:6

[idpaterson]

Excellent! Can you open up the test workflow in Alfred, double-click the wl Script Filter, then delete the first line highlighted below? I would like to see whether this is specifically what fixed the problem.

[hironemu]

@idpaterson I have removed the first line and tried wl:1 ~ wl:6. The results is same as https://github.com/idpaterson/alfred-wunderlist-workflow/issues/53#issuecomment-93331639

[idpaterson]

That's good! Since Alfred 2.7.1 essentially removed that line for other workflows I am glad to hear that it is not what fixed the XML error.

[idpaterson]

We have worked through most of the issues here, since the main remaining problem is that the yyyy/mm/dd date format is not supported I will open an issue for that on parsedatetime.