End position out after supplementary plane character in Windows

[woakesd]

This is an issue with Python under windows only. The attached sample illustrates the problem.

I presume this is because Windows python builds are narrow builds (16 bits per character and supplementary plane characters are stored as a supplementary pair).

I can add the 🙈 as a word and I get a match for it, though end is 9!

issue.txt

[WojciechMula]

Could you please attach output from the program, on my Linux machine it prints:

$ ~/github/pyahocorasick$ python3 bug_53.py 
(6, 'punched') punched
(16, 'punched') punched

[woakesd]

I get the following:

PS C:\Projects\pyahocorasick> py -3 .\bug_53.py (6, 'punched') punched (17, 'punched') unched?

I'm pretty sure this is because in windows the string passed in is two byte characters, and the higher plane characters are sent as complementary pairs (two two byte characters).

[WojciechMula]

@woakesd I will check it tomorrow, when download MSVC build toolchain. I have just MSVC 2008, Py3k needs the newest one.

[woakesd]

You can use VS2015 community edition.

Good luck.

On 17 Jan 2017, at 18:31, Wojciech Muła notifications@github.com wrote:

@woakesd I will check it tomorrow, when download MSVC build toolchain. I have just MSVC 2008, Py3k needs the newest one.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[WojciechMula]

@woakesd OK, I've just reproduced the error, now need to work out a solution.

[WojciechMula]

@woakesd I pushed the fix to the master. Could you please check it?

[woakesd]

Tested this and the simple test passes, which is good.

But on a larger test with 50000 strings in the Automaton I get ValueError exceptions thrown randomly when searching a series of messages. I'm repeating the test and it fails on a different message each time.

Example:

Malformed UCS-2 string: expected a high surrogate at 1662, got dd00 (the position and code change on each run).

I'm struggling to produce a simple test which demonstrates this though. Grr.

Initially I was suspicious of my data but after running it repeatedly it fails in different places.

[woakesd]

I don't really want to post my test data here, can I send them to you privately?

[WojciechMula]

Thank you very much for testing! Please send your test at wojciech_mula@poczta.onet.pl Nothing would leak, don't worry.

[woakesd]

Thank you! Will check out the changes later tonight or tomorrow perhaps. It may need to wait till Monday.

[woakesd]

This is fixed as far as I am concerned so I am closing it.

[woakesd]

I'm putting together another regression test for this as I found an issue which I think may be related

[woakesd]

PS I found a work around which is to add one space for each character in the supplementary plane.

haystack += ' ' * len([c for c in haystack if ord(c) > 65535])

[WojciechMula]

Sorry for the problem, will check it soon.

[pombredanne]

@woakesd Is this still an issue? I think this was fixed, but I am not 100% sure... would you be kind enough to check on the latest checkout?

[woakesd]

I think not, but I’ll need to check.

David

📞 0131 332 7571 📱 07966 444 615 📧 @.*** 💻 www.woakes.net

On 20 Feb 2022, at 11:09, Philippe Ombredanne @.***> wrote:

 @woakesd Is this still an issue? I think this was fixed, but I am not 100% sure... would you be kind enough to check on the latest checkout?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[pombredanne]

@woakesd I am closing this for now: we have windows tests that pass alright. Please reopen if this is still an issue