search

IronLanguages / main

Work for this repo has moved to https://github.com/IronLanguages/ironpython2
1.16k stars 347 forks source link

Issue with non-unicode script file encoding #407

Open ironpythonbot opened 9 years ago

ironpythonbot commented 9 years ago

--------------------------------------------------------------------------

DLR VERSION AFFECTED: 0.9
BUILD TYPE: Signed Binaries
OPERATING SYSTEM: XP SP3
CLR VERSION: .NET 2.0 SP1
SCENARIOS AFFECTED: non-unicode script files with non-western characters

--------------------------------------------------------------------------

BRIEF DESCRIPTION:
If I load script
ScriptSource source = _pyEng.CreateScriptSourceFromFile(filePath, Encoding.GetEncoding(1251));
all cyrillic characters in script corrupted.

Workaround works fine:
ScriptSource source;
using (var sr = new StreamReader(filePath, Encoding.GetEncoding(1251), true))
source = _pyEng.CreateScriptSourceFromString(sr.ReadToEnd(), filePath, SourceCodeKind.File);
}

Work Item Details

Original CodePlex Issue: Issue 20472 Status: Active Reason Closed: Unassigned Assigned to: Unassigned Reported on: Jan 6, 2009 at 10:26 PM Reported by: dinov Updated on: Feb 22, 2013 at 2:14 AM Updated by: jdhardy Test: EngineTest.cs (ScenarioCodePlex20472)

ironpythonbot commented 9 years ago

On 2009-01-07 06:27:05 UTC, dinov commented:

Ported from http://www.codeplex.com/dlr/WorkItem/View.aspx?WorkItemId=1112

ironpythonbot commented 9 years ago

On 2009-05-22 03:36:47 UTC, dinov commented:

This is now fixed in that we will report a syntax error if this happens instead of silenetly corrupting data. Python requires that you have a coding header to use non-ASCII characters, see PEP-263:

http://www.python.org/dev/peps/pep-0263/

ironpythonbot commented 9 years ago

On 2010-02-12 03:15:08 UTC, dfugate commented:

No dev-added regression I could find, and this is broken in that we do not actually report a syntax error in the case mentioned above. Reopening.

slozier commented 8 years ago

This is still somewhat buggy. The test case (IronPythonTest.EngineTest.ScenarioCodePlex20472) needs to be tweaked. To get an exception it needs to call GetCode on the ScriptSource , but the exception appears to be of type IronPython.Runtime.BadSourceException which is probably not what we want...