input function throws: LookupException: unknown encoding: utf_8

[eugenelepekhin]

Here is how I create an engine: Encoding encoding = Encoding.Default; ScriptRuntime runtime = Python.CreateRuntime(); runtime.IO.SetOutput(this.stream, encoding); runtime.IO.SetInput(this.stream, encoding); runtime.IO.SetErrorOutput(this.stream, encoding); this.scriptEngine = Python.GetEngine(runtime);

So, the python text: input("name: ")

will throw. Here is the exception stack:

at IronPython.Runtime.Operations.PythonOps.LookupEncoding(CodeContext context, String encoding) at IronPython.Modules.PythonIOModule.TextIOWrapper.GetDecoder(CodeContext context) at IronPython.Modules.PythonIOModule.TextIOWrapper.readline(CodeContext context, Int32 limit) at Microsoft.Scripting.Interpreter.FuncCallInstruction4.Run(InterpretedFrame frame) at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame) at Microsoft.Scripting.Interpreter.LightLambda.Run4[T0,T1,T2,T3,TRet](T0 arg0, T1 arg1, T2 arg2, T3 arg3) at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2) in /_/src/libraries/System.Linq.Expressions/src/System/Dynamic/UpdateDelegates.Generated.cs:line 434 at Microsoft.Scripting.Interpreter.FuncCallInstruction6.Run(InterpretedFrame frame) at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame) at Microsoft.Scripting.Interpreter.LightLambda.Run4[T0,T1,T2,T3,TRet](T0 arg0, T1 arg1, T2 arg2, T3 arg3) at IronPython.Runtime.PythonContext.CallSplat(CodeContext context, Object func, Object[] args) at IronPython.Runtime.Operations.PythonCalls.Call(CodeContext context, Object func, Object[] args) at IronPython.Runtime.Operations.PythonOps.Invoke(CodeContext context, Object target, String name, Object[] args) at IronPython.Runtime.Operations.PythonOps.ReadLine(CodeContext context, Object f) at IronPython.Runtime.Operations.PythonOps.ReadLineFromSrc(CodeContext context, Object src) at IronPython.Modules.Builtin.input(CodeContext context, Object prompt) at Microsoft.Scripting.Interpreter.FuncCallInstruction3.Run(InterpretedFrame frame) at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame) at Microsoft.Scripting.Interpreter.LightLambda.Run4[T0,T1,T2,T3,TRet](T0 arg0, T1 arg1, T2 arg2, T3 arg3) at System.Dynamic.UpdateDelegates.UpdateAndExecute3[T0,T1,T2,TRet](CallSite site, T0 arg0, T1 arg1, T2 arg2) in /_/src/libraries/System.Linq.Expressions/src/System/Dynamic/UpdateDelegates.Generated.cs:line 434 at Microsoft.Scripting.Interpreter.DynamicInstruction4.Run(InterpretedFrame frame) at Microsoft.Scripting.Interpreter.Interpreter.Run(InterpretedFrame frame) at Microsoft.Scripting.Interpreter.LightLambda.Run2[T0,T1,TRet](T0 arg0, T1 arg1) at IronPython.Compiler.PythonScriptCode.RunWorker(CodeContext ctx) at IronPython.Compiler.PythonScriptCode.Run(Scope scope) at IronPython.Compiler.RuntimeScriptCode.InvokeTarget(Scope scope) at IronPython.Compiler.RuntimeScriptCode.Run(Scope scope) at Microsoft.Scripting.SourceUnit.Execute(Scope scope, ErrorSink errorSink) at Microsoft.Scripting.SourceUnit.Execute(Scope scope) at Microsoft.Scripting.Hosting.ScriptSource.Execute(ScriptScope scope) at Microsoft.Scripting.Hosting.ScriptEngine.Execute(String expression, ScriptScope scope) at LogicCircuit.IronPythonConsole.<>c__DisplayClass26_0.g__run|0() in C:\Projects\LogicCircuit\Sources\LogicCircuit\Dialog\IronPythonConsole.xaml.cs:line 159

Originally posted by @eugenelepekhin in https://github.com/IronLanguages/ironpython3/issues/1625#issuecomment-1925570783

[slozier]

It looks like input is using the encodings module from the standard library so you'll need to include the standard library in your search paths, for example:

var paths = scriptEngine.GetSearchPaths();
paths.Add(@"C:\Program Files\IronPython 3.4\Lib");
scriptEngine.SetSearchPaths(paths);

Although ideally the encoding lookup would be able to find a .NET based decoder instead of relying on the encodings module. The open method has a similar issue. Related to https://github.com/IronLanguages/ironpython2/issues/659

[eugenelepekhin]

In my case it's embedded engine so, there is no separate Python installation exists.

[slozier]

I see, you're running entirely without a standard library. Python has been moving more of its core functionality to the standard library making it more difficult to run without it. CPython also fails to run anything without the encodings module.

Although I understand that this is not ideal, as a workaround you could try to package a minimal set of standard library required to enable the input functionality along with your application. For example, an IpyLib.zip file containing:

codecs.py
encodings/__init__.py
encodings/aliases.py
encodings/utf_8.py

and then you can include it:

var paths = scriptEngine.GetSearchPaths();
paths.Add(@"<path to app>\IpyLib.zip");
scriptEngine.SetSearchPaths(paths);

[eugenelepekhin]

Well, I am not sure what users will be doing with Python in my application, so fixing just input function might be not enough. There is IronPython.Modules.dll that is coming with Python nuget package. I'd expect it to be the standard library or any other that is just a member of the package.

[slozier]

IronPython.Modules.dll only contains the built-in modules (the ones that are not implemented in Python files). For example nt, _datetime, _socket are built-in modules, but users will typically interact with os, datetime and socket which are .py files in the standard library.

The standard library (Python files) is published as a separate nuget package IronPython.StdLib. I had assumed that your app was not including it because you didn't need all the bells and whistles, but if you think you want it all then I'd suggest packaging it as a zip file alongside your exe.

[eugenelepekhin]

This is awesome, thank you it's finally working.