UTF8 support

[JekSun97]

Is your feature request related to a problem? Please describe. My game has modding support, and support for many languages, but in the lua code you can only get a string in English, which makes it impossible to make modifications for other languages, such as Japanese, or Russian.

Describe the solution you'd like I'm not a pro in lua, but as far as I know, there is a utf8 library for lua, you need it, since godot adheres to multilingualism

Describe alternatives you've considered There are no alternatives, except to add the utf8 library itself

Additional context If you can implement this, I ask you to add this feature to the 3.5+ branch, I'm even ready to pay future thanks, my project is very large, and has been developed for 3.5.2 for a long time

[Trey2k]

So this is a bit complicated, lua at its core was designed for ascii. There is a utf8 library which is available in v2.0+ of this module. But it only supplies some helper functions to manipulate them from lua. In order to allow passing of UTF8 strings to and from lua a lot will need to change in the way we handle string in the background. It should be possible but this will take some time. Focus will be given to the current 2.1 beta, however since you mentioned your project is on going and utilizing the v1 version of the module, if a solution is found it can be back ported.

I dont have a ton of time atm so it may be a bit before I can get to this, my apologies.

[JekSun97]

Thanks, I'll wait! Good luck with development!

[Trey2k]

Some useful information can be found here regarding this issue. I hope to look into this over the weekend. Will post any updates here.

[RadiantUwU]

@Trey2k What version of lua does the 1.x module run?

[Trey2k]

@Trey2k What version of lua does the 1.x module run?

5.4

[GokuHiki]

I test the code with utf8:

extends Node

var lua: LuaAPI = LuaAPI.new()

func _lua_print(message: String) -> void:
    print(message)

# Called when the node enters the scene tree for the first time.
func _ready():
    lua.push_variant("print", _lua_print)
    lua.bind_libraries(["base", "table", "string", "io", "os", "utf8"])

    var err: LuaError = lua.do_string("""
    print("こんにちは")
    print("Xin chào")
    function get_message()
        return "你好"
    end
    """)
    if err is LuaError:
        print("ERROR %d: %s" % [err.type, err.message])
        return

    var val = lua.pull_variant("get_message")
    if val is LuaError:
        print("ERROR %d: %s" % [val.type, val.message])
        return

    var message = val.call([])
    print(message)

# Called every frame. 'delta' is the elapsed time since the previous frame.
func _process(delta):
    pass

And here are the errors:

  Unicode parsing error: Invalid unicode codepoint (3053), cannot represent as ASCII/Latin-1
  Unicode parsing error: Invalid unicode codepoint (3093), cannot represent as ASCII/Latin-1
  Unicode parsing error: Invalid unicode codepoint (306b), cannot represent as ASCII/Latin-1
  Unicode parsing error: Invalid unicode codepoint (3061), cannot represent as ASCII/Latin-1
  Unicode parsing error: Invalid unicode codepoint (306f), cannot represent as ASCII/Latin-1
  Unicode parsing error: Invalid unicode codepoint (e0), cannot represent as ASCII/Latin-1
  Unicode parsing error: Invalid unicode codepoint (4f60), cannot represent as ASCII/Latin-1
  Unicode parsing error: Invalid unicode codepoint (597d), cannot represent as ASCII/Latin-1

I think there should not be any problems at Lua side because Lua only sees string as an array of bytes, it doesn't care about encoding of string at all! And error happened when Godot LuaAPI at IO bound connect with lua don't specify any encoding convert bytes<->string but use default "ASCII/Latin-1": An array of bytes utf8 "こんにちは" can not match to ASCII/Latin-1 encoding. Well... it just my amateur through.

Thanks for your Godot LuaAPI. I hope that you can fix that bugs because I want do dev my next game on Godot with full lua support for multi languages!

[Trey2k]

I think there should not be any problems at Lua side because Lua only sees string as an array of bytes, it doesn't care about encoding of string at all! And error happened when Godot LuaAPI at IO bound connect with lua don't specify any encoding convert bytes<->string but use default "ASCII/Latin-1": An array of bytes utf8 "こんにちは" can not match to ASCII/Latin-1 encoding. Well... it just my amateur through.!

This is correct, the issue here is with the wrapper logic for passing and retrieving string from Lua. It currently assumes its an ascii string. I should hopefully have a fix fk4 rhia issue soon. And thank you for the information

[Trey2k]

UTF8 support is included for godot 3.x here https://github.com/WeaselGames/godot_luaAPI/releases/tag/v1.2-beta1

[JekSun97]

Thank you very much! This is a holiday for my project! Everything is working!