Add bitwise operators

[ghost]

Bitwise operators are a feature in Lua 5.3, and it would be nice to have them since your already implementing Lua 5.2 features anyway. The reason I want this is because of a project I'm making including binary files, and I need to swap endianness and stuff.

[SquidDev]

Worth noting they're actually a 5.3 feature. We'll get to them eventually. Maybe.

[Zirunis]

Hi, I have been missing these for mainly one reason several times now over the years: It's almost impossible to create fast but good hash functions without them. I'd be willing to look into this and help given some guidance. Is this still of interest? @SquidDev

[SquidDev]

In theory yes, in practice it's complicated. I think it makes sense to add bitwise operators as part of a wider update to Lua 5.3 (and its addition of a separate integer type), but it's not clear to me if that can be done in a backwards compatible way.

[Zirunis]

Ah, I forgot about the "it's all floats"-thing. Yeah that would probably not make much sense without an integer type. Can you still give me some of the problems with making it backwards-compatible? So that I can sleep on it a bit? Maybe we can at least come up with some ideas or a rough plan or something. So you know if and where to put it on the long-term dream roadmap

[SquidDev]

The biggest issue is that integer types will now overflow, rather than continuing on (like doubles do). For instance, under Lua 5.2:

> local x = 2 for i = 1, 66 do x = x * 2 end print(x)
1.4757395258968e+20

And under Lua 5.3:

> local x = 2 for i = 1, 66 do x = x * 2 end print(x)
0

Argument validation is now a bit stricter, and prevents floats where integers are required:

> string.rep("x", 2.3)
stdin:1: bad argument #2 to 'rep' (number has no integer representation)

[Zirunis]

Thank you! I thought about this for a little and it seems like backwards compatibility is pretty much impossible for integers. The only options I see would be to

1. make integers explicitly typed, which would make code using this feature incompatible to any lua version out there which is probably not at all what we want.
2. make integers accessible via some global api, which is less ugly than 1 imo but suffers pretty much the same issues.
3. allow the program to switch to "integer-mode" for some scope. Ideally just the file, but I'd assume that would be pretty difficult in practice because of imports? But even this approach while having backwards-compatibility doesn't really have compatibility with lua outside of cc:tweaked either because the functions/api to switch modes don't exist anywhere else. Maybe I've missed some better option?

The bigger question for me is: Does anybody really need the backwards-compatibility? I doubt many (if any) programs will run into the overflow issue. The argument validation being stricter will definitely break some programs but it seems to me that it would be quite easy to patch the affected programs. It would probably even be possible to write a script to do it automatically (even though doing it manually would probably yield cleaner code). Alternatively, couldn't we also overwrite the argument validation to be as strict as before (rounding values if needed)?

[Zirunis]

During my research I also encountered bit32 which is shipped in the standard library of lua 5.2 and provides bit-operation functions directly on the IEEE 754 floating point numbers. A backport to lua 5.1 exists as well. For my personal needs this would completely suffice, since these are as fast as you'd expect bit manipulations to be. Obviously the syntax is a little more verbose and switching between floating point numbers and their bitwise representations is a little awkward if one expects integer representations but who cares? People who need the bitwise operators and find the bit32 api will do so via the documentation anyways. I obviously understand the goal of doing it "the proper way" and going straight to 5.3 but this would be a perfectly fine temporary (even permanently temporary ;)) solution until we're there. And looking ahead, backwards compatibility also shouldn't be a problem as one could simply rebind the bit32 api to call the new bit-manipulation functions. What are your thoughts on this?

[SquidDev]

Oh, CC already supports the bit32 API!

[Zirunis]

What the heck? Well that's awesome! But how can I contribute to the docs, because this is getting a little bit out of hand. I've recently had a headache for days because of update file modes not being supported just to find out the docs simply weren't updated and now the same happened for bitwise operators :D I'm not blaming you, thankful for any second you spend on this project! But still a little frustrating so I'd like to do my part at least.

[Wojbie]

It is documented along with all other native lua backports and supports on https://tweaked.cc/reference/feature_compat.html

Otherwise its all native lua 5.2 https://www.lua.org/manual/5.2/

[SquidDev]

But how can I contribute to the docs

The docs are just stored alongside the source code, so doc changes can be done as a normal PR.

As far as documenting built-in Lua definitions goes, it's a bit trickier. Currently we don't document these at all, as they should just be the same as normal Lua. Maybe it would be nice to have it all in one place, but not sure how that should look yet.

update file modes not being supported

Oh, what version of the mod are you on? That should work as of CC:T 1.109

[Zirunis]

It is documented along with all other native lua backports and supports on https://tweaked.cc/reference/feature_compat.html

Ah, fair. Normally I always have this page open anyways, but for this one my brain was not braining. Since I only found out about bit32 trying to solve this issue I had never actually looked for it in CC:Tweaked. Before posting there was this 1 second of hesitation ("Maybe it's already supported?") but then I just checked the globals, didn't find it there and moved on. Looked in the wrong place obviously, my bad.

The docs are just stored alongside the source code, so doc changes can be done as a normal PR.

Then I'll try that for the file modes (as these are documented separately under io.open) as soon as I'm sure they actually work now.

As far as documenting built-in Lua definitions goes, it's a bit trickier. Currently we don't document these at all, as they should just be the same as normal Lua. Maybe it would be nice to have it all in one place, but not sure how that should look yet.

I personally don't have an issue looking at lua documentation as well. Some things are doubled like the file modes as they are provided via io but that also makes sense to me. If I were to miss anything lua specific I think it would be Cobalt implementation details. I'm just assuming it to be PUC lua unless stated otherwise but I think some performance differences would be interesting (if there are any relevant ones). What I mean is that when optimizing code for performance or memory (which I'm doing a lot at the moment for a project I've been thinking about for years) I do so following assumptions such as "string creation expensive", "concatenation kind of expensive" (have read your lovely blog, so I know about ropes), "tables allocate no space by default so double quite a few times when initially filling them up" and so on. As long as these hold true, everything is fine. But if it turns out for example that in Cobalt tables actually start with size 4, that would change some things. I would no longer have to initialize small tables with temporary values anymore and on the other hand would have to worry about not creating an insane amount of 1-to-3 element tables (however that would ever even come up). TL;DR: If Cobalt doesn't perform vastly different from PUC lua, I'm quite happy with the general state of documentation. Otherwise there might be some potential for a new doc page.

Oh, what version of the mod are you on? That should work as of CC:T 1.109

I was on 1.16.x but switched to 1.20.5 now for this feature. I'm on CC:T 1.110.3 (an alpha version I believe?) and can't get it to work.

local function testModeRPlusB(filename)
    -- Open file in "r+b" mode
    local file, err = io.open(filename, "r+b")
    if not file then
        print("Failed to open file in mode r+b: " .. err)
        return
    end

    print("Successfully opened file in mode r+b")

    -- Seek to the end of the file and read the last byte
    file:seek("end", -1)
    local last_byte = file:read(1)
    print(last_byte)

    -- Modify the last byte
    local modified_byte = "!"
    print(modified_byte)

    -- Write the modified byte back to the last position
    file:seek("end", -1)  --probably unnecessary?
    file:write(modified_byte)

    -- Flush shouldn't be needed because close should flush as well, but just in case
    file:flush()
    file:close()
end

local test_filename = "test_file.txt"
local test_file = io.open(test_filename, "wb")
test_file:write("Hello World?")
test_file:close()

testModeRPlusB(test_filename)

Running this little test script reading works fine but writing doesn't seem to? I'd expect the final files' content to be "Hello World!" but it is instead still "Hello World?" I have never worked with these file modes before though and have never used seek before so maybe I'm brainfarting something here.

[Wojbie]

Oh thank you kindly for this report. This seems to be issue with io shim lib error cause if i run this code modified for fs lib (one that is actually interacting with files) it works fine:

local function testModeRPlusB(filename)
    -- Open file in "r+b" mode
    local file, err = fs.open(filename, "r+b")
    if not file then
        print("Failed to open file in mode r+b: " .. err)
        return
    end

    print("Successfully opened file in mode r+b")

    -- Seek to the end of the file and read the last byte
    file.seek("end", -1)
    local last_byte = file.read(1)
    print(last_byte)

    -- Modify the last byte
    local modified_byte = "!"
    print(modified_byte)

    -- Write the modified byte back to the last position
    file.seek("end", -1)  --probably unnecessary?
    file.write(modified_byte)

    -- Flush shouldn't be needed because close should flush as well, but just in case
    file.flush()
    file.close()
end

local test_filename = "test_file.txt"
local test_file = fs.open(test_filename, "wb")
test_file.write("Hello World?")
test_file.close()

testModeRPlusB(test_filename)

local test_file = fs.open(test_filename, "rb")
print(test_file.readAll())
test_file.close()

EDIT: I just checked CC:T docs. According to them io lib don't seem to support + file mode. Only fs does. You can even see part of io code that strips + from file mode. https://github.com/cc-tweaked/CC-Tweaked/blob/f63f85921fb7ef27175df7bfe2f3c3d93475493b/projects/core/src/main/resources/data/computercraft/lua/rom/apis/io.lua#L384

[Zirunis]

God damnit, so it's all me again! And that means the documentation is correct too! Maybe I'm being stupid here, but why is there even this redundancy between io and fs? Is this difference in file mode support intended? I'm sorry, this is all pretty off-topic, happy to move this elsewhere.

[SquidDev]

So while the docs are correct, this is definitely a bug on CC:T's part! The io library should support r+/w+ too! I've pushed a fix to this in d48b85d50cdb30af941c0dc4eccbb8971e902011.

but why is there even this redundancy between io and fs?

io is part of standard Lua, so it's nice to have for compatibility reasons. However, fs offers a lot more functionality not present in the normal io library. I normally recommend using fs for most CC programs.

[Zirunis]

I see, thank you!

io is part of standard Lua, so it's nice to have for compatibility reasons. However, fs offers a lot more functionality not present in the normal io library. I normally recommend using fs for most CC programs.

I think this would be a helpful tip that should be written somewhere. Maybe in a "How to use the filesystem guide"? Especially now with the update mode there is quite a bit to it. Might include some notes about how to work with binary data in there as well. I've done quite a bit with that recently so I could write up some groundwork if you'd like.