search

rce-incorporated / Fiu

Luau bytecode interpreter for Luau
MIT License
90 stars 16 forks source link

New Test Suite Workflow #13

Closed SnorlaxAssist closed 7 months ago

SnorlaxAssist commented 8 months ago

New update to Test Suite Workflow https://github.com/rce-incorporated/Fiu/pull/13#issuecomment-1998754274

OLD

A new test suite that uses GitHub actions to run test files while also comparing current tests.

A GitHub Action test job: https://github.com/SnorlaxAssist/Fiu/actions/runs/7984464250/job/21801314981

TestRunner.lua

This has built in features such as static configuration in Config.lua, script has arguments for micro testing.

Design

Made to easily test pull requests and commits to view passing/failing builds with a readable test log.

image

Failing Cases

Currently there is quite of lot of failing cases, and majority of them being an issue with Fiu directly or indirectly(global), on local testing issues not caught before is more visible now.

TheGreatSageEqualToHeaven commented 7 months ago

I've been busy moving so I didn't have time to take a look at any pull requests. I'm not completely sure how github actions or lune works but we should prefer writing the test suite in C++ to properly take advantage of the compiler and vm. I don't think lune allows for us to setup vectorCtor and other requirements that future tests will require. In 1-2 weeks I'll probably be available to take a look at this.

SnorlaxAssist commented 7 months ago

I agree, that actually would be more accurate. Lune is rust based, a runtime similar to nodejs(but somewhat smaller).

Edit 1: I only used a switched to lune because I saw you were using regular lua test case, thus the new test system is in luau.

Edit 2: I would assume C++ test case would be based on luau's own main test

SnorlaxAssist commented 7 months ago

I checked vectorCtor in luau source, I'm not sure what use it has for FIU, since FIU is always interpreted. The only use I seen for vectorCtor is for Luau's CodeGen, for standard interpreted compile, it is converted into GETIMPORT.

CodeGen: Such that vectorCtor = "vector" & vectorLib = "vector" would use Vector based CPU Instructions. VM: Such that vectorCtor = "new" & vectorLib = "Vector3" would be GETIMPORT [Vector3.new] in bytecode.

Ref:

TheGreatSageEqualToHeaven commented 7 months ago

I checked vectorCtor in luau source, I'm not sure what use it has for FIU, since FIU is always interpreted. The only use I seen for vectorCtor is for Luau's CodeGen, for standard interpreted compile, it is converted into GETIMPORT.

Thankfully the actual interpreter Fiu has never needs to interact with vectorCtor however our deserializer needs to accept vectorCtor if we want to accept vector constants. Which goes back to also why having a C++ test suite would make life easier since we can both compile tests with our own ctor and also run the test with it.

https://github.com/luau-lang/luau/blob/209fd506c9017b9e252408834811c8f5c5529158/tests/Compiler.test.cpp#L4485-L4490

SnorlaxAssist commented 7 months ago

Wouldn't that be the constant vector, it should be a simple thing to add, Luau to my knowledge has a most of 4 axis vector, x, y, z, w, I have made those changes in my local Fiu version that has those Vector support.

Edit: You are right though, we would need a vectorCtor test so that we can generate Vector constants in the Fiu deserializer process.

Edit2: I could make an issue request on lune to add vectorCtor and vectorLib support in Luau Compile options.

TheGreatSageEqualToHeaven commented 7 months ago

Wouldn't that be the constant vector, it should be a simple thing to add, Luau to my knowledge has a most of 4 axis vector, x, y, z, w, I have made those changes in my local Fiu version that has those Vector support.

While we could substitute a table for it we're assuming Fiu is not always running on Roblox intentionally, which is why we need to accept vectorCtor ourselves to deserialize.

I could make an issue request on lune to add vectorCtor and vectorLib support in Luau Compile options.

I'd rather be able to directly adjust settings through C++ rather than having to rely on a Luau runner, we can more reliably plug our tests in that way.

SnorlaxAssist commented 7 months ago

Understandable, I do like the idea of custom runtime provider for these tests.

SnorlaxAssist commented 7 months ago

New Test Suite Workflow migrated from lune runtime with luau test runner to custom test runner written in C++.

fiu-tests

A compiled executable that runs test cases for Fiu with arguments for adjustment and advanced testing.

An example of how the executable is used:

Test all cases with all default options.

fiu-tests

with arguments

fiu-tests -t Conformance/Instructions/Logic -log ALL -O2 -D1 -vectorLib Vector3 -vectorCtor new

build serializer tests

fiu-tests mdt -bSrc Conformance/Instructions -bOut Conformance/Deserializer/Instructions -O2 -D2

TODO