Fully automated test run

[danielbrauer]

I've barely started using faketorio, but it looks like it always requires manual interaction in order to run tests. Is this true? It would be really nice to have it start Factorio, run the tests, and then exit with a report and appropriate exit code.

[JonasJurczok]

Hey, thanks for trying it out :)

Yes it always requires interaction. With faketorio test you can start factorio and launch a new map. However actually running the tests is a manual step for multiple reasons:

• you might need setup first (place some buildings or something)
• there is no clean way to end the game from a mod. You can only crash the game but then you still have to close factorio manually
• while technically possible observing factorio from the outside is complicated to do and I don't see an actual value. Unless you want to store the test reports somewhere, of course :)

Do you have a specific use case that is cumbersome due to this behaviour?

[danielbrauer]

Setting things up manually is always cumbersome. :D My use case is that I'd like to be able to execute one command and have all my faketorio tests run, just like with busted.

With that in mind, to your first point I would say that any setup should be done in a save file (https://github.com/JonasJurczok/faketorio/issues/78)

We should ask Wube to add API for terminating Factorio. Perhaps this should be provided through a command line argument so as not to give the scripting API too much weird power?

Once you're outputting to a file, it's probably easier to print results on the command line than to create a custom UI for them, no?

[danielbrauer]

Factorio does handle SIGINT properly, so you can ask it to shut down externally once results are finished

[danielbrauer]

I just found the command line --until-tick, which seems to run Factorio without any graphics until the specified tick and then terminate. This might not be useful for UI testing (I'm not actually sure), but it would be ideal for executing a set of tests and then exiting.

[JonasJurczok]

Yeah.. --until-tick is headless and there is no UI rendering done in it. So you cannot reliably run tests with it (and you would have to know beforehand how many ticks your tests are actually running).

The only solution that I really see is outputting to a file, reading that file in parallel and then terminating Factorio from the outside.

But this breaks the continious testing dev process in #81 :)

So we would have to offer both. I'll add it to the list of things I want to do :)

[danielbrauer]

I don't have any UI to test, so headless would be great for me. I guess it probably doesn't load any graphics either, but presumably still does all the data stages.

[JonasJurczok]

Yes it does the data stages and spawns a map but does not connect a player to it. This means ingame no time actually passes and there is nothing to run the tests from :/

I asked for a real headless mode a while ago in the forums but the devs said there is no reason for them to build and support that at that moment. :/

[danielbrauer]

I'm pretty sure that time does pass as part of the simulation, even if it's headless.

I can see the lack of a player being limiting, but there's still lots of things that could be tested (including whether a mod works without any players).

[JonasJurczok]

According to https://wiki.factorio.com/Multiplayer#Dedicated.2FHeadless_server the game is paused if there is no player connected to it. This is true for the dedicated headless server and if you run factorio with the --start-server option.

I don't know how the --until-tick option works (obviously time passes) but then the problem is the same that you have to know in advance how long your tests will be running.

[danielbrauer]

So even with all these limitations I think I would really like the following:

• I specify a set of scenarios
  • a scenario can optionally specify an existing saved game
  • a scenario needs to specify how many ticks the test will take
  • the code in a scenario can be a coroutine (https://www.lua.org/pil/9.1.html)
• faketorio runs each scenario using --until_tick
  • it uses a copy of the designated saved game
  • it runs test code as a coroutine from an on_tick event handler
  • it outputs results to a log file
• once all scenarios have run
  • the saved game copies are purged
  • the logs are parsed, then purged
  • a report is printed
  • appropriate exit code

I need to think about this some more but I feel like it could work.

[danielbrauer]

Ah, coroutine is out: https://lua-api.factorio.com/latest/Libraries.html

):

[JonasJurczok]

Sorry for the late reply. I was not in the country..

Yeah. The current design is roughly the same, except that it all runs inside of one factorio instance.

You start the game (and keep it running), in every test run you run all your scenarios and can use the before/after test methods to do setup and cleanup (place entities etc.).

I still really like the alternative of a headless, quicker execution but would have to think about that :)