wc3 crashes if too much data is migrated from compiletime

[rhazarian]

As far as I've understood, Reforged crashes on map load if a thread launched at map init crashes. I have thousand of lines of code which initializes objects in compiletime (e.g. HashMap's with game data SLKs). It seems that I've reached a limit. My map crashes. When I remove some of this code, the map launches. When I add some more code that initializes data that will be migrated from compiletime - my map crashes. I guess that compiletime data migration is performed in a single thread without op-limit reset. (It's just my guess, correct me if I'm wrong). If I'm right, probably you should perform compiletime data migration in another thread per each package.

[Frotty]

Sounds reasonable. In general you should remember that the migration feature is still experimental and undocumented.

[rolandc85]

I am facing the same issue where too I had too much @compiletime data in LinkedList and HashMap that causes Warcraft to crash the moment the map is loaded. After commenting some codes that initialises compiletime data, it will continue to work.

[peq]

you should perform compiletime data migration in another thread per each package.

It's not that easy to partition by package. If you have single hashmap that is filled from different packages, it would still be a single object belonging to a single package.

Since we don't have any loops or if-statements in the code for migrating compiletime data we could actually compute precise partitions to avoid op-limit.

[rhazarian]

we could actually compute precise partitions to avoid op-limit

Sounds awesome. Though about this aswell after posting 😄

[peq]

I now tried to implement this strategy here: https://github.com/wurstscript/WurstScript/pull/921

Could you provide your project so that we can test this in practice or can you try the updated version yourself?

Just replace your wurstscript.jar (usually in %HOME%/.wurst) with this version: wurstscript.jar.zip

[rhazarian]

Don't have a public repo with project yet. Tested myself.

Still crashes. Probably would work with lower fuel limit or a bug exists. It has not splitted this: https://paste.ee/p/QwKiR

More than 80k lines of code here. Inline & localOptimizations enabled, opt disabled (to see the names of the functions).

Note: Easy way to make up a test: allocate a few hundreds of linked lists with e.g. 10 elements inside each one. Creates pretty blown up initCompiletimeState func due to the allocation of nodes.

[peq]

Easy way to make up a test: allocate a few hundreds of linked lists with e.g. 10 elements inside each one. Creates pretty blown up initCompiletimeState func due to the allocation of nodes.

Thanks. I am getting the following code for the LinkedList example: out.j.txt

I had a bug in the splitting code. The partitioning was correct, but then it added all partitions to the first function.

I also added an option to set the limit manually (e.g. -functionSplitLimit 300).

Would be great if you could give it another try with the real map.

wurstscript.jar.zip

[rhazarian]

Good job! It works now, checked both in single player and on Battle.net (with Reforged).

[Frotty]

Good job! It works now, checked both in single player and on Battle.net (with Reforged).

Thanks for testing, I merged it now.

[rolandc85]

It seems that the map doesn't crash immediately upon load. However, I'm still getting crashes after the map loads halfway. Probably there's other stuffs which needs spliting.

[peq]

So does it crash on the loading screen or after the game has started? Can you share your code or provide some kind of example that triggers the crash ?

[rolandc85]

all of the call ExecuteFunc("initCompiletimeState0"), I tried commenting them and map loads without crash. When the first initCompiletimeState* was uncommented, it crashes. It crashes halfway while loading the map.

[peq]

Can you please share the code so that we can find the cause? It might be sufficient if we have the generated code for the first initCompiletimeState_ function, but it's often easier to find problems when also having the Wurst code.

You can also try using a lower split limit (-functionSplitLimit 300).

[peq]

You can upload it here on the issue using drag&drop.

If you don't want to share it publicly you can also send it via mail (Peter.peq@gmail.com).

I'll probably be on irc this evening.

[rolandc85]

Hi peq, one of the findings is as such:

This is called. call ExecuteFunc("initCompiletimeState_0") and this is the function header function initCompiletimeState_0 takes string w__wurst_stackPos returns nothing

I don't think ExecFunc work with a function that takes in parameters?

I tried changing it to call initCompiletimeState_0("Some string") and it worked

[rhazarian]

Oh, I had stacktraces disabled when tested.

[rolandc85]

Ok, I think i found the actual issue. Some functions requires parameters while some doesn't. The following works

call initCompiletimeState_0("initCompiletimeState_0")

call initCompiletimeState_1("initCompiletimeState_1")

call initCompiletimeState_2("initCompiletimeState_2")

call initCompiletimeState_3("initCompiletimeState_3")

call initCompiletimeState_4("initCompiletimeState_4")

call initCompiletimeState_5()

call initCompiletimeState_6()

call initCompiletimeState_7()

call initCompiletimeState_8()

call initCompiletimeState_9()

call initCompiletimeState_10()

call initCompiletimeState_11()

While the following original code didn't work:

call ExecuteFunc("initCompiletimeState_0")

call ExecuteFunc("initCompiletimeState_1")

call ExecuteFunc("initCompiletimeState_2")

call ExecuteFunc("initCompiletimeState_3")

call ExecuteFunc("initCompiletimeState_4")

call ExecuteFunc("initCompiletimeState_5")

call ExecuteFunc("initCompiletimeState_6")

call ExecuteFunc("initCompiletimeState_7")

call ExecuteFunc("initCompiletimeState_8")

call ExecuteFunc("initCompiletimeState_9")

call ExecuteFunc("initCompiletimeState_10")

call ExecuteFunc("initCompiletimeState_11")

[peq]

Thanks. The stack-trace bug should be fixed now.