Blender 2.8 Compatibility

[friggog]

Intending to merge 2.8 to master and leave 2.79 compatible version on secondary branch (blender-279)

TODO:

• [x] Random segfaults
• [x] LODs not hidden properly
• [x] Leaf LODs are not created correctly (i.e. verts not deleted)

See #18 for discussion

[friggog]

@luketimothyjones help appreciated on the random segfaults - is this an issue you are having? I'm on max which could also be a factor

[samipfjo]

Will give it a test in a bit. Need to get Blender 2.8 installed.

[samipfjo]

@friggog Blender 2.8 on Win10 x64 is crashing with EXCEPTION_ACCESS_VIOLATION on generation. We've experienced this issue before, it happens when trying to access objects in EDIT mode. See here: https://docs.blender.org/api/2.78b/info_gotcha.html#edit-mode-memory-access

[arpu]

testing this with final released blender 2.8 works perfect! the only problem i had was to enable convert to mesh after generation exportet gltf/glb file looks good on https://gltf-viewer.donmccurdy.com/

[samipfjo]

@friggog Run through this for a full list of 2.8 gotcha's https://docs.blender.org/api/current/info_gotcha.html#help-my-script-crashes-blender

[samipfjo]

While in pursuit of the cause of crashing, I noticed a weird behavior with the logging of stem creation:

-> 0 stems made
-> 0 stems made
-> 0 stems made
-> 100 stems made
-> 200 stems made
-> 300 stems made
-> 400 stems made
-> 500 stems made
-> 600 stems madeError   : EXCEPTION_ACCESS_VIOLATION

The repeat logging of "0 stems made" is not something that I've seen before and is a little worrisome.

[friggog]

Thanks @luketimothyjones that link is super helpful - could take a while to find where the issue actually is though :( Might be a while again before I can have a crack at it, but at least we're making progress!

[friggog]

Did a bit of investigation and it seems to be crashing in the secondary thread. Forcing the branch generation to run on the main thread prevents the crash, so it seems like it is a threading issue with the branch generation (perhaps due to the insane recursion going on).

As the blender docs recommend, the faulthandler module is somewhat helpful, but just points to random lines where an access violation has occurred due to some threading issue.

One fix is to just execute on the main thread, though this isn't ideal - @luketimothyjones I'd welcome you to have a go at further debugging based on my findings as you implemented the original threading stuff ;)

PS When running operations in other threads I think the generally accepted thing in blender is to treat this as a modal operation to indicate the operation is still going on , rather than returning FINISHED from the operation - this might help to prevent blender from doing it's thing and freeing memory we want to use? I have some code on this somewhere if you need, but I think it is documented fairly well

[samipfjo]

I'll take a look at the threading implementation when I have some free time, though I can make no guarantees as to when that will be. From the very beginning I labeled it as "hacky" due to Blender's lack of thread safety, so that may be finally catching up with us haha. I've played with the alternatives to FINISHED in the past and I don't remember what the issues I ran into were, but if I recall correctly there's a reason I went with FINISHED.

There are undoubtedly several things that are broken / unstable at present, but given the attention to threading that the 2.8 "gotchas" article gave (that was not present in 2.7's), I have a feeling that there were some changes to the underlying C that have thrown a wrench in some of the hacky things we've been doing.

[samipfjo]

So I've done some testing and have made a few interesting observations.

The crashes are 100% the fault of threading because we are 100% doing things that are not supported by Blender. According to the gotcha's page linked above, threads MUST terminate before a script ends execution (eg, the main operator returns FINISHED), otherwise it "may seem to work for a while but end up causing random crashes or errors in Blender’s own drawing code," which pretty accurately describes the behavior we're seeing.

By changing gui.TreeGen.execute() to call gui.TreeGen._construct directly rather than running it in a thread, the instability/crashes are resolved. Well, perhaps I should rephrase that. I have yet to see a crash despite generating compute-heavy trees that fairly reliably crash the threaded code. This does mean that the Blender GUI freezes during generation (after the "Generate tree" button has been pressed), but stability is seemingly restored. I toyed with making gui.TreeGen.execute() return {"RUNNING_MODAL"} rather than {"FINISHED"}, and it makes the threaded code more stable, but it still crashes for the aforementioned reasons. Also interesting to note that making the main executor thread non-daemonic straight up freezes Blender completely for unknown reasons.

With that said, there is some good news. The other unsupported use of threading that we're doing seems to work fine haha. The separate thread that handles console logging (causing a drastic increase in generation speed) does not cause crashes; I assume this is because it is using an inherently thread-safe data structure (threading.Queue) for data transfer, and because it does not use any Blender functionality.

In conclusion, our luck appears to have run out on (ab)using threads in the 2.8 version, and we will likely need to go back to seizing up the Blender GUI in order to not cause memory violation crashes. Fortunately, generation progress can be seen in the console, and somehow the leaf generation percentage complete cursor periodically works. If we cannot find a work-around, the only solution is to await proper thread-safety from the Blender API, which may never come (given the difficulty in doing so that I've seen alluded to by Blender Foundation programmers.)

[samipfjo]

So I spent the majority of yesterday playing around with various work-arounds including but not limited to reworking the entire threading system, switching entirely to modal operators, ensuring Python-side references got held as long as possible, and had zero luck.

I cannot find a way to work around the memory access violations. Unfortunately, as per my previous comment, I think we're out of luck.