Cycles OSL Support

[redchillipadi]

Initial ebuild for OSL. I am not sure how far away the peer review is and whether you want to freeze master. So I put this in a separate branch to avoid having to repeat all the final compliance checks.

OSL is highly dependent upon the correct versions of boost and llvm

OSL 1.5.12 failed to build. It is reported to work with LLVM 3.4 and Boost 1.55.0 under Ubuntu but I was not able to compile LLVM-3.4 on my system (install fails when it attepts to copy a file onto itself). It does not work under LLVM 3.5

OSL 1.6.9+ gave an undefined reference error compiling against Boost-1.61.0, which was resolved by using Boost 1.55,0 1.56,0 and 1.60.0 (I didn't try 1.58 or 1.59), although a missing symbol for ObjectBuffer in LLVMMCJIT prevented successful linkage.

By backporting a patch from the OSL master branch to RB-1.7 and Release it is possible to remove the dependency on ObjectBuffer and compile OSL-1.7.2 and 1.7.3 under LLVM-3.5. Apparently LLVM-3.4 might also work but I can't compile it. The original patch is at https://github.com/imageworks/OpenShadingLanguage/commit/77edb88eba19e0899da40e2d91ee04aaa3f0cb42

I have tested with Blender 2.77a which finds and uses 1.7.2, 1.7.3 and the master branch successfully allowing me to create a custom mandelbrot shader as per http://elbrujodelatribu.blogspot.com.au/2013/04/blender-cycles-osl-mandelbrot-fractal.html

The documentation now installs under /usr/share/doc

TODO:

• The test use flag is restricted as the tests fail with this patch (see original patch comment) - perhaps it could be removed
• There are additional options in the CMakeLists.txt which could be added eg FastMath

[dracwyrm]

@redchillipadi I was looking into adding OSL, but I heard issues with it. so I was a bit hesitant. The thing with LLVM, all packages in portage must compile with the latest stable which is 3.5. v3.5.2 isn't yet, but should be soon, then testing needs to be done with the next minor version, 3.6.x, to see the progress. This is where the fun begins of submitting upstream the patches that make it work.

If OSL compiles for 3.5 and the latest stable packages of the other depends, then we are good to go.

Maybe we can add an ewarn that OSL is experimental at the moment, maybe? In other words, have users tell us directly on either here or IRC that there's a bug rather than filing it in b.g.o. 📦

This is the Todo list that I put in the b.g.o https://bugs.gentoo.org/show_bug.cgi?id=536078#c55 OPTIONS: OpenSubDiv - http://graphics.pixar.com/opensubdiv/docs/intro.html OpenVDB Headless (renderfarm) Enable Ocen Mod? Jack/SDL dynamic load? LibMV - Motion tracking Cycles Standalone/GUI Cycles with CUDA (nVida only)/CUDA dynamic load Build with LLVM? JEMALLOC? Debug Valgrind support?

I think most of these are done. The remaining can be ignored for now and added to Blender-Next. Nobody misses them, but it would be nice to advertise to the world at some point that Gentoo's Blender can do this! Like Motion Tracking, may make something really like that??? Just a thought.

However, I think the one that we should focus on for next release is headless render farm support above the others. That seems like the most useful. With Gentoo, you could make the leanest install possible to make a render farm. This is why I love Gentoo so much -- you can do anything. It's also what makes it so damn difficult. haha.

[redchillipadi]

I am also excited by the thought of building a headless renderfarm, but agree we should focus on getting this version in the tree now.

As far as llvm goes, 3.5 to 3.5.2 should be a minor change, I can test it tonight, but 3.6 is a major leap as they removed the Jit api completely and changed to mcjit instead. The current code upstream relies on the old jit objectbuffer symbol being available in llvmmcjit so it may need conversion to MemoryBufferObject instead.

[dracwyrm]

At least we can support stable LLVM then, which is good. The nice thing about Gentoo policy is that you don't need to support unstable, but if there is ever a request for an application to be made stable in b.g.o, then a block needs to be added to it, in this case, LLVM 3.6.x stabilisation would be blocked by OSL.

I actually have spare computers that I can do headless render farm testing with, which is why I really want this. Big Buck Bunny was made with blender on a render farm, so there is the uses out there for it.

[redchillipadi]

Well I will attempt to merge the commits relating to whitespace corrections into master and leave just the OSL specific commits here.

[redchillipadi]

Well, that was tricky. Now only the OSL specific merges are left here in development, all the whitespace fixes are merged into master.

[dracwyrm]

Cool. Thanks. :D Cherrypicking is a bit tricky, but really nice when it works.

We should put OSL stuff in our overlays on our computers to test a bit before merging here as I put the link in the bug report to hopefully get some feedback. Then once it's compile tested and runs stable-ish, then merge it here. Might as well get it up for the big push to the tree. Unless it's just horribly broken.

[redchillipadi]

Great, it would be nice to get it into the tree. It compiles fine now that I have worked out to use Clang < 3.6 (for JIT) and Boost < 1.61 (something related to Boost::Wave). I can now confirm that both Clang 3.5.0 and 3.5.2 are suitable

Blender needs to be set to Cycles and CPU mode to show the check box to enable OSL.

[dracwyrm]

We almost need a manual on how to enable some of these extras. :P

[redchillipadi]

I agree. Once we are done I was thinking of updating the instructions on the gentoo wiki

[dracwyrm]

If you are confident that OSL will compile and be usable for users, then you can go ahead and merge this. Not seeing anything too serious with it. We can always do tiny fixes here and there as we go alone.

[redchillipadi]

I am happy that this will work now that I have worked out the dependency issues.

[dracwyrm]

@redchillipadi Doesn't the mcjit allow versions greater than 3.5? https://github.com/dracwyrm/gentoo-ebuilds/blob/master/media-libs/osl/osl-1.7.3.ebuild#L21 Can we remove the less than on the depend and support 3.5 and 3.6?

[redchillipadi]

The problem with the LLVM library is that they keep changing the api, and have JIT in 3.2-3.5 only, MCJIT in 3.6+ and OrcJIT in 3.7+. We currently support both stable LLVM 3.5.0 and unstable LLVM 3.5.2, but not LLVM 3.6 or greater.

The OSL 1.7 release code relies on JIT for ObjectBuffer. It does not link with LLVM 3.6 because ObjectBuffer was removed and relaced with MemoryBufferObject.

Upstream has a patch to provide MCJIT support, which would allow compilation in 3.6, but it fails all the OSL tests and runs much slower than the JIT from version 3.5. I think this may be because it no longer allows lazy compilation. See https://github.com/imageworks/OpenShadingLanguage/issues/480

LLVM changed again to provide OrcJIT in 3.7, and there is discussion about whether OSL 1.8 will support MCJIT or just skip to OrcJIT. OrcJIT does support lazy compilation so it may be faster. See https://github.com/imageworks/OpenShadingLanguage/pull/627

So while the API is in a state of change I think we should use the current OSL code which supports JIT only (LLVM < 3.6). If this is going to be a blocker on blender then I could look at integrating the patch for MCJIT, but we would need a warning to prevent people from actually using it! It might be better to wait to see whether OSL 1.8 will support MCJIT or only OrcJIT and enable it then.

[redchillipadi]

I am looking at updating the wiki. A lot of it includes patches and information only relevant to 2.72b. When we get 2.77a into the tree, will we still keep 2.72b around or can I remove the 2.72b specific information?

[redchillipadi]

I have submitted a first draft of the updated documentation to the wiki. I kept the 2.72b specific information at the bottom so it can be easily deleted when it is no longer in the tree.

[dracwyrm]

Good point on LLVM. I think they may go straight to LLVM 3.7 then because of the Lazy compilation. It won't block Blender, so there's no worries on that. There's a valid reason and valid reasons are perfectly fine. Some devs have been known to give not so good reasons why they do something, read that as they can't be arsed to change it. haha.

Unfortunately, 2.72 will need to stay in the wiki until 2.77+ stabilisation. Once it's in the tree for a month, we can call a bug report asking for stabilisation permission and the removal of 2.72, since it's ancient. That should be a quick process. They basically need to wait to see if there are new bug reports.