Final compliance checks for OpenSubdiv, OpenVDB, Ptex, and Blender

[dracwyrm]

Every tiny detail needs to be vetted as the peer review is very strict, so all files need to be double checked for full complete compliance. Then if they pass that, they go in testing, and then the tree. :)

[redchillipadi]

Many whitespace and other corrections can be merged from push request #7

TODO:

• Split the openvdb Improved Ptex patch into parts less than 20kb
• Repoman full -d blender-2.77a.ebuild QA issues: Bad dependency and Dependency.badindev related to media-libs/openexr-2.2.0. I am not sure what this means.

[dracwyrm]

There are two possibilities with Blender OpenEXR depend: 1) OpenEXR has abi 64 and 32 bit, and we need to declare on in the depend list, or 2) OpenEXR is ~Arch at the moment. If Blender can be compiled 32 bit, then maybe we can add that? Though, it's a Graphics program and 64 bit memory is more ideal. Even Adobe dropped some 32 bit programs and went all 64 bit because of the added work space. Besides, I think all 32 bit should be phased out. Blender has nothing depending on it, so it doesn't matter if it's 32/64 bit. I think I will add a depend to OpenEXR for 64 bit and see if that clears that.

About the Ptex patch, we can use a compressed patch. I will need to check how to add that to the portage tree. Then we don't need to break it up. It's designed to minimise download and save space on the Gentoo Mirrors. It's fine the way it is for testing, but in the tree, we'll just compress it. :)

Repoman can be handy, but it can be a pain in the ass.

[redchillipadi]

I had to use -abi_x86_32 for openexr as it would not compile on my system I will try removing the flag and see if that will fix it.

[dracwyrm]

Yeah. 32 bit is broken for some things anyways. This is why I really don't want to support it. Everything about Blender screams 64 bit. One of us can remove the keyword x86 from the ebuild and depend on everything with abi support by putting in [abi_x86_64] for depends.

[redchillipadi]

Changing the 32 or 64 bit keywords in package.use or in the blender ebuild don't alter the QA messages. I only have 32 bit packages because most emulators and wine require 32 bit, I don't want them to be used by blender.

I think your second possibility is correct. When I change the ebuild to request openexr-2.1.0 (which is stable amd64 and x86) the dependency.bad messages disappear, although some bad.indev messages relating to openexr with ffmpeg and libav under uclibc remain.

[dracwyrm]

@redchillipadi Apart from OpenVDB not showing up or how to use it, I think this is pretty much ready. The ebuilds are now passing a "repoman full" <-- the hardest setting to pass, so they are in good shape; we have lots of exciting new features for users to play with, break, and bug report; the ebuilds look to be in good order; some small patches combined into more simple ones.

The last thing that needs to be done is to compress the Ptex patch, but that's a non-issue. I'l put it up on a separate repo for patches only. There may be others in the future.

What do you think? Are the ebuilds there/nearly there?

[redchillipadi]

So if I got it right, the issues now are not knowing how to use openvdb, the pdfdoc still not working on your system, the compression of the ptex patch, and your enhancements to the openvdb makefile for upstream. I am hoping to look further into the openexr 32 bit build error on my system and future enhancements to patch osl to be more flexible with boost and llvm. None of these should be show stoppers.

I think we are ready, I will pull down the latest versions and do another compile check on my system.

[dracwyrm]

The Makefile for OpenVDB is fine now, just that there are major improvements that can be done, but that will be separate. Maybe we can work with OSL upstream on the patches? Submit pull requests to make it easy to include the patches? These two are after Blender and Co are in the tree.

PDFDoc works for you be not me. :( I have no idea why. I will submit my build log upstream and see what they say. I don't think it should be a showstopper though.

OpenVDB: is this a showstopper? Hmm. Thought: Is OpenVDB installed to the wrong directory from where Blender is expecting it?

@redchillipadi Are there any other issues we need to look at?

[redchillipadi]

I can't think of anything else.

I will do another compile and then probe further into openvdb.

[dracwyrm]

Me too. I will comb through blender files to see if I can find a mention of install path for OpenVDB. This is a huge mystery to me.

[redchillipadi]

Do we need to run everthing in test mode as well to make sure nothing fails? They all seem to work in blender (other than the mystery of openvdb). Many of the packages have RESTRICT="test", is that sufficient?

[dracwyrm]

If tests are broken by upstream, then we really don't need to have them available to users. OpenVDB tests do actually pass on my system, you need to manually call the tests make pytest. We could add that in, but you've seen what their installer is like. :)

OpenSubdiv Ptex patch is now compressed and uploaded to gentoo-distfiles on my github. There's no where else to put it as I don't have a dev account with Gentoo. It'll work for now.

Now to look in source code to figure out OpenVDB support. fun

[dracwyrm]

@redchillipadi

I think I found something.

The make file is looking for this:

if(WITH_OPENVDB)
    set(OPENVDB ${LIBDIR}/openvdb)
    set(OPENVDB_INCLUDE_DIRS ${OPENVDB}/include)
    set(OPENVDB_LIBRARIES openvdb ${TBB_LIBRARIES})
    set(OPENVDB_LIBPATH ${LIBDIR}/openvdb/lib)
    set(OPENVDB_DEFINITIONS)
endif()

if(WITH_OPENVDB)
    set(OPENVDB ${LIBDIR}/openvdb)
    set(OPENVDB_INCLUDE_DIRS ${OPENVDB}/include)
    set(TBB_INCLUDE_DIRS ${LIBDIR}/tbb/include)
    set(TBB_LIBRARIES ${LIBDIR}/tbb/lib/libtbb.a)
    set(OPENVDB_LIBRARIES openvdb blosc ${TBB_LIBRARIES})
    set(OPENVDB_LIBPATH ${LIBDIR}/openvdb/lib)
    set(OPENVDB_DEFINITIONS)
endif()

The path to openvdb however is this:

/usr/include/openvdb
/usr/lib
/usr/lib/libopenvdb.so -> libopenvdb.so.3.1.0
/usr/lib/libopenvdb.so.3.1 -> libopenvdb.so.3.1.0
/usr/lib/libopenvdb.so.3.1.0

Also, Blender is looking for libtbb.a, but Gentoo ebuilds delete those. So there's only:

/usr/include/tbb
/usr/lib64/libtbb.so -> libtbb.so.2
/usr/lib64/libtbb.so.2
/usr/lib64/libtbbmalloc.so -> libtbbmalloc.so.2
/usr/lib64/libtbbmalloc.so.2
/usr/lib64/libtbbmalloc_proxy.so -> libtbbmalloc_proxy.so.2
/usr/lib64/libtbbmalloc_proxy.so.2

So, I think the Blender make file needs a we bit of patching. Maybe then it will work as expected? I hope these paths aren't hardcoded in Blender C files.

[dracwyrm]

hmmm. I wonder how many other things are hard coded incorrectly like this now.

[redchillipadi]

I found a similar section but it was within a WIN32 only block.

I have found that there are scripts in release/scripts/startup/bl_ui/properties_physics_smoke.py that are included in our blender. They are meant to put an OPENVDB ui options in the physic panel under smoke.

And there is python function bpy.app.openvdb that returns the version string and status.

Just trying to compile blender again (after the documentation failure) so that I can check it out

[dracwyrm]

Yeah. Those are WIN32 and APPLE sections. There's no mention of it in Linux. Quite strange.

The doc compile error is back?

[dracwyrm]

@redchillipadi Think I found the doc error. I changed the logic a bit to make it cleaner. See if this works, if not I will put the original back. Change this line: addpredict "${cards}" To this: addpredict "${cards[@]}"

Sorry about that.

[redchillipadi]

On my system the following script

import bpy
print( bpy.app.openvdb )

returns bpy.app.openvdb(supported=True, version=(3, 1, 0), version_string=' 3, 1, 0') So blender can find openvdb

[redchillipadi]

I think I have found where the smoke cache is meant to be, just need to work out how to enable it

[redchillipadi]

cards=( /dev/ati/card* /dev/nvidiactl* )

I think the other issue might be that if test - e "${cards[0]}" is only looking for the first entry in the array? I don't have an ati card so it might be skipping the entire addpredict part

[redchillipadi]

By the way, does addpredict fail if the directory doesn't exist? Couldn't we just do addpredict("${cards[@])") on all systems?

[dracwyrm]

This is the beauty of lists. I posted the final solution in the api doc post. This cards=( /dev/ati/card* /dev/nvidia* ) creates a list of all those directories. If the directory doesn't exist, it's not added to the list. So, for card in "${cards[@]}"; do addpredict ${card}; done will not do anything if those directories do not exist.

[redchillipadi]

On Sun, 12 Jun 2016 08:06:13 AM Jonathan Scruggs wrote:

This is the beauty of lists. I posted the final solution in the api doc post. This cards=( /dev/ati/card* /dev/nvidia* ) creates a list of all those directories. If the directory doesn't exist, it's not added to the list. So, for card in "${cards[@]}"; do addpredict ${card}; done will not do anything if those directories do not exist.

When I was testing it in bash I used $ echo "${cards}" which returned the original string with ati in it

I see that I needed to use the array syntax $ echo "${cards[@]}" instead. Thanks for the explanation.

[redchillipadi]

I have fixed the openexr problem I mentioned in my todo list above, by integrating a patch I found upstream. They have been slow to apply it, possibly because they still want to support <gcc-4.3, but hopefully I can get it included in our version in the tree given that gcc-4.9.3 is stable.

Either way this should not hold up blender because putting media-libs/openexr -abi_x86_32 in package.use is a suitable workaround.

[dracwyrm]

Oh, it's a patch for OpenEXR? You can put that patch in a bug report to that team. Tell them that it makes repoman happy for anything that depends on it and they will probably apply it faster than you can sync the portage tree. :P

[dracwyrm]

@redchillipadi I went through and did some deep inspection of the ebuilds. If I made changes, I made sure to fully document it. I think were about ready to have the ebuilds inspected by devs. If you could look over them to make sure everything is there, that would be good.

I made some use flag and depend changes to blender, so will need to look at the diffs to spot the changes. Another set of eyes on them would be good, as you can do cuda. :) I can't :'(