Beginning Blender 2.5 integration

[GoogleCodeExporter]

Hello,

Blender 2.5 is in feature freeze for quite a time now and the current bug 
correction/stabilizing period will soon lead to a stable 2.6 release. It is 
quite safe to say that the file format will not change a lot anymore and it 
did change a lot between 2.49 and 2.5. I was thinking that it might be a 
good time for beginning integration with blender 2.5.

I am interested in working on it but I would appreciate your impressions.

Is is still the plan to just pass a blend file to a gamekit executable, if 
yes do you have some doc/point of view on how to modify blendloader to 
accept blender 2.5 files.

Thanks

Xavier

Original issue reported on code.google.com by xavier.thomas.1980@gmail.com on 4 Jun 2010 at 6:55

[GoogleCodeExporter]

I also am Interested in this and have been, in my spare time, looking at the 
code for this.

To start with, it looks like the file format structures (defined in a series of 
header files) are generated 
programmatically and then the code uses them in a gamekit specific manner. If 
this is the case we need 
to have the same process applied the new file format structures.

Secondly, the code seems structured in such a way that only the one file format 
will load properly. If we 
code in the 2.5 file format, it looks like the 2.49 format will no longer work.

Original comment by BenT.Sol...@gmail.com on 5 Jun 2010 at 6:52

[GoogleCodeExporter]

I also began looking in the code and saw some code are generated using a tool 
named 
bParse. I didn't found any trace of this tool, only some python scripts in the 
bullet 
svn.

Concerning the switch from 2.49 format, we have the choice to:
-Only load the newest version and keep something like the blender do_version() 
to 
automatically update old file before loading.
-Modularize the loading code and have one loader for each version.

Original comment by xavier.thomas.1980@gmail.com on 5 Jun 2010 at 5:40

[GoogleCodeExporter]

It would be not hard to support both 2.49 and 2.5/2.6.

Is the new mesh format already introduced (bmesh)? As far as I know, 2.6 is the 
first 
'stable' Blender, so that would be a good target.

Original comment by erwin.coumans on 6 Jun 2010 at 5:58

[GoogleCodeExporter]

You need to generate header files and dna for newer Blender versions.

The HeaderGenerator extracts the dna and header files from a .blend file. I 
couldn't 
find it in the latest gamekit trunk, but we use the same tool for Bullet (the 
.bullet 
file format is similar to the .blend format and shares the same tools). 

The process is not documented (yet) but see for example:

http://code.google.com/p/bullet/source/browse/trunk/Extras/Serialize/HeaderGener
ator/
createDnaString.bat

Original comment by erwin.coumans on 6 Jun 2010 at 6:05

[GoogleCodeExporter]

My understanding is that BMesh will not be introduced until after the next 
stable
release of Blender. I'm not a member of the core devs for Blender, but have 
spoken to
a few of them at length. 

The progress (or lack thereof) on the Durian film project leaves the Blender
developers without time to integrate Bmesh into the Blender platform, change 
the mesh
tools to utilise the new data structures, and test it sufficiently for them to
release it. Combine this with the fact they have Joeedh, the author of BMesh, 
hired
to work on bugs in non-related code (i.e. he is a general coding asset for 
Durian,
not for BMesh at this time); and it doesn't look like we're going to see BMesh 
for
well over six months from now.

Original comment by BenT.Sol...@gmail.com on 6 Jun 2010 at 6:23

[GoogleCodeExporter]

Hey All,
I agree with beginning to integrate Blender 2.5/6.

The first thing is we need to make the Api generation more manageable by 
generating
all headers into a single file. There is a test generation program, already in 
svn:
http://tinyurl.com/24yau8d

It will need to be modified to use specific namespaces (Blender249 and 
Blender26)

I'm working on a re-factored converter to make the process easier. So now in 
theory
all that will need to be done is to write separate version loaders using each 
namespace. 

Original comment by snailr...@gmail.com on 6 Jun 2010 at 2:40

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

OK, I manged to create the auto-generated files (with default blend file of 
blender 
svn r29275).

Original comment by xavier.thomas.1980@gmail.com on 6 Jun 2010 at 10:26

Attachments:

• autogenerated25.tar.gz

[GoogleCodeExporter]

snailrose, I tried to generate into a single file using the test program you 
linked. 
Unfortunately I didn't manage to get it to compile. There are some cyclic 
dependencies 
that prevent building bApi when building the branch and when trying to build 
only this 
executable it complains about missing headers. I found those missing headers in 
bullet 
svn, so I will need to learn CMake a little and try to import the requested lib 
from 
bullet. 

Is there an easy way to set up a build environment to get it compile?

Original comment by xavier.thomas.1980@gmail.com on 6 Jun 2010 at 11:56

[GoogleCodeExporter]

A namespace might be a good idea to separate out the header files and prevent 
mixing
autonamed structures in the compilation. 

That said, unless the data gets loaded into some base set of structures (i.e 
same
structs regardless of Blender version), there will need to be alot of duplicate
(redundant?) functions for the loading of data from BLEND format to (for 
example) the
OGRE data objects.

I think this needs to be thought through properly or we'll wind up with a
spaghetti-code implementation that will be hard to maintain!

Original comment by BenT.Sol...@gmail.com on 7 Jun 2010 at 5:56

[GoogleCodeExporter]

If I understood well the data is loaded in a set of structures similar to the 
blender 
DNA structs, which changes slightly with each Blender version. So we 
effectively need 
2 loaders (in addition to the 2 serializers) and some loading code will be 
similiar 
but duplicated because of the use of 2 different namespace.  For example 
loading mesh 
data from 2.5 files would be similar to 2.4 but loading armature or animation 
data 
relative to this mesh will be different.

Original comment by xavier.thomas.1980@gmail.com on 7 Jun 2010 at 2:41

[GoogleCodeExporter]

I keep forgetting that bParse is setup to be backwards compatible, so 249 files 
can
still be loaded from the 250 Api. We can always use the most up to date Blender
headers and do version checks for older files.

After some testing, everything loads as normal from 250 files except action 
channels,
which have been replaced by FCurves. So to stay compatible with 249 files, 
action
channels will need to be converted to the newer FCurve structures.

Original comment by snailr...@gmail.com on 7 Jun 2010 at 3:13

[GoogleCodeExporter]

snailrose: I managed to build bApi (with ugly hacks)and generate the attached 
single 
file include. However I got folowing warning while generating:

BLENDER-v252

sizeof(void*) == 8
Swapping endian? no
File format is 64 bit
Varing pointer sizes? no
swap = 0
swap = 0
Warning, file DNA is different than built in, performance is reduced. Best to 
re-
export file with a matching version/platformWarning, file DNA is newer than 
built 
in.Building datablocks
Chunk size = 24
File chunk size = 24
resolvePointers start
resolvePointers end
numAllocs = 657

Maybe I should rebuild the BlenderSerialize project with the headers generated 
with 
Header generator first?
Do you think this is OK to begin writing a 2.5 loader?

Original comment by xavier.thomas.1980@gmail.com on 7 Jun 2010 at 3:15

Attachments:

• Blender25.h

[GoogleCodeExporter]

snailrose: the fcurve change is to biggest because there were not a lot of 
change in 
the GE in the 2.5 development phase. But some other small changes have bee 
introduced 
recently:
-Mostly Dalai & Matt game logic stuff
-Armature deform modifier instead of armature parenting
But those are small changes. 

So we need to keep some gamekit do_version() almost identical to blender 
do_version(), but with different liscences; and only load the last stable file 
format? This seems more reasonable than maintaining multiples almost identical 
loaders.

In this case maybe we need to include the tool to generate the headers 
(HeaderGenerator/bApi?)in the gamekit svn and integrate it in the build systems.

Original comment by xavier.thomas.1980@gmail.com on 7 Jun 2010 at 3:48

[GoogleCodeExporter]

Yeah, version checking seems the best way to go. I need to get all my local 
changes
back up into svn beforehand though. Then the headers/DNA can be updated to 
reflect
250 files.

Original comment by snailr...@gmail.com on 7 Jun 2010 at 4:20

[GoogleCodeExporter]

Ok, I will wait for for your commit then. 

Original comment by xavier.thomas.1980@gmail.com on 7 Jun 2010 at 4:48

[GoogleCodeExporter]

Fantastic. The last copy of OgreKit I built on OSX (where the OSX LogicDemo 
issue was noticed) had no animation and no textures. I gather the textures have 
been fixed from the above comments?

Original comment by BenT.Sol...@gmail.com on 8 Jun 2010 at 1:27

[GoogleCodeExporter]

No textures are still not working here. But once bParse is updated for blender 
2.5 DNA it should be easy for contributors to correct the loader problems and 
support more blender options.

Original comment by xavier.thomas.1980@gmail.com on 8 Jun 2010 at 3:42

[GoogleCodeExporter]

I got it working. Patch attached.
-I updated DNAstring for 64bit only (can I generate the string for 32bit with a 
32bit file but on a 64bit platform?)
-After this textures was working whithout anything else.
-I rewrote a part of the skeleton loader to load 2.5 animation (form fcurve 
instead of old IPO)
-For this I added a function in gkAction to get the gkActionChannel 
corresponding to a bone, for this I introduced a gkActionChannel::getBone() 
wichi return the protected gkBone m_bone. I know this is bad but what is the 
best solution?

Patch attached, I hope I didn't miss anything with svn/cmake ...

Xavier

Original comment by xavier.thomas.1980@gmail.com on 9 Jun 2010 at 2:05

Attachments:

• gamekitb25_01.diff

[GoogleCodeExporter]

Xavier, sounds great.

>>and only load the last stable file format?

As far as I know (and Charlie already mentioned) we should be able to load 
older versions of Blender with the latest loader: it is backwards compatible. 
Just don't remove support for the old data structures (for animation etc).

And indeed, if you write a do_versions, it needs to be under a ZLib/MIT/BSD 
style license, just like the other parts of GameKit/OgreKit.

You can create 32bit DNA a 64 platform: just compile a 32bit executable that 
runs on the 64bit platform (and load a 32bit .blend file).

Original comment by erwin.coumans on 9 Jun 2010 at 7:28

[GoogleCodeExporter]

Thank you.
I also wrote a very basic python script (attached) for blender 2.5. It permit 
to launch gamekit for within blender by integrating it as a render engine.

Original comment by xavier.thomas.1980@gmail.com on 9 Jun 2010 at 7:59

Attachments:

• render_gamekit_01.txt

[GoogleCodeExporter]

I made a small video demonstrating the blender plugin: http://vimeo.com/12441883
I will work on the 32 bit issue and polish the whole thing soon.

Original comment by xavier.thomas.1980@gmail.com on 9 Jun 2010 at 11:41

[GoogleCodeExporter]

Awesome - I shall try this out pronto!

Original comment by BenT.Sol...@gmail.com on 10 Jun 2010 at 2:00

[GoogleCodeExporter]

Damnit - I'm not on a 64bit platform, so this is not going to work for me. I'll 
have to wait until a 32bit version is available.

Original comment by BenT.Sol...@gmail.com on 10 Jun 2010 at 2:19

[GoogleCodeExporter]

Hi,

My first patch was missing some files, this one is much cleaner and should work 
on 32 bits too. momo_ogre is working but not AnimationMomo (when converted to 
blender 2.5).

I reworked the blender plugin also but I am trying to get some changes in 
blender svn so that this stays a standalone py sript.

xavier

Original comment by xavier.thomas.1980@gmail.com on 10 Jun 2010 at 10:50

Attachments:

• gamekitb25_02.diff

[GoogleCodeExporter]

This fails to compile for me with the compiler throwing this error:
-----
Compiling...
gkSceneLoader.cpp
c:\projects\\gamekit\bullet\extras\blender\blender_View3D.h(67) : error C2208: 
'float' : no members defined using this type
-----

From what I can ascertain, that would imply that somewhere in the gkSceneLoader 
include file hierarchy, something is redefining float!

Original comment by BenT.Sol...@gmail.com on 11 Jun 2010 at 1:26

[GoogleCodeExporter]

I don't anything anormal with this file here, can you post yours after patching?

Also as the blender python scripts are not really my domain I did not commit my 
Blender modifications, but Dalai reviewed and committed it. So now the 
integration plugin is a stadalone py scripts (no more need to patch blender).

It definitely needs polishing but here it is.

Original comment by xavier.thomas.1980@gmail.com on 11 Jun 2010 at 1:41

[GoogleCodeExporter]

I forgot the file.

Original comment by xavier.thomas.1980@gmail.com on 11 Jun 2010 at 1:42

Attachments:

• engine_render_gk_01.py

[GoogleCodeExporter]

The error is, as you can see, being thrown in the header file of 
blender_View3D.h but it occurs whilst compiling gkSceneLoader.cpp

My experience is that this generally has nothing to do with the header file or 
cpp file in question as the type override is in come other header file. I am 
attaching the files to this comment anyhow.

Original comment by BenT.Sol...@gmail.com on 11 Jun 2010 at 1:45

Attachments:

• blender_View3D.h
• gkSceneLoader.cpp

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Ok, maybe this define should be limited to a certain scope to permit 
auto-generating those files without further manual editing.

Original comment by xavier.thomas.1980@gmail.com on 11 Jun 2010 at 2:03

[GoogleCodeExporter]

Think I have found the problem. WinDef.h defines "near" as nothing and that 
leaves the variable declaration of "near" on the line given to be "float ;".

By changing the variable names (which are not used in OgreKit anyway) from:
    float near;
    float far;
to:
    float near_f;
    float far_f;

The code now compiles.

--> Damn it! Wrong link - I was hoping to add detail. This goes before the 
above comment

Original comment by BenT.Sol...@gmail.com on 11 Jun 2010 at 2:12

[GoogleCodeExporter]

> Ok, maybe this define should be limited to a certain scope to permit 
auto-generating those files without further manual editing.

That would be nice, but it is in the standard Windows headers. In other words, 
adding needed OS/system includes is almost bound to add this into the header 
tree. Windows is known for being somewhat... "invasive" in their code 
definitions.

Original comment by BenT.Sol...@gmail.com on 11 Jun 2010 at 2:15

[GoogleCodeExporter]

Maybe an "#undef near" well placed. Anyway I would apreciate any feedback, Does 
is works on 32 bit (animations) ? What os are you using ? 

Original comment by xavier.thomas.1980@gmail.com on 11 Jun 2010 at 2:28

[GoogleCodeExporter]

I'm rebuilding it now. The patch applied fine on my code but the applications 
then crashed due to not finding an active scene. I've updated to the latest 
SVN, with my minor patches to detour/recast logging added (added carriage 
returns to messages so I could read them properly). Rebuilding now and will let 
you know.

An --[ #undef near ]-- could be placed prior to the --[ include "blender.h" ]-- 
line if desired, but might have repercussions elsewhere.

Original comment by BenT.Sol...@gmail.com on 11 Jun 2010 at 2:44

[GoogleCodeExporter]

OK, it seems to parse the file OK, but when it attempts to get the "active 
scene" from gkEngine::getActiveScene(), it crashes as it is apparently null.

Note that this is called from the GameKit_GetActiveScene function, implying 
that this is post load and when it is running the Lua script from within the 
momo_ogre.blend file.

Original comment by BenT.Sol...@gmail.com on 11 Jun 2010 at 3:09

[GoogleCodeExporter]

Well, the issue exists in both the Blender 2.5 & Blender 2.49b versions of the 
momo_ogre.blend file.

The issue appears to be that the first scene in the file (according to the 
code) has no objects in it. Therefore the gkScene::implLoad(...) function does 
not reach the location in the code where the gkEngine::setActiveScene(...) 
function is called. Not that it would work if it did (having no objects).

Original comment by BenT.Sol...@gmail.com on 11 Jun 2010 at 4:49

[GoogleCodeExporter]

Surprising, maybe I failed the rna string generation on 32 bit. I will retest 
it but I can't test under windows.
I also opened another issue concerning the playAnimation that does not works 
with MomoAnimation.blend once converted to 2.5 (issue 68)

Original comment by xavier.thomas.1980@gmail.com on 11 Jun 2010 at 1:06

[GoogleCodeExporter]

I've applied your code, Would you be willing to generate a 64bit DNA with the 
newer header / DNA generator ?
In any case, thanks for the patches!

Original comment by snailr...@gmail.com on 13 Jun 2010 at 12:29

[GoogleCodeExporter]

Sure I will do it as soon as I can but it maybe not possible before a few days. 
Thanks for the great commits. What revison of blender did you used to create 
the DNA?

Original comment by xavier.thomas.1980@gmail.com on 13 Jun 2010 at 12:47

[GoogleCodeExporter]

Hopefully I managed to did it in the small time that left me today.

By the way, there is still the problem with the function I added:

gkActionChannel* gkAction::getChannel(gkBone *bone) 

to get the actionchannel corresponding to a bone, which calls (in 
gkActionChannel)

GK_INLINE gkBone* getBone(void)

which return his private member m_bone.

This is bad, a simple work around would be to use a isBone(bone) function that 
return true if the actionchannel bone is equal to the one passed in the 
function. Other solution might be possibles.

xavier

Original comment by xavier.thomas.1980@gmail.com on 13 Jun 2010 at 1:17

Attachments:

• dnaString252-64bit_r2928.txt

[GoogleCodeExporter]

Sorry I tought you modified the previous header generator and didn't see the 
new one in a different path. Here is the file.

Original comment by xavier.thomas.1980@gmail.com on 13 Jun 2010 at 1:39

Attachments:

• dna_252_64_r29286.cpp

[GoogleCodeExporter]

with eol

Original comment by xavier.thomas.1980@gmail.com on 13 Jun 2010 at 2:10

Attachments:

• dna_252_64_r29286.cpp

[GoogleCodeExporter]

Thanks for the DNA file.

Access to the private bone will eventually be needed from Lua scripts, so I see 
no harm.

Original comment by snailr...@gmail.com on 13 Jun 2010 at 2:11

[GoogleCodeExporter]

Since the dna update I have got a totally magenta screen and crash on exit.
A lot of :
Warning: Skipping image xxxxxxxxx no packed file information is present!

then on exit:
*** glibc detected *** ./OgreKit: corrupted double-linked list: 
0x00000000027c2e10 ***

Time to sleep I will look at it on Monday.

Original comment by xavier.thomas.1980@gmail.com on 13 Jun 2010 at 2:36

[GoogleCodeExporter]

Also when I generated the dna string I got these notes:
Note: using void* on undefined DNA type: ImBuf
Note: using void* on undefined DNA type: RNG
Note: using void* on undefined DNA type: RenderInfo
Note: using void* on undefined DNA type: RetopoViewData
Note: using void* on undefined DNA type: ViewDepths
Note: using void* on undefined DNA type: SmoothViewStore
Note: using void* on undefined DNA type: wmTimer
Note: using void* on undefined DNA type: PanelType
Note: using void* on undefined DNA type: uiLayout
Note: using void* on undefined DNA type: ImBuf
Note: using void* on undefined DNA type: ImBuf
Note: using void* on undefined DNA type: anim
Note: using void* on undefined DNA type: ImBuf
Note: using void* on undefined DNA type: ImBuf
Note: using void* on undefined DNA type: Particle
.
.
.

Original comment by xavier.thomas.1980@gmail.com on 13 Jun 2010 at 3:02

[GoogleCodeExporter]

I will look into testing this issue soon.

Those notes are safe to ignore, it's structures that are never written
to the DNA, just the file. It's mainly structures from the blender kernel 
source.

Original comment by snailr...@gmail.com on 13 Jun 2010 at 3:14

[GoogleCodeExporter]

The last update resolved the problem. It also resolved the keyboard problem 
when opening the 32bit file. The only one remaining is crash on exit when using 
32bit blend file on a 64bit arch.

Thanks a lot for the great work.

Original comment by xavier.thomas.1980@gmail.com on 13 Jun 2010 at 4:01

[GoogleCodeExporter]

Great!

I've finally managed to recreate the remaining issue, so I should be able to 
get it corrected.

Original comment by snailr...@gmail.com on 13 Jun 2010 at 5:20