[Feature request] Export to .kvx

[axredneck]

Use case: creation of voxel models for GZDoom mods

[guillaumechereau]

Yeah, actually it shouldn't bee too complicated, since I already have the code to import kvx files.

[guillaumechereau]

Anybody has the complete specification of the kvx format? I cannot find any documentation on that.

[YarlBoro]

It is in package with SLAB6 and it’s source on Ken Silverman page (author) and on some external site with Build engine resources(they mention KVX file format description). Ken’s page: http://advsys.net/ken/download.htm Build engine resources page: http://www.dukertcm.com/knowledge-base/downloads-rtcm/general-tools-voxel/

Have a good day

[monyarm]

Please someone do this, there's not a voxel editor with KVX exporting that can run on linux

[YarlBoro]

Hello! It's been a long time since this feature was discussed. With recent revival of Build engine games, IMO it is relevant more than ever. I will try to prove my point.

In few months Ion Maiden is coming out. This is completely new 3DRealms game based on much enhanced Build engine (EDuke32). It uses voxels for pickups and some of level props. This game will be VERY modders friendly & has very enthusiastic following by Build-games fans & mappers scene.

Blood (1997) will be officially remastered and re-released. Goodbye DosBox.

There is alive and kicking scene of modders & mappers of Duke3D, Blood & Shadow Warrior. Every month brings improvements to projects like Eduke32, NBlood or GDXBlood. There is constant flow of new content created by Build engine fans, voxel enthusiast too. Outdated & terrible SLAB6 is only option for exporting KVX models.

Imagine, to save in KVX format I have to save model as .QB in Goxel, load it in Voxelshop, save it as KV6, open it in SLAB6 on other Windows machine, save it as KVX, load it into the game on my MacOS machine...and cross fingers that I have not **** something in the beginning.

Please reconsider adding KVX export to Goxel, thanks!

[guillaumechereau]

OK, this workflow is not optimal for sure. I will try to see if I can do it. I'll need to find some KVX loader I can run on linux though, or it will be tedious for me to test it out.

[YarlBoro]

Fantastic! In regard to documentation of KVX format, there is this description of loading KVX in SLAB6 readme file. Here it is: slab6.txt

SLAB6 has it source code attached to its package on Ken Silverman homepage. SLAB6+SRC In file slab6.c I've found KVX relevant code with commented-out descriptions of various functions. I'm not programmer, I hope it suffices.

[YarlBoro]

Voxelshop export module for KV6 written in Java: https://github.com/simlu/voxelshop/blob/develop/src/main/java/com/vitco/app/export/Kv6Exporter.java Since KV6 is KVX predecessor, it might be useful (?).

[guillaumechereau]

By the way, have you tried the kvx import of goxel? Does it work? I wrote it a long time ago and basically never tested it.

[guillaumechereau]

Nevermind :P

[YarlBoro]

You beat me to it! ;) Yes, all KVX models that I've opened in Goxel appear to be fine in terms of form, color & orientation.

In regard to KVX, there is also Ken Silvemans Poly2Vox. It is CLI utility that converts mesh models to voxels. It can export to few voxel formats, KVX too. Source code is attached to package: http://advsys.net/ken/poly2vox.zip

[YarlBoro]

I have just found description of palette.dat. It is palette file necessary to set right colors in KVX. It is different for each Build engine game. Otherwise colors are going to be messed up. Here is description file: PALETTE.TXT Tomorrow I'll extract palette files from Duke3D, Blood, Shadow Warrior in case You need them. Maybe it would be possible to load custom palette.dat file on KVX export? That way any possible palette/game would be supported. Just a humble suggestion.

[guillaumechereau]

Yes I think I will try to generate the best palette during export, the same way I do for the magica voxel export.

[YarlBoro]

You are probably right. I mentioned it, because I had a problem with appearance of KVX files taken straight out of SLAB6. In SLAB6 model looked fine, but colours were completely off in EDuke32. I found out that before export in SLAB6, I have to use function 'Replace palette and Convert colors', choose palette.dat extracted from Duke3D game files, and after that export. Only then KVX looked properly in game.

In case You need it, here are palette.dat files from Duke Nukem 3D, Shadow Warrior, Blood & Redneck Rampage. I also attached two KVX samples for each game, that I taken from community voxel packs created for corresponding games. No samples for RR, since this game has no voxel content yet. Ion Storm has no palette.dat in its game files.

BUILD_PALETTES.zip

I'm surprised that all of those KVX samples load in Goxel with proper colours, without being fed with appropriate palette.dat files. I probably have wrong idea about how it works on technical side.

[guillaumechereau]

I an not sure either. I guess slab6 uses different format for editing and export. The KVX format seems to be to be optimized for fast rendering (it's actually pretty clever), but not for modelling. I haven't read the source of slab much, but my guess is that the models can be saved either as 'rendering ready' (what i am trying to implement now), or as 'modelling friendly'.

[guillaumechereau]

OK I wrote a first draft of export.

It's probably quite buggy, and the axis are inverted, but I could export the cup model as KVX and reopen it with goxel.

[YarlBoro]

Blazing fast, as usual :)

[YarlBoro]

Thanks for time and effort You put into this feature!

Regarding what You said before about KVX 'rendering ready' and 'modelling friendly', maybe it is the 'Mips' setting (I could remember name wrong) that can be set to ether 1 or 5 value. 5 is appropriate for Build games according to Ken Silverman documentation.

I'll try to compile latest Goxel and will come back with results.

[YarlBoro]

Unfortunately it fails to compile. Here is txt file with pasted compilation error: goxel_compile_errors.zip

[guillaumechereau]

Yeah the usual problem, that occurs because I mostly work on linux and don't update the osx build each time. Should be fixed now. Let me know if you can get it to work. Tomorrow I'll try to fix the orientation bug.

[YarlBoro]

It compiled successfully! :) What I observed, Goxel seem to save KVX properly - obviously with orientation flip - only when I draw new model from scratch. Those files seem to always save properly without crash. If try I to import some file(s) and export as KVX, Goxel succeeds only sometimes. I couldn't find deciding factor, like size.

[axredneck]

@guillaumechereau

I'll need to find some KVX loader I can run on linux though, or it will be tedious for me to test it out.

GZDoom ? EDuke32 ?

[guillaumechereau]

I couldn't figure out how to make GZDoom work with voxel support. Anyway, I fixed a few bugs in the KVX import/export. It seems to work at least with the example files from slab6: I can import them, export them and re-import them in goxel.

[YarlBoro]

Great, I will check it out and get back with results. Thanks!

[YarlBoro]

I made few tests, this is what I found out. KVX files generated by Goxel open in SLAB6 with proper orientation and appearance of colours. However on attempt to use use any of such files in Build engine map editor Mapster32, they hang editor on startup. EDuke32 starts with those assets, but their in-game appearance of colours is completely messed up. Here is screenshot of KVX file generated by Goxel right after loading to SLAB6:

I have tried to save such file in SLAB6 and use it (5 mips on export). Same as before, hang of editor & wrong in-game colours. So I used 'Replace palette & convert colours' with PALETTE.DAT from Duke3D. Please notice how palette changed:

Such file was (obviously:)) working perfectly.

Here KVX files for comparison: 0961_exit_GOXEL.KVX - file straight out of Goxel 0961_exit_SLAB6.KVX - SLAB6 export (5 mips) 0961_exit_SLAB6_COLORCONV.KVX - SLAB6 export (5 mips) with color conversion applied EXIT_VOX.zip

[guillaumechereau]

OK. Does it mean that Duke3D ignore the KVX file palette and always use its own palette? If that is the case what I could do is ask before export if we should use force the Duke3D palette I guess.

[YarlBoro]

Yesterday I've read somewhere in Build files formats documentation by K. Silverman, that KVX files store palette inside of them. If that palette differs from palette set arbitrarily by Build game with its palette.dat file, it is displayed incorrectly (not a bug, feature). What do You think about option of setting custom palette (via various games palette.dat's or some other more common palette format like .pal)? Maybe adding variants of KVX to export formats list? Forcing Duke3D palette is probably easiest solution. That limits all KVX functionality to Duke3D modding, but this is still waaaay better than that conversion process that I explained You in the beginning.

[guillaumechereau]

I see. I can probably add support for other palette formats, maybe even the .dat used by build engine to make it easier.

[YarlBoro]

If that palette differs from palette set arbitrarily by Build game with its palette.dat file, it is displayed incorrectly (not a bug, feature).

This is why each Build game has different PALETTE.DAT file. They all use different palettes. All sprites and voxel models for each given game have to use same palette.

EDIT: I posted before seeing Your post. Thats great!

[guillaumechereau]

Quick question: do you have some .dat palettes I can use to test it?

[YarlBoro]

I already posted them on this thread :) Here: BUILD_PALETTES.zip

There You will find palette.dat files for DN3D, Blood, Shadow Warrior & Redneck Rampage in separate folders with samples of KVX files using palettes corresponding to games.

[guillaumechereau]

OK thanks. I'll have a look at it later today or tomorrow if I have time.

[YarlBoro]

Thanks! Have a good day

[axredneck]

option of setting custom palette (via various games palette.dat's or some other more common palette format like .pal)

What about importing palette from .png or .bmp ? Usually these files ripped from corresponding game have proper palette already.

[guillaumechereau]

I did two very quick fixes for this. So now it should be possible to use custom .dat palettes (need to put them in the user data dir, though I forgot at the moment where it is on mac), and exporting a KVX file with a maching current palette with use it by default.

[YarlBoro]

Can't wait to try it out, thank you!

[YarlBoro]

So I tried exporting few of models. Unfortunately on KVX export their upper part is removed fully or partially like here:

So now it should be possible to use custom .dat palettes (need to put them in the user data dir, though I forgot at the moment where it is on Mac)

I have tried putting palette.dat here, but it was not used: System partition\Users\MyAccount

[guillaumechereau]

There was a bug in the user path on OSX I realize.

You need to get the last version, and put the palettes in the location: ~/Library/Application Support/Goxel/palettes/duke3d.dat

The file has to end with '.dat' in lowercase.

Then you should see the palette in the palette list.

[YarlBoro]

It works! :) Now I can see palette.dat on palette list & load it. Color arrangement of color swatches is same as in SLAB6 in Goxel UI. However, after export KVX palette colours seem to be arranged in random way, not in same way as in Goxel UI or SLAB6 UI. That causes hang in map editor & bad colours in EDuke32. Maybe You could force colour swatches arrangement on export?

The other thing, on KVX export Goxel seems to throw away not only upper part of model but also lower part. You can observe this with this model: 0915_GATE.gox.zip

I'm not complaining or anything like this, thanks for all work You put into this.

[YarlBoro]

If my description makes no sense, let me know.

[guillaumechereau]

No worries, I also want to make it work. It would be easier if I figured out how to compile and use EDuke32... I'll check your file soon. A bit busy with other things today. Thanks for the reports!

[YarlBoro]

Ok! :) About compiling EDuke32, this might be helpful: https://wiki.eduke32.com/wiki/Building_EDuke32_on_Linux

It seems that it is very similar to compiling it on MacOS. I did it successfully few days ago. I had to install makefile, SDL2 & all libraries that are listed on Wiki page. It compiled via terminal, 'make RELEASE=0'. I used TortoiseSVN on Windows to download package from svn.eduke32.com, I couldn't figure out how to use SVN on MacOS. I had to do one more thing to make it work. I had to manually add some SDL files to source\build\include folder. Here is list, those with yellow dot:

Without it compilation failed. I took those files from fork of EDuke32, NBlood. Look in same location as above for those files.

NBlood: https://github.com/nukeykt/NBlood

[guillaumechereau]

Thanks, I will have a look at that, cause I think I will never get it working otherwise. I pushed a fix that might fix the clipping bug already.

[YarlBoro]

No problem. If you will encounter some big problems with compilation, I think good place to look for info is Duke4.net. This is official spot for EDuke32 affiliates.

Yes, clipping bug seems to be fixed. All of my larger models - like gate that I've sent - save ok. I can convert bigger sprites to models, nice! I noticed one other thing - I hope I'm not pushing it - Goxel seem always save pivot point as 0,0,0 in KVX. Besides palette conversion that is VIP, pivot adjustment is only reason left do use (damn) SLAB6 at all.

[axredneck]

@YarlBoro

System partition\Users\MyAccount

It's Windows path

@guillaumechereau

I had to manually add some SDL files to source\build\include folder. Here is list, those with yellow dot:

I think it isn't needed on linux, just install "dev" package of SDL2.

[axredneck]

make RELEASE=0

Usually you don't need "RELEASE=0" except if you are going to debug EDuke32

[YarlBoro]

@YarlBoro

System partition\Users\MyAccount It's Windows path

It is MacOS path too. Full path for placing palette.dat: System partition/Users/AccountName/Library/Application Support/Goxel/palettes/duke3d.dat

Usually you don't need "RELEASE=0" except if you are going to debug EDuke32

Ok, thanks!

[YarlBoro]

Just to let you know Guillaume that your work was not in vain, here is topic from Duke4.net with about 17 voxel models that I posted under nickname Borion: https://forums.duke4.net/topic/3322-duke-3d-voxel-pack/page__st__660

All models were converted and adjusted in terms of texturing in Goxel. I mentioned in other thread importance of Goxels alpha painting, super useful. It's perfect in tandem with Qubicles reprojection tools.

Again, thanks for implementing KVX support and other great features! Goxel is essential for my whole voxel workflow, I appreciate its existence :)

EDIT: Minor clean-up for clarity.

[guillaumechereau]

Really nice! What is Qubicle reprojection?

[YarlBoro]

Glad to hear that! :)

Reprojection - I’m traveling at a moment. I will get back with little explanation tomorrow.
Have a good day!