Cannot display DirectX file at ObjectViewer.

[ginga81]

These sample of Directx object files exported from Blender-exporter and Metasequoia 3.0LE, these cannot display at Object Viewer. Please correspondence these DirectX files.

Blender:2.79b with DirectX exporter(Built-in) RailShinkansenBallast.blend.x

Metasequoia 3.0LE http://www.metaseq.net/jp/download/2/ metaseqle30.exe RailShinkansenBallast.x

ObjectViewer:OpenBVE-2018-11-09.zip

logs: Warning Unsupported template template Header encountered in x object file RailShinkansenBallast.x Warning Unsupported template template Matrix4x4 encountered in x object file RailShinkansenBallast.x Warning Unsupported template template IndexedColor encountered in x object file RailShinkansenBallast.x Warning Unsupported template template Boolean encountered in x object file RailShinkansenBallast.x Warning Unsupported template template Boolean2d encountered in x object file RailShinkansenBallast.x Warning Unsupported template template MaterialWrap encountered in x object file RailShinkansenBallast.x Warning Unsupported template template MeshFaceWraps encountered in x object file RailShinkansenBallast.x

Operating system Ubuntu 16.04LTS Metasequoia:Wine3.0 stable

Sample data of DirectX Object file: ballastx-issue.zip

[ginga81]

I am very sorry. From Metasequoia directx object is incredibly minimum. Please zoom in maximum at Object Viewer. DDS picture's texture color is not good.

[s520]

In my environment, the color is not good like the screenshot below.

My environment

• Windows 8.1, Intel HD Graphics 4400
• fedora 28, Intel HD Graphics 500

[leezer3]

The DDS issue appears to be because your texture uses RGBA compressed DXT5.

The plugin appears to expect DXT5 to be compressed as BGRA (It's heavily derived from Nvidia reference code).

Let me do a little digging, and I'll see if I can get to the bottom of the texture issue.

---

The object issue is caused by the double-semi-colon at the end of Line 191 - Switching this to a single makes it work. We can probably special-case this one, again will do a little playing.

[s520]

Another problem is an error in RailShinkansenBallast.blend.x.

Log: Unexpected comma encountered in template FrameTransformMatrix in textual X object file RailShinkansenBallast.blend.x

What is the cause of this?

[leezer3]

The short answer on that one is that the file isn't really supported at all by the current parser.

Both Frame and FrameTransformMatrix are incompletely implemented. In this case, the fact that there are two frames present in the model is confusing things no end. Don't know if this is easily fixable, especially as it appears to need the FrameTransformMatrix to position the object correctly.

[ginga81]

When we are create DirectX object for OpenBVE, I created by Metasequoia LE3.0 and Blender 2.79b. If we know what kind of you are using 3D modeling editor, it is very easy to create DirectX model for OpenBVE. Mr.leeser3, please tell us what kind of 3D modeling editor that you supposed for the DirectX object for base parsing.

[leezer3]

The DirectX parser was written by Michelle. It was designed to support the output from the BVE4 Structure Viewer, which is a limited subset of the X format.

I've never actually used an editor personally, just run them up by hand. You might try obj format instead, and see if that exports any better?

[s520]

Thank you for teaching us. I think that it is important to be able to read many formats of X file when reading the route data of BVE5 in the future. (BVE5 only supports X files.)

[ginga81]

Thank you very much for tell us! The BVE5 can read only DirectX object as you known. Also as s520 says, when we created for the sample DirectX data, we researched for the future implement of BVE5 route parser.

Most of BVE5 route creators are using by Metasequoia's Directx exporter. And a part of them are using by DDS file using DXT5 option. So we choosen for testing.

This is out of topic, thank you very much for the implementation of wavefront(obj) parser! At the first create for OpenBVE object, this implementation is very useful for blender user.

[leezer3]

No problem.

There is actually a WIP replacement for the DirectX importer here: https://github.com/leezer3/OpenBVE/pull/280

This has most of the same issues as the current one, but is structured considerably more sensibly. Doing some more work on this one is on my todo list, but at the same time, I want to get a lot of other things done too......

[leezer3]

Current build makes things marginally better, but not much- This stops the Blender object from failing to load, but as the frame transform matrix is not supported, it's broken.

[s520]

I ported a part of Assimp to C#. You can download the project file from the link below. XFileParser.zip This program can parse the X file with the following code.

XFileParser.XFileParser paser = new XFileParser.XFileParser(System.IO.File.ReadAllBytes(file));
Scene scene = paser.GetImportedData();

Assimp can also parse X files output by Blender. Please try it.

[s520]

I'm sorry. I fixed the problem that the program could not read the compressed X file. XFileParser2.zip

[leezer3]

No problem :)

That parses the file alright, but the trouble is more getting it into openBVE. The current animation system has no support for frame-key animations or matrix operations.

On the plus side, our frame transform matrices are a relatively standard 4x4 matrix described here: https://docs.microsoft.com/en-us/windows/desktop/direct3d9/transforms

If the X files simply use the transform matrices to create the final object, it should be (relatively) easily possible to just apply the appropriate matrix to the final object within the parser.

[s520]

Thank you for testing my program.

I think that animation related can be ignored.

And, I think this program will improve reading of X files.

But, I am not familiar with the OpenBVE object management method, so I do not know how to incorporate the analysis results into OpenBVE ...

[leezer3]

It's dead simple if you understand a mesh- Take a look through the CSV / B3D parser for a simple example. All of the world creation stuff gets done by the Object Manager, so all you need to worry about is creating a StaticObject with the appropriate mesh.

I've done a little more work on the PR I linked to above this morning, but it's not ready at the minute. Will try and do some more work later this week or next :)

[leezer3]

My replacement parser now appears to load most things I throw at it, including correctly applying the FrameTransformMatrix, which is still an issue with the Assimp version.

What I could do with now is some much more complicated examples with multiple Frames / FrameTransformMatrix to test on both, so we can see what is still an issue.

[s520]

I submitted an Issue on treeSet2.x. #425

I attached treeSet2.x converted to text format and text compression format together with sample program.

I will delete it if there is a problem.

[leezer3]

Already been there: https://github.com/adamhathcock/sharpcompress/issues/418

The best I can tell is that there's a bug somewhere in the Mono, Microsoft and Sharpcompress Deflate algorithms somewhere. All of these appear to have been based on Zlib 1.1.3, and the only real lead I've got is that a compiled Zlib 1.2.11 binary decompresses the data sucessfully. Unfortunately, the entire Zlib library was (essentially) re-written between v1.1 and v1.2, so that's not really helpful.

I'm guessing that there must be a subtle math issue somewhere, but I've never had anything to do with archive math. Probably something to do with the block length / block boundary. Might be better chasing this up with .Net / Mono, as they're somewhat lower level.

It'll probably have to stay as a broken edge-case. sample though, as I think that's the only file I've ever seen which does this. (Will check the rest of that route later, as I definitely fixed several bugs in the current X parser as a result of it)

[s520]

Of course, I know about #418.

However, I thought that it would be better to present example that actually succeed or program.

I found the following items. Deflating data from MSZIP format

I think that it will succeed in decompression of one data block, but if it exists more than two it will fail to decompress.

I succeeded in decompressing only if there is one data block.

So, I think that zlib's inflateSetDictionary() function is a point.

[leezer3]

From your link and some more digging, I think these must actually be MSZIP. There's also the nasty possibility that we may find some compressed with pure deflate or something from third-party tools.

https://msdn.microsoft.com/library/bb417343.aspx#microsoftmszipdatacompressionformat

[leezer3]

Been messing around with various C# deflate libraries today.

Haven't got any further really- They all appear to be based upon the same port from a Java implementation, and this is missing some crucial functions.

Assimp does the following with pure Zlib: https://github.com/assimp/assimp/blob/master/code/XFileParser.cpp#L241

::inflateReset( &stream );
::inflateSetDictionary( &stream, (const Bytef*)out , MSZIP_BLOCK - stream.avail_out );

The first call to reset puts zlib into it's default state & resets all buffers etc. inflateSetDictionary then sets the dictionary bytes to the last 32k of uncompressed data.

https://icsharpcode.github.io/SharpZipLib/help/api/ICSharpCode.SharpZipLib.Zip.Compression.Inflater.html#ICSharpCode_SharpZipLib_Zip_Compression_Inflater_SetDictionary_System_Byte___System_Int32_System_Int32_ All the C# implementations appear to require the previous return state of the zlib decoder to be Z_NEED_DICT

I had a brief attempt to brute-force past this check with DotNetZip, but that crashes the deflater elsewhere.

[leezer3]

Got it!

MSZip appears not to need the previous return state or the adler value checking before setting the dictionary.

My new branch now loads the broken X file correctly. It includes one further dependancy- Ionic.Zlib.dll , as most of the required bits in Sharpcompress are non-public (although near identical), and I didn't want to mess around fixing that up to work.....

Would be interested in further testing,

[leezer3]

A specific case I think we need a test for is a X file with 3 or more compressed segments (e.g. 64kb or greater when uncompressed)

[s520]

Congratulations!

Also, thank you for using my code.

However, some files still have errors.

I attach the file where the error occurs. test_files.zip

These files can be read correctly by the modified AssimpXParser.

I made it possible to choose from three parsers in #291 .

Please check it.

[leezer3]

Thanks, will try and get a look at those as soon as I can.

By the looks of the filenames, they're using slightly different compression types, which are presumably specified in the header bytes.

[s520]

These files are converted from one text format X file.

[leezer3]

OK, fixed the tzip version (forgot the header is not present in the decompressed data....), but the compressed binary may take a little more work to figure out.

From what I can see, it's got an embedded set of binary templates at the top of the file, which the parser isn't liking. Will do some more excavation and see what I come up with.

[leezer3]

Fixed the two binary examples in the latest push to the branch. Mostly an oversight on my behalf.

I'll do some more testing, and then port the branch across to the main sim and Route Viewer. I think the best way to do this (at least for the moment) will be to have a choice between the three available parsers, with a fallback to the original if a new one is selected but fails.

Will also probably want a setting for routefiles (so that you can deliberately force the new parser for BVE5 converted stuff), , e.g. Options.XFileParser 1

[s520]

I think it is a very good idea to add options to the route file.