dta parser - remaining issues

[volca02]

List of issues the dta parser still exhibits

• [ ] Extraction of sound files sometimes fails - DataFormatDTA::getFile() has length of a buffer to store data into, yet the compressed block loop fills it up and exceeds it. (Is there a reason the compressed data are copied into dstBuffer and then overwritten with decompressed data?)
• [ ] dta tool does not seem to be able to create directories (did not unpack music subdirectory of A0.dta)
• [ ] error reporting seems to be lacking

[drummyfish]

The goal is to eliminate the need for using the third party Mafia DTA extractor, so if anyone wants to tackle this, they can simply extract all files using the Mafia DTA extractor, then do the same using our extractor and focus on correcting files that failed to be extracted or differ from the reference (diff). Once we correctly extract all files, we're done.

EDIT: (This is just a note and is not supposed to answer the proposed question, I can't remember how the decompression worked now, will have to take a look at it again.)

[volca02]

I'll try to tackle this. Long therm approach should probably be not to need to extract the files, but that still needs the dta format parser to be without issues.

[drummyfish]

We wanted to have on-the-fly file loading from the DTA, but there were issues for example with OSG image plugin which can only read files based on filename, not from ifstreams.

I think having an installer that would extract the data before the first run would be an acceptable solution, it would also make modding easier. But if anyone comes up with nice on-the-fly DTA loading, we can use that, or even have an option to chose.

[volca02]

Okay, just looking at it really fast - there could be a way to do it via DTA file ReaderWriter plugin (similary to ZIP file plugin - ReaderWriterZIP).

[drummyfish]

Sounds like a possible solution - that would give OSG the ability to read directly from DTA, but non-OSG components would have to use some other method I guess and we'd probably want to not open and parse each file twice (for OSG and non-OSG), so that would have to be considered.

[volca02]

Yes, this is the case of either jumping in and using the virtual filesystem from the graphic library for everything, or the other way round - VFS in the main project, with a plugin to adapt it to look like a ReaderWriter to OSG. The project will likely need a VFS implementation anyway, for stuff like mods.

[drummyfish]

We don't want the whole project to be dependent on OSG, so we can't use it for everything, just the rendering. We have a VFS already, but now it only basically abstracts away the file locations on different platforms etc. It could be extended to support opening files directly from DTA (we kind of had that but then we scratched it when we moved to loading directly from files due to the mentioned problems).

[volca02]

So this can wait, it would seem. If there is a VFS layer and it can be adapted, direct usage of DTA files can come any time.