8-bit BITD issues

[MrBrax]

So the converter doesn't work properly for the files i have, it skips a third of the height, and the length of the files does not correspond to (width*height) so there has to be some kind of compression (RLE?).

I've almost completed a python script that can extract text, WAV & AIFF, and BITD is the one i'm stuck on, been working on this thing almost every wake hour for a couple of weeks now, and just found this project. Really interesting stuff!

Since the converter only has 24 & 32 depth selectable, I'm assuming the format hasn't been deciphered yet?

oh and also, exported bmp's from director all have the same palette (in-game images always have palette set to -1) so i'm thinking the bytes aren't corresponding to colors but palette positions?

[MrBrax]

I almost have a working 8-bit extractor now, just gotta fix it leaving out some pixels at the end

[tomysshadow]

I haven't had a chance to look at a lower bitdepth BITD yet, that's all. However from what Brian tells me the lower bitdepth BITDs use colour palettes in the form of Photoshop's CLUT format, which are in different chunks. Multiple can exist in one file, and I'm not sure yet how those BITDs are linked with their CLUT. Either way, I'll add it in at some point - the BITD Reader is still quite new.

[MrBrax]

It's quite an interesting format. It uses linear & rle encoding in the same way, but with palette entries instead of rgb yeah.

My script can put out proper bitmaps of most bitd files, but some get distorted like so:

Setting the X coordinate to (length - j) in the loop fixes it:

But breaks all the other files. Must be a value somewhere defining this.

[tomysshadow]

That distorted image looks to me like the width is just one pixel off. I'm going to have to try implementing 16-bit images, to see if I get similar problems. It may be something implicit. Perhaps the rightmost pixel is getting cut off because it's all the same colour. If you hadn't already noticed, alpha values are sometimes reversed if alpha isn't used in a 32-bit image.

I am aware of how the format works. I figured it out in a hex editor in order to write SeismoGRAPH.

[MrBrax]

Figured it out with a tip from a friend. If the width is even, go to x -1 instead of 0 on the line change! Works great, using the palette from a BMP extracted with Director. Transparency is gonna be a pain though, since it's on the sprite itself and not the bitmap.

[tomysshadow]

So you're suggesting the width is always even and if it isn't an extra column of pixels is added? I'll have to add that in. I wonder what your code looks like in comparison to mine, it's probably more advanced considering it is actually inspecting the entire file. We're currently in the process of switching over to using Haxe instead, since with JavaScript which we've been using thus far we won't be able to load Xtras. That's why SeismoGRAPH is disconnected from everything else.

If the image is 32-bit then alpha transparency is just another channel before the red, green and blue. If it's using JPEG compression, which is completely different of course, it's in a separate ALFA chunk.

[MrBrax]

It's difficult to explain. I uploaded my script here: https://gist.github.com/MrBrax/1f3ae06c9320863f1d7b79b988c03e60

[Brian151]

palettes are generally linked by their cast member ID. However, there's an issue I've had with specifying a palette from another cast. (fun fact: this seems to crash Director when attempting to edit said bitmap) The linkage is done in the bitmap 'specificData' block of a bitmap member. the CLUT chunks should be linked to their respective palette cast members like everything else? The linking of CASt header chunks (i won't call them meta-data because the INTENDED meaning of this is "extra data not needed by program", CASt are quite important, and specify the type of information typically found in a header) to the actual data chunks is something I need to start working on again...

As for the bitmap data itself, my efforts can confirm that at least the 8 bit BITD CAN be uncompressed. I have yet to explain how it differentiates between the two, or what would cause it to use either. One of my sample files (I imported MineCraft textures) compressed the 8-bit bitd. I later tried to create an 8-bit bitmap from something that actually would benefit from compression. No such luck. Director outright REFUSED to compress the bitmap. I tried many times and got raw indexed 8BPP data in my BITD. On my compressed sample, I tried finding any indicator how that worked, and no particular luck. And then my desktop started acting up. I suspect hardware failure is really the cause, at this point. Anyways, I'm back on my parent's computer. No Director (or a lot of my latest progress/samples), but Git works, at least.

All I'll say about BITD's is I HATE them... I just can't come to understand them, and I'm glad you two can. embedded resources are easy, W3D will be a challenge to parse/display, and the CLUT things, well... only thing I don't understand is the second value. But, I feel confident I can ignore it for the time being.

[MrBrax]

It's the worst! I've been trying to extract all the CXT/DXR resources from a game and it's coming along pretty good, except now some images come up corrupted (png are converted, ignore):

And just as i'm writing this, i'm noticing that this: https://github.com/Brian151/OpenShockwave/blob/master/docu/FormatNotes.txt#L235 could infact indicate if the image has alpha somehow, i dunno really.

There's also the "Sprite" object thing, is that bound to the CAST, or is it a separate object somewhere? Would maybe explain the "Ink" feature and more. scratch that, they use the "score" list thing

[Brian151]

alright

This observation is among the progress I have lost. Sadly, I once again can't actually check that kind of stuff out in Director, so...

[MrBrax]

Can't you install the Director trial? But i guess if it's not your PC then it's not very appropriate.

I think there's something wrong with the Director UI or dirOpener's conversion of the files, because except for a few bitmaps, it says it's 8-bit and no alpha, but because i can't convert those files like i can with all the other ones, there has to be something different.

[MrBrax]

My god, it was this line all along, that i used: https://github.com/Brian151/OpenShockwave/blob/master/tools/BitdReader/js/BitdReader.js#L72

I changed it to 0x100 and now it works.

[tomysshadow]

It shouldn't? I specifically had to use 0x101. Does it work in all cases?

[MrBrax]

Yes, but it might be for 8-bit only. 1-bit bitmaps are still corrupted (which i haven't gotten to yet).

[Brian151]

It's a matter of disk space and CPU/RAM that I won't... Pretty sure this computer couldn't handle it. As is, my computer kinda struggled running it.

[MrBrax]

Really? It's a pretty simple program in my opinion, i use MX 2004 (10.1) and it's like 80MB installed

[MrBrax]

Anyway, here's something fun:

161: 13x16, 1bpp

00 E0 07 20 38 20 20 10 10 10 10 10 10 08 20 38 20 40 40 40 40 80 80 80 81 00 61 00 19 00 06 00

and

181: 16x13, 1bpp

00 0C 00 32 39 C2 26 02 20 01 40 01 40 01 80 0E 80 30 80 C0 71 00 0D 00 03 00

Is it using every byte multiple times or something like that? I can't see how it would work otherwise.

[tomysshadow]

For such a simple image I don't see any semblance of a pattern. What happens if you change the bytes to slightly different values? Experimentation is the easiest way to figure this out. Just make a backup of the original.

[MrBrax]

So from what i understand it now, every byte corresponds to a different pattern in 8px wide chunks, that's more of a clue.

[tomysshadow]

Each byte is 8px wide? Is it a bitmask then? That's what first comes to mind.

[MrBrax]

yeah i suppose, i haven't gotten that deep in binary stuff yet, so i barely know how that works haha

[tomysshadow]

Every byte has eight bits, it's likely that one bit represents one pixel. That'd make sense for 1bpp. That's super easy to implement, so I should add it in right away.

[MrBrax]

There we go, with some help of bitstring!

Director itself can't export 1-bit images, quality stuff.

[Brian151]

The 1BPP most likely is 8 pixels to a byte, this would be the ideal way to implement 1BPP IDK if it'd use compression or not. It technically could, but most likely only if a whole byte can be repeated

As for exporting 1BPP, I'm pretty sure anything lower than 8 BPP is fairly non-standard in graphics. About the only machines that natively support things like 4BPP and lower are older computers, and more specifically, game consoles, like the NES or SNES. I may be mistaken, though. Most PCs use 24 and 32-bit, and ofc, there's now... HDR. That said, there's no practical advantage, or at least, no perceived one, in Director exporting something lower than 8BPP. None of the 'standard' programs would be able to read it.

[Brian151]

on director installation: 30 GB disk capacity, usually less than 1 GB free The CPU and RAM are most likely, totally insufficient. My desktop was upgraded, and Director still is a resource-intensive program to run, especially with larger files. I don't feel safe risking that. As-is, Firefox uses on average, 70-80% of the CPU, and at least half of the RAM.

[MrBrax]

To note, the CAST entries for the 1-bit images are different than the rest, the data ends after the regx/regy values, so it does not have that unknown byte and bit depth.

Ouch, that's bad.

[tomysshadow]

Well, if anything that at least gives us a clue that the unknown byte has something to do with something 1bpp images don't have.

I implemented 1bpp images but I'm having trouble pushing the commit, I'll try again later.

[Brian151]

It would make sense for 1BPP images to lack the palette entry, as they have no concept of color. The lack of the bit depth is kinda odd, though

Well, good to know. And overall, I do not feel this makes the BITD's too much harder to deal with. Have 1BPP images been encountered in any color besides black? From my understanding, the typical process for colorizing a 1BPP would be the programmer specifies a color for it to display as when it's in use. Technically, it can have two colors: FG and BG, but the BG is usually the stage BG, or made to appear transparent. Not sure how Director handles this, I can't find any information about how Director deals with lower BPP besides it can import them.

[MrBrax]

Well, the "foreColor" sprite attribute can be used to color it.

And with the "ink" attribute you can make it transparent.

[Brian151]

I'll have to look at that stuff later

[MrBrax]

damn, some bitmaps still come out as corrupted. changing it back to 0x101 fixes it by a tiny bit though

should i put my python script in the tools folder maybe?

[Brian151]

We'll figure it out soon I guess

You could? How well is it commented/documented?

[MrBrax]

Not very well, but i think the code is pretty self-explanatory. More for sharing my interpretation of the file format than being a proper tool.

[tomysshadow]

If you look at ProjectorRays you can see I'm aiming for something a bit more advanced when it comes to reading in the file format. It might be of assistance though. A lot of it will eventually be pretty straightforward since stuff like sounds are just using pre-existing standards. I say go for it.

Oh, and I should mention that our tools are under the Apache License 2.0, so be sure to specify otherwise in the whitelist if you want.

[MrBrax]

i don't mind at all what happens to my stuff, as long as it's not proprietary and obfuscated by obscurity, like corporate stuff :)

[Brian151]

proprietary code and obfuscation has definitely done immense damage to the overall environment of computing. The only place I feel maybe code should be closed is in matters of security. But even then, there's openSSL, so...

When you think of it, proprietary code is what's led us where we are, now. Trying to pick-up the pieces of another failed example of corporate software.

I'll review your pull request later, have some rather bothersome things on my mind...

[MrBrax]

So i'm back at it again with this corrupting issue. There has to be a value somewhere that defines if it's linear/run length encoding, because it's either fixed or at least less corrupted if i force linear on it:

These follow a completely different pattern, throwing the "0x101" stuff off.

Scratch that, it's just palette values, no length descriptors

[tomysshadow]

MrBrax, please refer to chunks such as Lnam by their big-endian names, thanks

[MrBrax]

@tomysshadow i'm just following the styling of the other entries in that file, Lnam should be the correct name

[MrBrax]

Reworked the python script massively and it now outputs cast libraries. Got palette output, but haven't found how the cast selects it, so it's just referring to the palette cast in the code. Fun stuff!

[Brian151]

https://github.com/Brian151/OpenShockwave/blob/master/docu/FormatNotes.txt#L238

Aside from some edge cases, palettes are linked via their cast member ID in whatever cast they belong to From what I gather, using the KEY section, you can associate a given palette's CASt with its respective CLUT section.

you may also want to look at: bitmapCastCompare.txt bitmaps002.txt both are in the examples directory

I might need to review the specificData thing, as information about palette linking might have gotten compromised by an edit, somewhere...

*

1. Some palettes are not stored in the movie, namely, the system defaults
2. I have had some issues when referencing a palette from another cast. It's possibly not intended, but it does seem Director is somehow storing that information (or trying)... However, I haven't figured-out where it's storing that information.

That said, we already have pretty much everything we need to link bitmap cast members to the palettes they use.

[MrBrax]

@Brian151

I have had some issues when referencing a palette from another cast

this is exactly what i'm trying to figure out, does it support that at all? because yes i know about the palette value and it seems to match the cast id, but if it can use palettes from other libraries then there's gotta be a lookup table somewhere, right? and what about the built in palettes? i can't really find the value where it's defined if used

[Brian151]

It allows a palette from another cast to be specified... No idea where it puts the information to use that palette.

As I said, Director crashed any time I tried editing a bitmap where a palette had been specified like that...

The most logical way (which I'm guessing Macromedia/Adobe didn't do...) would be: Uint16 castID Uint16 palMemberID

From all my observations, the only thing that changed was the 0XFFFF before the member ID But, it only ever changed to 0x0000 Again, I don't have those examples right now... >.<

Perhaps in this case, the information is stored in the main part of the CASt? IDK...

Anyways, I'm going to shower, then I'm off to bed Only up now cuz I got caught-up browsing things

[akelagercrantz]

For anyone stumbling on this thread as I did, I made some updates to the script from @MrBrax and published it as a Gist (since this repo seems to have been deprecated for a set of other repos that no longer contain the script).

[MrBrax]

@akelagercrantz I worked on that script a bit more after I posted it, here's an even newer one: https://gist.github.com/MrBrax/75a23fe8d3b404df5a8041364d5774d8