ObjectID subtype interpretation

[wsu-cb]

After previous discussions, I described the top 4 bits of ObjectID as a subtype field where value 0 represents the "invalid" subtype. However, after looking at ObjectID again, I am noticing that ObjectID's ID field can contain a PieceID (instead of a CMarkObj's random number), and that the subtype is 0 in this case. So would it be better to describe subtype 0 as representing a PieceID instead of invalid?

[DLLarson]

I'm still thinkin' on this...

[wsu-cb]

I'm still thinkin' on this...

Thank you for letting me know it hasn't been overlooked.

[wsu-cb]

To add to the headache here, I now realize that tag v3.5-prerelease-2 has CDrawObj::GetObjectID() return 0 as if that is the "invalid" ObjectID, but it looks to me like CPieceObj::GetObjectID() will also return 0 when CPieceObj wraps PieceID(0). That suggests to me that it would be better to use DWORD(PieceID(0xffff)) as the invalid ObjectID since that can't be mistaken for a valid Piece's ObjectID.

(Some of the type safety work I have already done is intended to make the ObjectID construction more explicit, but the actual final bit patterns should be unchanged by that work.)

[DLLarson]

Hi Bill,

That suggests to me that it would be better to use DWORD(PieceID(0xffff)) as the invalid ObjectID since that can't be mistaken for a valid Piece's ObjectID.... but the actual final bit patterns should be unchanged by that work.

I agree. I've been trying to historically wrap my brain around the whole scheme's evolution and I just can't seem to remember many the particulars that drove a decision.

Most of the "packed" data items are really the result of Cyberboard originally running on Win16 systems which had extreme limits on memory (real mode memory footprints), speed (plus no floating point), graphics (256 color then 16bit color) and even the number of font's that could be "realized" into memory at a given moment (hence the use of "atoms" to prevent redundant font realizations.) Much of these are no longer true limitations on modern platforms.

So this evolving of the codebase is really important to surface, segregate and compartmentalize those areas that had limitations so that CB can better take advantage of modern capabilities without breaking past work.

Thanks for your keen insight and applying your skills to ferret out the edge cases!

-Dale

[wsu-cb]

I have noticed that CGamDoc::CreateMarkerObject() and CGamDoc::CreateLineObject() use CGamDoc::CreateObjectID() to generate the high 4 bits (the subtype portion) of ObjectID, which results in the high 4 bits being CDrawObj::CDrawObjType() + 1. However, the ObjectID for Pieces effectively leaves out the + 1 logic, which is part of why the ObjectID's PieceID subtype is 0.

Should there be a PieceID variant of CGamDoc::CreateObjectID() (maybe explicit, maybe implicit), thus restoring the original description that subtype 0 means invalid? Or is it too late to try to change the ObjectID bit pattern, at least for file format version 3.90?

I started on this analysis as preparation for my goal of actually extending TileID/MarkID/PieceID to 32 bits, which will necessitate a new file format anyway. If you want to get back to the original statement of subtype 0 meaning invalid, I suspect the file format change gives us the chance to do so, even if file format version 3.90 still treats PieceID as having subtype 0.

On the other hand, from a long-term-consistency point of view, maybe we should say subtype 0 means PieceID, and subtype 1 is the invalid subtype. I'm pretty sure nothing can currently generate subtype 1, so I think it's safe to declare it invalid, even if it's a pretty counterintuitive rule to have 0 and 2 be valid, and 1 invalid.

[DLLarson]

The current legacy version of CB files is 3.10. The 3.90 version is/was intended for development work for the open source version that we are talking about. So the 3.90 version would be where we can correct these issues.

However, I had this, perhaps, naive view that we would get the open source code as close to parity with the legacy version as is reasonable. Build it and lock it down as an official release to replace the legacy version.

But in that process inconsistencies have been discovered in the some of the code that could be remedied right now with the 3.90 version to capture those adjustments. My greatest fear when locking down a file version is that, recognizing that there is no going back for those who use those changes, we create a dead-end version. This is always a risk but I've been fanatical at not having that happen.

Note that it's best to collect these types of changes into groups so we have as few file versions as possible--as each commit that changes a file format needs a new version number so they can be distinguished when loading.

As far as the much broader changes to file formats you're working toward, I think those should go under a major file version up-tick like 4.00.

As all this is making my brain hurt right now ;) I'll support any decision you make that takes the above concerns into consideration.

All that being said, I will say that providing an official 4.00-ish release that has actual feature additions like you are after might be just what this project needs to get people off the fence to actually try out the open source version!

-Dale

[wsu-cb]

The current legacy version of CB files is 3.10. The 3.90 version is/was intended for development work for the open source version that we are talking about. So the 3.90 version would be where we can correct these issues.

I haven't done anything (yet) that is intended to change the file format. If I did change the file format, then it was a bug. So is 3.90 actually binary identical to 3.10? I.e., have you already changed the fie format?

However, I had this, perhaps, naive view that we would get the open source code as close to parity with the legacy version as is reasonable. Build it and lock it down as an official release to replace the legacy version.

As I have been working, I have discovered various bugs, and we have fixed most of them. Of the ones left, I'm afraid the WinState.* problem is too small an annoyance to me for me to be motivated to fix it. Also, I don't have high DPI hardware capability, so that problem doesn't bother me at all, and I have no way to work on it. Aside from those, I am not currently aware of any problems aside from the help file graphics being out of date. What is holding you back from an official release? I assume the high DPI problem also exists in CB 3.1, so it looks to me like you pretty much do have parity.

But in that process inconsistencies have been discovered in the some of the code that could be remedied right now with the 3.90 version to capture those adjustments. My greatest fear when locking down a file version is that, recognizing that there is no going back for those who use those changes, we create a dead-end version. This is always a risk but I've been fanatical at not having that happen.

Note that it's best to collect these types of changes into groups so we have as few file versions as possible--as each commit that changes a file format needs a new version number so they can be distinguished when loading.

Broadly speaking, I am inclined to agree that we should limit the number of file formats. (For example, see my comment at Tile.h line 63.)

However, I am currently thinking that I can make changes to the in-memory representations (e.g., use 32bit TileID/MarkID/PieceID), but still, for the moment, use the existing 16bit file format. Basically, just convert the 32bit ID to a 16bit ID before storing it. I would have to throw an error if a large ID would be truncated in the file, but that would allow for some testing/validation of the new data types while still postponing the file format change. Obviously, the testing would be incomplete, but I think it could still provide some initial validation of the work. (Actually, I am even thinking about making the file (output) version configurable.)

As far as the much broader changes to file formats you're working toward, I think those should go under a major file version up-tick like 4.00.

Isn't the file version really only internal? As far as I can tell, file versions could just be counted 1, 2, 3, etc., rather than having major/minor designations.

As all this is making my brain hurt right now ;) I'll support any decision you make that takes the above concerns into consideration.

All that being said, I will say that providing an official 4.00-ish release that has actual feature additions like you are after might be just what this project needs to get people off the fence to actually try out the open source version!

Possible issues:

1) Has the hiatus caused most users to switch to Vassal? I feel like almost all of the "new gamebox" notices I see for GMT Games are Vassal, not CB.

2) Do users want to use Beta software? I generally don't, but I am all too familiar with how the sausage gets made. Even "release quality" sausage. There may be a catch-22 here of not releasing untested code, and not getting the code tested because it's not released.

3) I'm afraid the only user-visible new feature I expect in the near future is support for very large (32bit id) .gbxs, so I'm afraid that won't interest many people. With luck, the people at http://www.starfleetgames.com/discus/messages/37/41064.html?1595293550 might be interested. I have ideas for extending geomorphic boards, many-sided pieces, and "clonable" pieces, but those are all probably far future (i.e., after 32bit ids and 64bit code). Basically, I want to modernize the infrastructure before getting into "new" work. I don't know where multi-platform fits in. There is a severe problem there (https://trac.wxwidgets.org/ticket/14415).

-Dale

[wsu-cb]

As all this is making my brain hurt right now ;) I'll support any decision you make that takes the above concerns into consideration.

I'm currently leaning towards declaring that ObjectID subtype 0 is PieceObj, subtype 1 is Invalid, subtype 2 is MarkObj, and subtype 0xF is (GameElement's) MarkID. That appears to me to be the actual current state, and it seems like it should work just fine in the future, even if the order is somewhat counterintuitive. I will add a new enum in ObjectID to make these definitions explicit.

[DLLarson]

So is 3.90 actually binary identical to 3.10?

Yes. If we set it back to 3.10 any file we saved for testing would still raise an alarm but only because the stored version is newer.

I.e., have you already changed the fie format?

I don't think so... Not intentionally anyway. I bumped the version in anticipation of possible changes needed to remove XtremeToolkit but I don't think any ultimately needed a file format change.

the WinState.* problem is too small an annoyance to me for me to be motivated to fix it.

I'm currently thinking it should be removed completely. That would change the file format though. If it's interfering with app visual layout the code could just ignore the data upon load and not save it when writing out the file while still keeping the same file format.

Also, I don't have high DPI hardware capability, so that problem doesn't bother me at all, and I have no way to work on it.

This one can be ignored for now as there are EZ workarounds and not many people have 4K screens.

What is holding you back from an official release?

Fear of the unknown. ;) I also need to break the links with the legacy site (brainiac) currently hosting the legacy website and files. by redirecting them to GitHub. That will probably be an irreversible action.

Isn't the file version really only internal?

At first the file version info was published in the About box but I think I removed that in the open source version.

As far as I can tell, file versions could just be counted 1, 2, 3, etc., rather than having major/minor designations.

True but that's not how it's been done. Originally there were version numbers that synced with the program version but that isn't really true anymore. I'm inclined to not add to the confusion and change this now.

However, I am currently thinking that I can make changes to the in-memory representations (e.g., use 32bit TileID/MarkID/PieceID), but still, for the moment, use the existing 16bit file format.

That's an excellent approach!

(Actually, I am even thinking about making the file (output) version configurable.)

Sounds intriguing! How would this work?

Has the hiatus caused most users to switch to Vassal? I feel like almost all of the "new gamebox" notices I see for GMT Games are Vassal, not CB.

There is some truth to that. I think the major attraction for Vassal is real-time play. CB doesn't address that area of play. People are willing to screw around with various Java issues because it provides that valuable feature.

CB is single executable program with no other runtime dependencies but it's original aims are still dominant: PBEM play. People like live play. On the other hand CB has a very large game library:

http://limeyyankgames.co.uk/cyberboard?items_per_page=100

A live play capability that was lean and usable would be a major boost for CB. So would the addition of a user programming subsystem to add automation, etc... to games.

Do users want to use Beta software? I generally don't, but I am all too familiar with how the sausage gets made. Even "release quality" sausage.

A very good point.

I don't know where multi-platform fits in. There is a severe problem there (https://trac.wxwidgets.org/ticket/14415).

I see what you mean. Kind of essential to have that working.

I'm currently leaning towards declaring that ObjectID subtype 0 is PieceObj, subtype 1 is Invalid, subtype 2 is MarkObj, ... and it seems like it should work just fine in the future, even if the order is somewhat counterintuitive. I will add a new enum in ObjectID to make these definitions explicit.

Sounds good!

-Dale

[wsu-cb]

However, I am currently thinking that I can make changes to the in-memory representations (e.g., use 32bit TileID/MarkID/PieceID), but still, for the moment, use the existing 16bit file format.

That's an excellent approach!

(Actually, I am even thinking about making the file (output) version configurable.)

Sounds intriguing! How would this work?

For the sake of argument, let's call 32bit ID files v4.00. Then, each time the output code is about to write a 32bit in-memory ID, it checks the output version. If the output version is >= 4.00, it writes the in-memory 32bit IDs in the obvious way. If the output version is 3.90, then check whether casting the 32bit ID to a 16bit ID is value-preserving. If not, throw an error. If so, write the 16bit ID in the obvious (CB3.1) way.

So then the question becomes how to determine the output version. I see multiple options: 1) Have a compile time switch that decides whether the system outputs files in the v3.90 or v4.00 format. 2) Have the system default to writing v3.90 format, but add a command line switch (e.g., --id32) that sets the output file version to v4.00. If --id32 is not set, and a v4.00 file is read, report a IDS_ERR_GAMEBOXNEWER error. 3) Have the system track the output version on a per-CGamDoc basis. The system writes the file as the same version it had when it was read. Add a command line switch (e.g., --id32) that overrides the read version, and makes files always be written as v4.00, to provide a way to convert files to the new version. 4) Have the system track the output version on a per-CGamDoc basis. The system writes the file as the same version it had when it was read. Add a File | Upgrade menu option that changes the CGamDoc version to v4.00 to provide a way to convert files to the new version.

I don't like 4) because I feel like it clutters the user interface with something that will not be used once the initial flurry of upgrades is performed. I don't like 1) because it's too inconvenient for us to switch between versions. I'm not sure whether I prefer 2) or 3). 3) seems more convenient but possibly more subtle/confusing.

It's possible the command line switch needs to be --v4.00 rather than --id32 because the next file format change will always include 32bit ids. It doesn't seem reasonable to make file-format changes be piecemeal rather than cumulative. (E.g., if the next upgrade is pieces with 128 sides, all gameboxes with pieces with 128 sides will also have 32bit ids.)

A live play capability that was lean and usable would be a major boost for CB. So would the addition of a user programming subsystem to add automation, etc... to games.

Both would be great, but both will take a lot of work. I would think trying to postpone release of the open source version until either of those features is available would mean a very long delay. Even adding a beta version of one of those features in order to attract beta testers seems like a large undertaking.

[DLLarson]

So then the question becomes how to determine the output version. I see multiple options:

I like (2) best because most people can just use the program without any additional information. It just works. Those working on testing the newer version would know to do that magic incantation to produce the newer output file format.

Both would be great, but both will take a lot of work. I would think trying to postpone release of the open source version until either of those features is available would mean a very long delay.

Totally agree. It's just me imagining what things would look if a switch was thrown and the features existed. Either feature is probably about the same effort--lots.

Perhaps automation would be a bit easier but it requires an external object model be defined to surface capabilities.

Real time play is difficult because it involves external communication services that are unreliable plus a host of synchronization problems. Then there's an additional approach of creating real-time play using the automation feature if it's robust enough.

-Dale

[wsu-cb]

Totally agree. It's just me imagining what things would look if a switch was thrown and the features existed. Either feature is probably about the same effort--lots.

Perhaps automation would be a bit easier but it requires an external object model be defined to surface capabilities.

Real time play is difficult because it involves external communication services that are unreliable plus a host of synchronization problems. Then there's an additional approach of creating real-time play using the automation feature if it's robust enough.

-Dale

My gut says live (aka real-time) play is more important than (and should therefore precede) automation. Of course, I am still advocating infrastructure (32bit ids, 64bit code) first.

Something like 128-side pieces isn't trivial, but doesn't seem as big as either live play or automation, so might precede (or even run in parallel with) the big ones. I have several games (Roll for the Galaxy, Alien Frontiers, Quantum, Deep Space D-6) that use dice as pieces (not just custom dice like War of the Ring), so multi-sided pieces is valuable to me personally. Also, I have Space Empires, which doesn't use dice as pieces, but it would probably help play if the ship markers had six sides to cover all legal ship-count values.

[DLLarson]

My gut says live (aka real-time) play is more important than (and should therefore precede) automation.

I agree. Automation would only appeal to a small set of game developers whereas real time play is focused on actual players--a much bigger group. It would move CB closer to vassal. I think vassal provides automation by using the underlying Java language--although I could be wrong--so it is somewhat "free".

-Dale