Make a release?

[santagada]

I wanted to know if there is any intention on releasing this as boxer 2.0 in the future, and if there is any work that need help for that. Boxer is way to awesome to vanish or stop in the 2012 1.3.2 release.

[Kentzo]

:+1:

[cthielen]

+1

[stoz]

I am also interested in this.

[lavadrop]

I just compiled it and although it runs well, the performance is not up to par with 1.3.2. At least on X-Wing Collector's Edition and TIE Fighter CD ROM.

[JoshuaPettus]

really? runs fine for me. What cpu do you have?

[lavadrop]

i5 iMac. I'll try again later today.

[JoshuaPettus]

how old is the i5? I use to run the boxer dev version on my intel core 2 duo laptop and was ok.

[lavadrop]

1. I am running TIE Fighter CD ROM with SVGA graphics, SB16 emulation and General MIDI with output being directed to a real Roland Sound Canvas. There's a lot of tearing and some slowdown in certain areas. Current Boxer hasn't displayed such slowdowns under the same circumstances.

[bigfatbrowncat]

+1 Project that was last time released in 2012 looks as a dead one for a simple user. Isn't the current state of the project better that the released version? If it is - release is surely needed.

[Jarvik7]

It would be sad if this died. It's so much better than using vanilla DosBox on OSX. That said, it would be interesting if this switched to a dosbox-x core instead of mainline dosbox. There has been a lot of improvement lately that might make maintenance of this fork easier.

GoG needs to hire some staff to contribute!

[davidrenne]

Ya I wonder what version I have on my Mac if I downloaded from the website. I bet it is two years old. I think I will recompile from source because I have a feeling it's 1.3.2.

This project is too friggin awesome.

[alinebee]

My apologies for letting this issue languish without a proper response for so long.

The plain answer for why I haven't done a 2.0 release is that vital parts of the 2.0 codebase are still either broken, feature-incomplete, dated, or just plain worse than 1.3.2; that a fixed-up 2.0 would not represent a genuine improvement for users over 1.3.2; and that I have not had the time or the enthusiasm to hammer what's there into something that is a genuine improvement. There are a host of reasons for this, which I'll discuss in detail below.

---

First, some core features are still broken in the master branch: primarily game importing, which currently crashes if it tries to show a DOS installation window, owing to changes in the window model for regular DOS sessions. Those window changes were needed for the new full-window program launcher UI, which turned out to be an unfit UI design that needs to be rethought. That has discouraged me from fixing the game importing until a new design is in place, as otherwise it'd need to be fixed all over again.

As lavadrop noted, Boxer 2.0's rendering is noticeably laggier than Boxer 1.3.2's. I rewrote Boxer's renderer for 2.0 to use hardware shaders for rendering styles, which at the time I saw as a big win; but when I wrote that code I did not understand the foibles of OpenGL as well as I do now, and I made a number of mistakes with the rendering path that result in worse performance than 1.3.2. This is an unacceptable regression, and those mistakes are hard to fix without rewriting that whole subsystem again.

Aside from the renderer, I rewrote large swathes of the Boxer codebase to what at the time I considered modern coding standards. Most of those rewrites were more successful than the renderer's, insofar as they didn't make the application any worse, but they still left me with a legacy 32-bit-only Objective C codebase that has no migration path to Swift, nor lets me use modern ObjC features like ARC and automatic property synthesis. A codebase that will be worthless as soon as Apple decide to drop 32-bit binary support.

These rewrites did not even (though they were intended to) fix what are now serious structural problems. Boxer is riddled with hidden assumptions about view and controller hierarchy; key-value bindings all over the place that invisibly trigger action at a distance; deep coupling where all roads lead to the app delegate; implicit reliance on emulator events occurring in a specific order that not even I can keep straight; and so on. These problems stemmed from unwise design decisions I made as a neophyte Objective C programmer back in 2010, and are now blindingly obvious to me as an experienced developer. They have made Boxer's codebase brittle, hard to improve and extend, hard to debug, nearly impossible to write tests for, and increasingly frustrating to work on.

The rewrites also got me no closer to my ultimate goal, which is to split Boxer from a single-threaded app into separate processes for UI and emulation. This would allow a host of important improvements: like not stalling the emulation whenever the UI is interacted with, like being able to restart an emulator session without restarting the whole app, like finally being able to migrate Boxer to 64-bit, like being able to substitute dosbox-x or even another DOS emulator altogether. Instead, my 2.0 rework ended up wedding Boxer ever more tightly to DOSBox 0.74, making that goal even harder to achieve.

Meanwhile OS X 10.10 put a bullet through the head of the skeumorphic aesthetic that was so fashionable during Boxer's glory years. A lot of the UI work I did for 2.0 is now grossly tacky and would look immediately dated upon release. Modernising the look of the application is probably less work than it sounds - it's about taking away, rather than adding - but it's also extremely dispiriting to have wasted so much time on an aesthetic that is now definitively dead and buried.

---

So while I regard the 2.0 codebase as ultimately a failure, it does have a number of key improvements over 1.3.2:

The underlying launcher system (where a gamebox can have multiple named program-launch options, with custom parameters) is a lot better than 1.3.2's single 'default program' system. This will be the foundation of a simple and flexible launcher panel, once I figure out how that should be presented in relation to the rest of the UI. (The full-window launcher panel design that 2.0 currently has is not it.)

The game detection system is more robust and capable, theoretically allowing detection of multiple games within a single source, which will help with game disambiguation during importing (a big problem with Sierra CD collections, for instance). This capability is not currently used by 2.0's game importer, however.

There are classes to natively read ISOs and BIN/CUE images, which will speed up game scanning and will finally allow importing from BIN/CUEs. These are not currently used by 2.0's importer either, and it still needs a raw FAT image reader as well to provide full coverage.

The joystick controller subsystem has improved considerably behind the scenes, among other things allowing joystick buttons and axes to trigger key inputs. This requires a proper controller configuration UI to be of any use to anyone - a UI which has not been designed yet and will be a major amount of work.

There is drive shadowing, a feature which I am not convinced should be turned on by default but which is vital for some use cases (such as standalone apps, like those that GOG release). This needs a simple and clear per-game UI to control it, along with clear documentation on what it does and where all your savegame files have gone.

There is the documentation popover, which was the only wholly successful UI addition, and which provides an elegant way to view and add documentation to your games.

There is printer emulation, which is easy to use, fairly reliable, looks cool and is already benefiting a number of business application users. It is also cheesily skeumorphic; the presentation needs cleaning up to show more of what you're printing and less of tacky faux-printer chrome.

There are many new game profiles, and proper handling of OS X 10.9+'s accessibility permissions, and fixes for small but serious bugs like the import hanging forever because of busted .conf files.

---

So, why not port just the bits that worked back to 1.3.2 and do a release from that? Because I shot myself in the foot with all my code modernisation. The underlying code has changed dramatically, such that even the improvements I made to pre-existing features cannot be dropped into the 1.3.2 branch without significant rework. (This, people, is why you shouldn't embark on a Big Rewrite unless you're damn sure you can finish it in one sitting.)

Because of the structural problems I outlined above, I also don't want to put significant time into building anything new upon the 1.3.2 codebase. Boxer needs to be dragged kicking and screaming into the 64-bit era or it will be lost for good, and 1.3.2 is not the starting-point from which to do that. The most I would want to do from that base is a 1.3.3, that included the small bugfixes mentioned above along with Retina artwork. And that's hardly the grand update people would expect after 3 years of silence.

Thus Boxer sits wedged in a proverbial hallway, like the sofa in Dirk Gently's Holistic Detective Agency, unable to move forward or backward. And for the past few years my personal and professional life has left me with no time or energy to address that and decide what to do about it.

[JoshuaPettus]

Sad all around...and now I miss Douglas Adams too.

[stoz]

@alunbestor I appreciate your candid reply.

[HiPhish]

Thank you for the in-depth status update, that should be copy-pasted in the blog in my opinion, not everyone browses GitHub.

To me as an end-user Boxer is already amazing as it is, the only thing I really miss is the ability to map gamepad/joystick keys to keyboard keys. Joystick support in DOS is limited and some games need more than four buttons to be well playable.

Another thing that would be nice to have would be an option to make the mouse cursor linear per game. The OS X mouse acceleration is really awful for first-person games, and in most cases I can set up a linear acceleration curve in ControllerMate and have it turn on automatically when a particular app is running. But since all DOS games are running in the same app I have to turn it on or off for the entire Boxer.

[JoshuaPettus]

I know it must feel a total kick to the groin to start all over again off of the 1.3 branch, but at least anything you do to the 1.3 branch from here on out is a step forward right? :)

I suppose in reality, dosbox should be made 64 bit upstream. OSX will still able to run 32 bit for the foreseeable future. Granted, It would be nice, but I think the push should be made there. It would make your work easier.

[getaaron]

@alunbestor Do you think you could break the work that needs to be done into smaller, independent issues? I would enjoy starting to work on one. We could work together on a 2.0.x branch

[getaaron]

(And if not, could you list what we should keep if we decide to start from scratch?)

[JoshuaPettus]

Well for the record, I'm told the SVN Doxbox does compile with 64bit, but under performs for OSX relative to the 32 bit version.

[alinebee]

With regards to 64-bit: crappy DOSBox performance or not, it is absolutely essential for Boxer's future. I expect 32-bit binary support to be dropped within the next 3 OS versions - it's already been announced that OSX 10.12 will drop support for garbage-collected Obj-C applications (which Boxer, fortunately, never adopted). Apple have a precedent for dropping architectures: PowerPC binary support lasted three OS versions after the shift to Intel processors before it was dropped altogether in 10.7.

Beyond basic survival, 64-bit is essential for migrating any existing code to ARC and to Swift, neither of which support the legacy 32-bit Objective C runtime.

---

I should state that I have been working slowly on a Boxer rewrite in Swift for 10.10-era Cocoa. The goal of this is to design it from the ground up as a multi-process application, making use of modern AppKit tools, while avoiding the impenetrable kudzu of implicit dependencies that Boxer became.

However, I'm doing this largely as a learning exercise for myself, not for the sake of others, and I will not be publishing it until or unless it reaches a functional stage and gets enough momentum that I can see it through to completion (or is at least in good enough shape for someone else to do so.)

I'm painfully aware of the risks of such an endeavour, and do not wish to set any expectations for its success. I mention it only because it affects the amount of work I'm willing to do on the 2.x branch (which is zero) and the work anybody else should put into Boxer's current Objective-C codebase.

---

With that in mind: the least redundant thing anybody could do would be to update Boxer to DOSBox-SVN or DOSBox-X. Swift or no Swift, single- or multi-process, the next Boxer version would continue using the existing Objective-C++ wrapper to talk to DOSBox's C++ code. So that code is safe to touch and will make the biggest difference to Boxer's functionality.

Unlike the gordian knot of dependencies within Boxer itself, the interface between DOSBox code and Boxer is fairly well-defined: there are a set of C functions that Boxer has surgically inserted into otherwise-untouched DOSBox code to hook into events (or to just wrest control away from DOSBox altogether for particular operations). These C functions are all documented in BXCoalface.h, and they call into a BXEmulator Objective-C++ singleton that is responsible for managing the lifecycle of the emulator and talking to the rest of the application.

All of my modifications to DOSBox 0.74 code are well-documented and bracketed with comments, and so it should (ha) be straightforward to create a diff from DOSBox 0.74 and then apply that diff to DOSBox-SVN or DOSBox-X in order to make it 'Boxer-ready'. Of course, if there have been major structural changes in DOSBox-X then all bets are off; I haven't checked their code to see.

---

getaaron: The above may be one avenue you could help, but beyond this I feel like there are no independent issues that can be worked on in isolation. That's why Boxer got into this dead-end in the first place, and why I felt the best use of my time would be to chuck it out and start again.

If you would like to contribute to a Swift codebase once/if it goes public, then you are most welcome; if you would like to port some of the fixes and improvements I mentioned back to 1.3.x, that would lay the groundwork for a much-needed maintenance release; if you'd like to fork Boxer 2.x and rewrite the bits that are broken, then I cannot physically stop you. But as I said before, I feel like 2.x is just not good enough to release, and too old and convoluted to provide a suitable point to continue from.

[JoshuaPettus]

Personally speaking, I hope 32bit will be supported for a little while longer, especially as wine is limited to 32 bit for the time being. (wine64 isn't capable of executing 32 bit programs) Unlike PPC code, intel chips are more then capable of executing 32 bit code, so there isn't a good reason not too...
Anyway. I was speaking with the vogon's dosbox moderator, apparently the real bottleneck with 64bit is supposedly the dynamic recompiler. This is the case with all OSs though it is not known how bad. Why OSX is particularly effected with 64bit is a bug they haven't been able to sluth out. Perhaps all of your code replacement will fix that problem?

Here is the thread in question http://www.vogons.org/viewtopic.php?f=32&t=44122

[alinebee]

Doubtful, none of my changes to DOSBox go anywhere near that code. The dynamic cores are written in hand-tuned assembly, and perhaps the x64 version cannot be written in as efficient a way - I have no idea, and doubt I'll ever be able to understand that part of the code well enough to know. I do recall that OSX has a number of rules about e.g. stack alignment that differ from the conventions in Windows and Linux, so there might be some difference there that is disproportionately impacting the performance in OSX.

As far as the end-of-life for 32-bit goes: there wasn't a pressing architectural reason for Apple to drop the Rosetta translation layer from 10.7, which allowed PPC code to be run on Intel - but they did it anyway, so they wouldn't have to maintain that whole infrastructure anymore. As far as I know, supporting 32-bit binaries within a 64-bit environment requires the OS to jump through more hoops, and it also requires that every single OS framework has to be compiled with 32-bit binary support otherwise any 32-bit applications that link against them won't work - and that in turn may be holding back their own framework developers from writing modern 64-bit-only code.

The last OSX that supported 32-bit CPUs was 10.6, and Apple has been forcing all new app store submissions to be compiled for 64-bit for over a year now. Apple are not shy about phasing out stuff that has become a maintenance burden, and to them the price of dropping 32-bit support is that a few obscure or unmaintained cross-platform projects will stop working.

[JoshuaPettus]

Shame dosbox and wine are having trouble... both are quite important for online retailers.. It would be terrible if OSX gaming were to take that hit..

[HiPhish]

How can a program have problems with 64 bits? From what I understand it means the size of a word (largest amount of memory that can be read at a time). I also know that in C exact type sizes are not dictated by the standard, only the minimum size, so an int could be 32 bits large or 64 bits (or even 16 bits), depending on the implementation of the compiler. If a program relies on a particular size things will break. But that's also why C has fixed-size types like int32_t and int64_t and the sizeof operator. So if one recompiles the code for 64 bit, shouldn't if just work out of the box?

[MaddTheSane]

It's harder for assembly code, as you can specify a call (for example, div) that has different data types (divi, divl, divq), and calling, for example, a divi instead of divq can cause the CPU to do calls that reference data that is out of bounds of what is expected. The best case scenario is that the data is padded and the numbers are so small that it doesn't matter. Usually, this results in a program that does bad things and/or crashes.

Under DOS, this would cause the whole computer to hang, as it had little to no memory protection.

[HiPhish]

But DOSBox is not written in assembly, it's written in C++. Unless they were using inline assembly, in which case... why? I know this used do be done in the old days, but modern compilers can out-optimise a person.

[JoshuaPettus]

Please reread Alun's post 4 posts above.

[MaddTheSane]

So I got Boxer to compile in 64-bit mode. This was done by replacing fu_instructions_x86.h to a more recent version. Sadly, the frameworks included have been stripped of their 64-bit parts, making linking fail.

[MaddTheSane]

I've got it to link by updating/replacing the old frameworks.

[JoshuaPettus]

Fantastic, how well does it work?

[MaddTheSane]

It compiles, but the DOS side of things isn't stable.

[alinebee]

I unfortunately don't have time to look into this right now, but could you submit a pull request with the 64-bit changes and the destripping of the libs?

[MaddTheSane]

Okay, just plomping down the frameworks, new header, and a minor fix makes Boxer get to the command prompt. As opposed to my main branch maddsV2, which crashes.

Edit: Commander Keen 4 ran fine, but Dark Forces caused Boxer to crash.

[JoshuaPettus]

Another thing we should bear in mind is to use the code from the latest svn of dosbox, the last version .74 is over 5 years old, and a lot has been done on the 64bit front since then (never mind evreything else). Also keep an eye on the munt svn, a lot has changed since the last release file wise.

[MaddTheSane]

It does make me wonder: what are they planning for the next release of DOSBox.

On Dec 1, 2015, at 11:15 PM, Joshua Pettus notifications@github.com wrote:

Another thing we should bear in mind is to use the code from the latest svn of dosbox, the last version .74 is over 5 years old, and a lot has been done on the 64bit front since then (never mind evreything else). Also keep an eye on the munt svn, a lot has changed since the last release file wise.

— Reply to this email directly or view it on GitHub https://github.com/alunbestor/Boxer/issues/56#issuecomment-161190926.

[Zearin]

Perhaps GitHubs new Projects feature would be helpful for organizing and planning the work that needs to be done…?

[Njordy]

Well, the 32bit era end is in about a year. And so the Boxer's... :(

[MaddTheSane]

Good news is that I was able to get it to build under 64-bit on my 64-bitFix branch. Problem is, only basic/old DOS games run.

[Njordy]

That's something. Didn't expect a reply here :) Gives a smudge of hope in hopelessness.

[almeath]

It is a scary thought having to resort to running DOSBox in a Windows VM .. I hope that does not come to pass.

[HiPhish]

@almeath Or you could run DOSBox on macOS natively instead of using a Windows VM. Still not as good as Boxer, but better than what you proposed. You can download a pre-compiled binary from the DOSBox website or install it via Homebrew package manager.

[almeath]

@HiPhish, I thought that the Mac build of DOSBox, like Boxer, will not run properly in 64 bit? Of course, I would much prefer to see Boxer as a 100% compatible 64 bit app, but if there is a build of DOSBox (vanilla or otherwise) that is now fully 64 bit compliant please let me know where to find it.

[HiPhish]

@almeath Oh, I didn't consider 64-bit. Well, I found these instructions, but I haven't tried it myself: https://hexeract.wordpress.com/2016/09/10/building-dosbox-as-x64-binary-for-macos-sierra/

[almeath]

@HiPhish, thanks I will definitely try compiling and testing out that experimental binary, but it seems that it only offers limited compatibility, just like @MaddTheSane's 64 bit compile of Boxer mentioned above. The underlying issue being that the DOSBox code/core (on which Boxer partially relies) is fundamentally incompatible with 64 bit code.

It looks like our only hope is either the DOSBox team or its Mac porter will figure out how to get it fully 64 bit compatible, or Alun might hopefully have the time to work on 64 bit Boxer before Apple entirely drops 32 bit support in either 2018 or 2019 .. it would be such a shame to lose a brilliant app like Boxer and so if Alun cannot work on it anymore I hope someone else can pick up where he left off.

[PatTheMav]

@almeath The clock's ticking - Apple's next macOS High Sierra update will start warning users once (!) at launch of respective apps about their impending incompatibility with coming macOS versions.

So the writing's on the wall and I hate to be that guy, but it looks like Boxer's being sunsetted by Apple that way. 😕

(Also, IIRC wouldn't this also affect all (!) macOS versions of games on GOG.com that rely on @alunbestor's 2.0 branch or is that already 64-bit compatible?)

[almeath]

@PatTheMav Yes, it looks like Apple might force the transition quicker than we were initially led to believe. They said 10.13 would be the last version of macOS to support 32-bit "without compromises" but they did not elaborate on what those compromises would be. Possibly needing to download 32-bit frameworks manually, similar to Rosetta, or maybe aggressive warnings every time you open a 32-bit app. It is really hard to tell.

So I am going to start preparing for the worst case scenario; 32-bit support being entirely removed by 10.14 this September. Unfortunately, at the moment that means compiling a 64-bit build of DOSBox with degraded performance and re-learning how to manually construct and configure macOS application wrappers for each game. Quite messy and not nearly as elegant as Boxer. Probably very unstable and prone to crashes as well.

I am fairly certain that the GOG.com games using Boxer are 32-bit as well, otherwise Alun would have had no trouble releasing a 64-bit version of Boxer standalone several years ago.

Unless there is a lot of work secretly going on in the background, it looks like the end of 32-bit for macOS is likely to pretty much shut down Wineskin, Boxer/DOSBox and any games that rely on them to function.

[PatTheMav]

@almeath The process will effectively be similar to what happened on iOS - first no 32-bit apps are allowed on the App Store, next updates to pre-existing apps that have no 64-bit binary will be rejected before the OS itself pulls the plug. The warnings will start with the upcoming 10.13 update, so 10.14 will probably be the end of the road (and this applies to all (!) apps, not just AppStore variants).

This "issue" has been mentioned on the DosBox Forums as well, but the reaction so far reads like a collective shrug, but the way Microsoft is going with Windows 10 I wouldn't be surprised if they'll be doing something similar soon (or double down on 32-bit binary support for PR purposes).

But I agree, as it stands we seem to be SOL - @alunbestor has correctly called this way before time (https://github.com/alunbestor/Boxer/issues/56#issuecomment-121939301), but neither macOS nor 64-bit support were ever a priority topic for DosBox (understandable due to the dynamic core having been written in 32-bit assembly). Sure there are SVN releases (they use SVN, ffs) but the last official release is over 5 years old, so I'm more in a "it was fun while it lasted" mood.

At this point we might need for Dosbox/Boxer what happened to PyPy3 - someone taking a 6-figure sum to pay for development of modern releases/branches.. (*cough* GOG, Valve *cough*).

[getaaron]

I’m sure the money is there but it’s unclear who to pay it to. GOG etc. would pay some money but I doubt they want to hire full time employees for this effort.

[almeath]

The indifference on the DOSBox forum is not surprising. I know that some people will just say "don't upgrade" or "switch to Windows", but that is missing the point, which is to ensure the viability of DOS/Windows retro gaming into the future on the Mac platform.

I will hold out hope that something will get done as the reality sinks in that dollars might start to be lost. But I imagine nothing will be updated until everything breaks first. Hopefully businesses like GOG make enough money from this niche market to warrant the effort, sooner rather than later.

In the meantime, I will do some research into the performance differential between a 64-bit build of DOSBox versus running DOSBox builds in Windows 10 through virtualization in either Parallels or Fusion.

[MaddTheSane]

I've updated my fork to DOSBox svn HEAD over here: https://github.com/MaddTheSane/Boxer/tree/updateDosBox

Some functionality was removed: The file system code was changed enough that changes from Boxer don't integrate well. Everything else seems good, but I haven't tested anything.