RFC: Proposal for a full code rewrite fork

[ideafarm]

I'm toying with the idea of forking and then using that fork to comprehensively polish the code for readability. Initially, this would involve global replacements of identifiers. Within that, the initial focus would be to globally replace those identifiers that consist of common words such as "release". The initial goal would be a code asset in which every significant identifier is unique, so that a global search for the identifier would hit only true uses of that identifier. Insignificant identifiers such as names of local variables used as temporary buffers would be renamed to indicate type but would not be renamed for uniqueness.

Renaming identifiers would immediately effectively disassociate the forks in that it would become impossible to automatically merge commits from one into the other. So such a fork should not be done while the base (open-watcom/open-watcom-v2) is being actively enhanced.

In considering a fork, I am motivated primarily by my perception that this base fork's development appears to be unconcerned with making sure that OW works well on Windows, which remains the dominant personal computing and server platform. Over the years, I've watched the debugger, WDW.EXE, become buggier and buggier, and as of this writing it is so buggy that it is almost unusable.

For such a fork to be successful in promoting use of OW, it would have to be friendly. Mr. Malak and the others who have contributed over the past 20 years have deep familiarity with the code, and preservation of this knowledge is critically important to preserving and growing a community of users. However, Mr. Malak has consistently refused to respond to my requests over the years for information about what his vision is for OW, both technically and in marketing terms (how he envisions promoting the use of OW).

My sense is that OW is being kept alive by its continued usefulness to people running DOS applications. If so, I'd like to discuss how we can preserve OW's support for DOS while refocusing the marketing vision to what I think is the most promising future for OW, which is to focus on serving the Windows and Linux targets for 32 bit applications and, once that is debugged and polished, to complete Mr. Malak's work on 64 bit code generation.

I write all of this in the hope of having a friendly exchange of ideas on how OW is currently being used and how people would like to see OW develop, both technically and in the market.

[rdos314]

I disagree to most of this.

First, I have no interest in the DOS, DOS-extender, OS/2 or any other long dead systems that OW still supports. I suspect people that have interest in this mainly are using the original 1.9 version. These dead systems don't benefit from new code, and I suspect that's part of the reason why development of the 1.9 version stalled.

Second, I disagree about OW having poor code quality or being very hard to understand. I've tried to add support for RDOS to newlib, something that didn't go well because of a very strange software model. Doing the same with gcc and libc proved to be a very hard thing to do. I'm pretty convinced that GCC is harder to understand than OW. You need to understand that OW is a very complex software project that you cannot just start to modify or fix bugs in as a complete novice.

I'm moderately interested in Windows and Linux support. I want the debugger & IDE to run on new Windows versions, but that's it. I'm very interested in a 64-bit version, and if completed, I might use that for continuing development on a 64-bit RDOS version.

So, in conclusion, I have no interest in a new fork of OW.

[ideafarm]

That's helpful; thanks, Mr. Ekblad. I have a grand vision in which OW plays a crucial role and in which the development of OW grows beyond the "one man show" (Mr. Malak) that it appears to have become. The first priority for me is to restore quality for 32-bit Windows host and target; I want the debugger to be totally debugged so that it works perfectly. Once that is accomplished, my second priority is to develop a web site that markets OW aggressively by clearly articulating (truthfully) that OW is the best tool set for developing multiplatform C++ applications and for teaching C++ programming. Mr. Malak's work adding 64 bit host and target support excites me but is orthogonal to these priorities. The immediate concern for me is that OW's marketing and market penetration appears to be nonexistent.

Now, the context. My motivation for investing time and thought into OW is that I envision releasing and deploying an aggregate product that consists of OW tightly coupled with IPDOS (tm) IdeaFarm (tm) Piggyback Distributed Operating System. The latter will be released as copyrighted open source, i.e. as a fully proprietary product for which the source code is disclosed and for which code development is driven both top down (proprietary management and control) and bottom up (a community of open source user/contributors). IPDOS (tm) will complement OW, and vice versa, with each multiplying the value of the other to C++ developers and to end users.

In a perfect world, all of this would be accomplished without forking. But Mr. Malak has had years of opportunity to show an interest in my vision for OW and for responding to my occasional invitations for him to tell me what his vision is and for allowing me to become acquainted with him. Although he's been responsive and polite, none of those conversations have occurred, and I've seen no indication that he is interested in the marketing and business side of taking OW forward to its full potential.

At this moment, without more information, I see Mr. Malak as a valuable technical resource who is (I am guessing) focused on adding 64 bit support and who is approaching OW as a code developer rather than as either a software architect or a marketer or a CEO with a business/strategic vision. Unless Mr. Malak is willing to yield his personal control over the overall development of OW and become an individual contributor in a larger scale, organized effort that is driven by a marketing and business vision, I will have no choice but to fork OW at a hopefully optimal point when his 64 bit work is completed and we have a stable and functionally complete baseline for vectoring off in the new direction that I envision.

The first step in any of this is for me to encourage everyone to participate in a conversation about where each person wants to see OW go, and about how each person is currently using OW and about what needs to be done to ensure that current OW users aren't alienated by any new direction or vision.

[rdos314]

RDOS is running a commercial product on a couple of thousand installations right now. That's enough for me and I don't have any further plans to expand its use into new environments. It was initially (1988) a fun project. At this moment it still runs on modern CPUs, but due to neither Intel nor AMD any longer caring about segmentation, it has a bit bad performance on modern CPUs. Writing new 64-bit drivers might fix that problem, but it will be a huge undertaking that I'm not sure if I'll complete or not. It also depends on a 64-bit compiler, or at least a 64-bit assembler compatible with WASM.

[ideafarm]

What's the status of the 64 bit effort for OW?

I looked at your web site. I tried to look at the "RDOS (Ekblad)" page on Wikipedea but it appears to have been deleted. (If you have thousands of commercial installations of your OS, you certainly should be able to have a page there, IMO.) Since you wrote RDOS in 1988, we are probably about the same age. I used my first Watcom (University of Waterloo) compiler in 1973. I have a special love for OW partly because I began my programming career using a compiler produced by the same organization that eventually produced the very same tools that your RDOS and my IPDOS are built with.

[rdos314]

RDOS didn't use Watcom to start with. It was built with Masm using the DOS executable format with no relocations. Later, I switched to Borland's Tasm instead. Sometime in the 90s I started emulating Win32, and eventually made a good enough emulator to fool Borland's debugger it actually run on Windows. A few years after Open Watcom was launched I got interested in it, mostly because it could support the memory models I used. I then made the first native environment and stopped using Win32 emulation.

[ideafarm]

Thank you so much for telling me a bit about yourself. I traveled a similar path, although at an early point in my coding career I had the opportunity to write satellite telemetry code in Assembler, and another job doing healthcare application programming, and I decided that I was going to be an applications programmer rather than a systems programmer. In retrospect, it was a terrible career choice, but it ultimately proved to be the right choice. Ironically, I ended up writing a "piggyback operating system", which is technically just a Windows application (a service), but conceptually a true operating system.

At the moment, I am learning about YACC so that I can debug the OW build with debug information. YACC is fascinating and the exercise is taking me back to experience again the intellectual engagement and playfulness with which I fell in love with coding so many years ago. Once I have a handle on how OW is using YACC, I intend to get OW to build successfully with the debug flag on, and then I am going to replace many of the "-d1" with "-d2" in order to have OW optionally build with full debug information for all of its projects except those that can't handle "-d2" due to segment size limits.

Can anyone tell me why "-d1" is used almost everywhere rather than "-d2"?

[jmalak]

you should read notes about building. The build system is not so stupid how you expect. you should learn something about build system before you ask similar questions. -d1 is used everywhere in building system by default, read what -d1 option mean in compilers documentation.

[ideafarm]

Mr. Malak, I am totally in love with OW and I think that what you are doing, and have done, is awesome. I see you as a godsend. (I also see you as a DTM, which most, if not all, good coders are.) So there's no need for you to feel defensive about OW when conversing with me. Please do explain, if you know, why -d1 (line numbers only) is used almost everywhere rather than -d2 (full symbolic information).

When building for 16-bit with segments limited to 64K, some modules might require -d1 because -d2 would make a segment too large. But I'm not sure that even in that case -d2 can't be used, as long as the symbolic information is put into a separate file rather than embedded within the object module.

My real question is whether anyone can think of a reason to not make the following change to OW's full build with debug on: (1) Change most -d1 to -d2 and test full build. (2) For projects that won't build, try to fix by emitting symbolic information to a separate file. (3) If that doesn't fix it, change that project back to using -d1. (4) If it still doesn't build, change that project to use -d0. The end result is that a full build with debug on will give full debugging info for all targets and all projects except when that isn't possible.

WDYT? If I do all of the work to complete this polishing, will you (or anyone) object to it being pulled into OW's master?

[rdos314]

The way to build the debugger with debug info is to set OWDEBUGBUILD=1. You then should clean the projects you want debug information in and rebuild them. This could be clib and any or all of those that Jiri mentioned as being part of the debugger.

[ideafarm]

Thanks. I've already figured that out and also now know that WDW.EXE does not display locals info because it is built with -d1. That's what got me onto this track of polishing the build to make -d2, rather than -d1, the default behavior during a full build with debug info.

Does anyone have any objection to making this change? I'm about to do a fair amount of work to make this change, and I want to know before I do that whether anyone is going to object to pulling it into OW master.

BTW, the 16 bit segment size limit does appear to be the reason why -d1 must be used when building 16 bit modules. I suspect that that is why -d1 is used everywhere. Rather than change -d2 to -d1 only where necessary (the correct way), someone just changed it everywhere (the lazy way). In looking into this just now, I see that the compiler cannot be told to use a separate symbol file, so the 16 bit segment size limit probably is the reason for -d1 everywhere.

[rdos314]

I don't follow. There is no reason to build anything in OW for debug unless you are debugging OW itself.

[jmalak]

Do you understand what is OW build system is doing? It is building OW distribution at all. !!! It is release, which is delivered to user !!! It is main purpose. Unfortunatel it support to build debug version of OW tools which you need for development, bug fixes etc. But build all OW as debug version is absolutely useless (you can do it with command builder build OWDEBUGBUILD=1). You need to debug only some OW tool therefore debug build is used selectively. It can be used on various hierarchical levels, by example One project all platforms or for one platform (it is typical use). Some projects has extra target for debug version that it is build by command builder build debug By example code generator and compilers. If you use these command then it setup all necessary compilers, linker and librarian options to build debug version of OW tools.

[jmalak]

Unfortunatel it works the same way for non-Watcom tools (64-bit version of tools) but you need appropriate non-Watcom debugger.

[ideafarm]

Still no answer to my simple question. I ran across a text file that indicates that -d2 was originally used everywhere. Today, -d1 is used everywhere. At some point in time, someone changed it all from -d2 to -d1. The question is, why? -d2 should be the default behavior for any project that can handle it. So I propose that it all be changed back to -d2. Does anyone have any objection to that? Some specific reason that I don't know about that makes -d1, rather than -d2, the best default behavior?

Here's how this is going to play. I'm going to make the changes. Then I'm going to request that Mr. Malak pull them. If he is going to reject such a pull request, I want to know now, before I do all of the work.

[jmalak]

You are still not accepted reality that OW build system control debug options. you want change what is done automaticaly by existing build system Why? It looks like you don't understand build system or don't read documentation

[jmalak]

I don't understand your resistance to what is writen in documentation and what I say you. Don't expect answer about something if you are ignoring information from others Anyway don't submit any change request to something if you don't understand how it works. It is useless. You should try to understand how OW build works before you wan't change anything.

[jmalak]

Here is place in documentation how you can create debug version Debug build by OW build system

[ideafarm]

Still no answer to my simple question. I am going to change -d1 to -d2 nearly everywhere. After testing it all, I am going to ask you to pull the change into OW master. Assuming that it passes testing, will you do the pull?

What I am really trying to find out is whether I can work with you. I have big plans for OW, plans that are orthogonal to whatever plans you have. But I need your cooperation. Are you willing to work with me as a peer as we both cooperate as equals, developing OW independently according to what each of us envisions, yet cooperating fully with each other? Are you capable of this, Mr. Malak?

If you are not willing or able to do this, if you think that OW belongs to you and that you alone will control its development according solely to your vision, then I will have no choice but to fork OW, which will create many unnecessary probems, for the two of us, for others who want to contribute code changes, and for end users.

[rdos314]

I think you need to understand how OW works before you go on to change code. Forking the code will not help you to understand it better, and you will likely introduce a lot of new bugs into it. I don't want to rely on a compiler full of bugs, so I don't think your method for how you want to modify the code is acceptable.

[jmalak]

You invest your time to useless activities. Change of debug/release build is simple by OWDEBUGBUILD environment variable. You can use it on multiple levels with OW builder. If you want to switch between debug/release for wmake then wmake release=0 switch to build debug version. I don't want waste my time to explaining you complex build system if you ignore elementary information and don't read documentation. I will not accept any similar request for change if they don't resolve any "real" issue. Your knowledge about OW building system is near zero, therefore you want change -d1 by -d2. Untill you will understand what -d1 and -d2 mean and why they are used it is hard to explain you anything. Try difference between output of builder build OWVERBOSE=1 and builder build OWVERBOSE=1 OWDEBUGBUILD=1. May be you will accept that all is OK.

[ideafarm]

Mr. Malak, I understood the difference between -d1 and -d2, and what each does, in 1996. Where were you in 1996? How old were you at that time, when I was already four years into the development of a 1/2 million line C++ "piggyback" operating system?

You obviously have zero interest in the appearance of another coder who has a vision for the future of OW. That's OK. It just confirms that you are a coder, with no interest in the overall development of the code or in its marketing and general management. You are likely a very good coder, likely with a deep understanding of the code. That is a wonderful thing to have, and to be. But OW needs more. It needs what I, and potentially many others, have to offer.

The solution that I am envisioning is a friendly fork, mated with automation (using IPDOS (tm)) that can keep multiple forks in semantic synch while allowing each fork owner the autonomy to do his or her own thing without anyone else's permission regarding identifier renaming, build control options such as -d2 versus -d1, and eventually module reorganization and renaming. In this automation that I am envisioning, each fork owner will be able to fairly easily pull desired code fixes without having to pull the syntactic changes.

Happy coding, sir!

[jmalak]

It is advantege of open source code, you can do your own fork and release your own ideas if you don't want accept OW 2 rules.

Good Luck

[ideafarm]

I know that I am stuck in "old thinking" and am only beginning to understand how open source works, both technically and in human terms. Although I can now begin to focus a bit on OW, there continue to be many demands on my time. One question. Is the GitHub and Git software smart enough to allow you to pull some of my changes while rejecting others, so that you can pull bug fixes from me, and I can pull code polishes and enhancements from you?

I would like to contribute to your fork as much as possible with bug fixes and testing, while relying on your fork for enhancements and the kind of code polishing that you are currently doing. You might not want all of my changes, but I want all of your changes. Will that be easy or hard or impossible, due to Git limitations?

[jmalak]

All changes to OW 2 must follow rules. If not than will not be accepted. What you will add to your fork is your decision. Possibility is given by Git not by GitHub

[groessler]

I'm just a casual user and most of the time just lurking..., but

@ideafarm: You want to change -d1 to -d2 for whatever reason I might have missed in the thread. Why do debug builds of Watcom not work for you? Can you give a rationale?

[ideafarm]

"If it isn't tested, it doesn't work."

That general rule emerged in the 1980's. It addresses the natural optimism of coders, who generally speaking love to write code and hate to test code. The natural inclination of coders, generally speaking, is to indulge themselves as much as possible by convincing managers that testing should be done as a separate function, by (of course) someone else. The rule is intended to encourage coders to test their own changes before submitting them or even considering them "finished".

Mr. Malak appears to be the typical genius coder (a.k.a. prima donna DTM) who thinks that testing should be someone else's job.

Applying the rule, automated continuous integration builds should be done with -d0 everywhere and then again with -d2 everywhere. The fact that -d2 is never used for release and building with -d2 everywhere has little practical value is irrelevant. Building everything with -d2 constitutes a test.

The fact is, the current code base CAN'T be built with -d2 everywhere. Mr. Malak either doesn't know that or he doesn't care. I care. I suspect that I am much more QA/QC oriented than Mr. Malak is, and that's fine. I have confidence that Mr. Malak will generate lots of good code enhancements with a reasonable number of regression bugs introduced. But I have no confidence in Mr. Malak's QA/QC instincts; it just doesn't appear to be his focus. It is my focus, both of necessity and because I am as much a manager as I am a coder.

[groessler]

"If it isn't tested, it doesn't work."

A strong statement, but often correct.

So you want to do continuous integration tests?

Applying the rule, automated continuous integration builds should be done with -d0 everywhere and then again with -d2 everywhere.

Why exactly -d0 and -d2? What's with -d1 and -d3?

The fact is, the current code base CAN'T be built with -d2 everywhere.

Where exactly can't it be built with -d2?

And, please refrain from attacking Mr. Malak directly. He has the job of overviewing what goes into the repository, and in your case (understable to me from what I know from your posts) doesn't see a benefit.

You should start small: Create your continuous inegration environment for the current state (-d0 and -d1), and try to get it integrated. Then, but only then, discuss about -d2 (and -d3).

[ideafarm]

Nothing I write is ever intended to be an ad hominem attack. Repeating for emphasis, I view Mr. Malak as a godsend. He is also very much like me, I think, as far as being a "prima donna DTM". I am as prima donna DTM as one can get, and I take pride in that and so should he. The fact that he and I are different and have different visions is a good thing. That is what creates the potential that excites me.

-d2 is the important level. -d1 is line numbers only. -d3 is -d2 plus unreferenced type information, which is normally not needed when stepping through code. The current code will not build with debug turned on, even though practically all projects use only -d1. My first effort will be to debug this.

[rdos314]

I regularly test debug builds, and I can assure you it works. I use a debug build of clib as well in our production code, and I always compile my own code with all debug information. It was a long time since I tried aggressive optimization switches. This typically only results in buggy code that is only marginally faster.

The reason you cannot build all of OW with debug information is that in the old DOS executable format, the debug information is put in executables, and so they overflow. Could also be that some code gets too large segments when no optimization is done. Since I find little value in DOS applications at this point, I would have removed those (or used separate build switches for DOS) to allow for complete debug builds of the whole OW project. Still, this is not a big issue, and building parts of OW with debug information (like the debugger) works fine.

[ideafarm]

Getting the full build to work with OWDEBUGBUILD=1 is mostly just an exercise for me to start to become familiar with the code. But I might have already found a problem with the way that YACC and AS symbolic constants are interacting with each other in the AS project. This bug affects three projects, which I am guessing are related to using YACC to generate code during the parsing of Asembler, C, and C++ source files. AS and the other two projects will not build with OWDEBUGBUILD=1.

Whether anyone would ever need to build these three projects with debug information is irrelevant to my goal, which is to find bugs and become familiar with the code. "If it's not tested, it doesn't work." applies to these three projects. Builds for them don't work.

Regarding removing support for "dead" target platforms, I've thought about that over the years and would oppose it, although I would yield if a compelling reason emerged. Some day, the fact that OW supports targeting 16 bit DOS might attract academic or even commercial users to use OW.