WaitZar version 1.8 super-bug

[GoogleCodeExporter]

This bug will block on all bugs that we hope to fix before version 1.8.

Version 1.8, being even-numbered, will be introducing some ground-breaking
changes, including the ability to change the language. 

Original issue reported on code.google.com by seth.h...@gmail.com on 24 Nov 2009 at 2:57

• Blocked on: #13, #26, #63, #66, #67, #71, #72, #74, #77, #81, #83, #84, #90, #95, #97, #98, #99, #100, #103, #104, #110, #117, #122, #123, #125, #126, #127, #129, #130, #133, #134, #141, #143, #144, #145, #148, #151, #153, #154, #160, #164, #165, #168, #169, #172, #173, #176, #179, #180

[GoogleCodeExporter]

More blocked on bugs.

Original comment by seth.h...@gmail.com on 24 Nov 2009 at 3:12

[GoogleCodeExporter]

We're having a vote:
http://myanmaritpros.com/forum/topics/please-vote-for-waitzar-18
...and on other sites. 

Also, several of the changes (Shan, TTF, ZgKeyboard) will require changes to 
wz-core.
As I see it:
  1) "input", "internal", "output", "display" encoding must have reasonable
workflows, so that enabling a language (e.g., "Shan") takes work, but enabling a
keyboard/output/display encoding (e.g., "shan.keyman/zg2010/shanZG.ttf") 
requires no
code at all, and minimal (GUI-driven) configuration.
  2) Our "config" file needs a complete overhaul. (Also, I'm disabling
non-powertyping for good). I want to store it as JSON. Also, I like the idea of
having a config DIRECTORY, which contains one config.txt file and any number of
directories, each with config.txt and various other files. That way, a 
developer who
wants to develop, say, Karen support, can just make config/karen/config.txt (and
config/karen/myfont.ttf, etc.) and distribute these ---no messy config file 
editing.
We will need to find a way to avoid slowdown on loading WaitZar... maybe spin 
it off
into a thread and join when the data is needed? --but I'll file a bug report
specifically for this.

For c++ json parsers, I like JsonSpirit:
http://www.codeproject.com/KB/recipes/JSON_Spirit.aspx
...because it uses templates and boost and it's actively maintained (last 
release: 2
months ago). However, if it requires too much "boost" linking, it's out.

I used to like TinyJSON, but it's been removed for... unknown reasons? The 
WayBack
machine has it:
http://web.archive.org/web/20080314075556/http://blog.beef.de/2008/01/04/tinyjso
n/
...but reading it now, there were a few defects. Nothing serious, but if it's 
out of
active development, then I'm worried. 

The whole point of this comment? Several main features depend on a mature 
wz-core
infrastructure. I've got plenty to do even while the voting remains incomplete. 

Original comment by seth.h...@gmail.com on 24 Nov 2009 at 8:14

[GoogleCodeExporter]

Compiled json-spirit with VS2008. Some interesting points:
  1) Requires Boost headers, but not Boost libs (Good!)
  2) Requires Boost at all (Good in general, but Bad for svn checkout -> compile)
  3) Takes 2-3 minutes to compile (Bad!)
  4) Generates a 75mb static library file (Bad!)
  5) A test program generates a 0.85mb exe file (Very Good!)

Recall that WaitZar is 0.65mb, and a lot of the test exe's 0.85mb is probably 
already
in WaitZar as Windows stuff. So! Let's overcome these problems.
  1 & 2) A quick (but careful) look at Boost.Spirit's source, I estimate 4MB of
header files could be copied into our repository. I dislike manual tricks like 
this,
but I dislike unstable builds even more. (We currently use 50mb, so the amount 
is
negligeable).
  3, 4, & 5) Ok, so we need to put our entire configuration module into its own
project, and compile this into WaitZar as a .lib file. This fixes long compile 
times
(they still exist, but only when we recompile our config module). Next, we make 
sure
to limit template explosion by only using several functions of the json parser, 
and
several template types (like wstring ONLY; no string. Also, int, bool are ok). 
This
fixes library size, and also ensures that WaitZar doesn't grow too big from
incorporating Boost. 

So, my next step: (open a config file bug and) draft a test config syntax that 
is
non-trivial and compile it (per the recommendations above) directly into WZ. 
Test
startup times, too. Also have many fake "modules", like 100 (for various
languages/encodings/etc). Call this a "general feasibility test" for the new 
config.
Try to find a way to test on an old machine. 

Original comment by seth.h...@gmail.com on 24 Nov 2009 at 8:53

[GoogleCodeExporter]

Note: wrote a script to manually extract all potential Boost headers needed. 
Total
size: 1.5mb. Perfectly fine for svn, and the script allows easy migration to new
versions of spirit-json and boost-spirit.

Original comment by seth.h...@gmail.com on 25 Nov 2009 at 10:23

[GoogleCodeExporter]

Ok, fixed a bug in the Python tagger (it missed lines like "#    include") and 
got it
to read 3.5 mb. Then, because boost has a lot of directive-expanded 
dependencies, I
had to manually copy over about 30 files. Total file size: 3.91 mb. Still 
acceptable,
however, migration to new versions of Boost will be a pain.

Output library is 77.1mb. Now, I need to write some non-trivial configuration 
classes
and compile that into WaitZar to see how small we can get all this.

Original comment by seth.h...@gmail.com on 25 Nov 2009 at 11:33

[GoogleCodeExporter]

For reference:

 * The entire boost header directory is 44.8 mb (waaay too big for our SVN)
 * All directories Spirit mentions needing (including classic) total 9.85 mb (still
big, but ok if needed)

The reason I mention this is that we need a .kms parser. I would hand-write a
recursive descent one myself, but,... well... Boost.Spirit _is_ recursive 
descent,
and meant for tiny projects, and way better than anything I might come up with. 
So,
we may have to mirror the entire thing (sans the ~30mb not related to Spirit). 

I'm still leaving it out of SVN until I find a good place for it, though. Seems 
like
it'll be shared between the "Config" and the "Keymagic" library projects. I'm 
not
committing 10mb of 2kb files until I know for sure they're not going to be 
moved. 

Original comment by seth.h...@gmail.com on 27 Nov 2009 at 1:37

[GoogleCodeExporter]

Ok, here goes:
   1) I refactored existing code to use our library, and built off of it,
implementing stub classes to force it to compile in the library.
   2) The library still builds big (80mb) but WZ is only 0.64 mb after compiling
(0.01 mb bigger than before). 
   3) We don't invoke our templates yet, so the code will undoubtedly get bigger. But
it's nice to know that non-templated code from our two libraries compile to 
nearly
nothing.

Next step: loading config files (for real).

Original comment by seth.h...@gmail.com on 28 Nov 2009 at 11:32

[GoogleCodeExporter]

All code is implemented and cleanly separated. However, linking nonsense is 
getting
in the way.

I think I'll add all libraries to nice, clean directories in the main WaitZar
project. Then, in ConfigManager.h, just:

#include "json_spirit_value.h"
namespace json_spirit {
    extern bool read( const std::wstring& s, wValue& value );
}

This will remove the need to link json_spirit_reader.h (although VS should 
really be
smart enough to figure that out). Still, can't hurt. 

I'll also take the opportunity to clean up some of WaitZar's inherently messy 
include
structure. I want to do the #include "folder/file" syntax a lot more.

Original comment by seth.h...@gmail.com on 30 Nov 2009 at 6:08

[GoogleCodeExporter]

Finally got the re-organization done.

So, our code compiles pretty fast, but it takes a while to link. This is 
because CL
tries to optimize the entire program at link time. This is fine for the release
build; I still haven't configured the debug build yet (but I will have to 
soon... the
program crashes at startup!)

Anyway, we've now got much more sensible directories/filters:
   Contrib:  anything not WZ apache 2.0
   Engine:   anything to do with our new typing system
   Settings: anything to do with settings parsing, etc.

Hooray, super-update.

Original comment by seth.h...@gmail.com on 1 Dec 2009 at 2:33

[GoogleCodeExporter]

Added U+200C request to the issue list.

Original comment by seth.h...@gmail.com on 3 Dec 2009 at 11:21

[GoogleCodeExporter]

Got comments to work within JSON files, like so:

   # This is a comment

Also, hierarchical loading of config files works, and the dot-notation for 
settings
is also working. All the infrastructure is in place to finally load the language
configurations dynamically. (Well, I have to embed default.config.json.txt, but
that's easy...)

So! Call it a milestone.

Original comment by seth.h...@gmail.com on 7 Jan 2010 at 5:30

[GoogleCodeExporter]

I changed WaitZar to use a "MyWin32Window" class, instead of straight Win32 
calls. 

This is useful; it allows us to pass around Window objects, painting on them 
(etc.)
in any class that has the object. 

Additionally, it lets me centralize code, so that I can clean it up later. For
example, resizing and repositioning Windows is done at least four different 
ways;
this is bad for maintenance and future improvements.

Furthermore, it reduces bloat. Previously, adding a new double-buffered window 
would
require managing at least six different variables, and handling circular
dependencies. Now, it only requires 2 variables, and dependency management is 
easy.

Unfortunately, there's a buffer overflow bug. This feature is still "alpha" 
(SVN is
slightly broken for now).

Original comment by seth.h...@gmail.com on 12 Jan 2010 at 9:45

[GoogleCodeExporter]

All crashing bugs are fixed, but my LayeredWindows are acting up again 
--neither is
visible. Need to debug...

Original comment by seth.h...@gmail.com on 12 Jan 2010 at 12:41

[GoogleCodeExporter]

Fixed; it was a simple typo. Fixed a few more bugs finding it though....

Time for a small amount of code cleanup, chasing that thread bug again, and then
finally getting down to the real refactoring.

Original comment by seth.h...@gmail.com on 13 Jan 2010 at 8:28

[GoogleCodeExporter]

Icon issue added.

Original comment by seth.h...@gmail.com on 14 Jan 2010 at 7:02

[GoogleCodeExporter]

One more bug.

Original comment by seth.h...@gmail.com on 24 Jan 2010 at 7:58

[GoogleCodeExporter]

Broke some highlighting while fixing recalculate(); will fix later.

Original comment by seth.h...@gmail.com on 25 Jan 2010 at 7:57

[GoogleCodeExporter]

After weeks of broken commits, I finally managed to recompile the program, and 
it
worked just fine with a few quick bug fixes. Thus, I decided to fix the pat-sint
highlighting glitch. The sentence construction code is somewhat flimsy, as is 
the new
word-picking code (since -1 is now a valid entry); I'll have to review it 
later. 

At the moment, though, I'm concerned with the slow start-up time of WaitZar. 
Basic
sentence typing works, so I want to add in FULL support for reading from 
configs.
Then, I can profile and see what's taking so long. It could be config parsing, 
object
initialization, file locking, or even just random Windows nonsense (e.g., 
searching a
directory). In fact, this could just be "Debug Mode" slowdown, as experienced 
with
vectors in Vi After getting all the file parsing logic in place, I'll profile 
the code. 

Summary: Huge progress! Input is now modular, and Transformations are there in
theory. Also, typing using a Letter-Based input "should" work, and will be 
debugged
extensively as I add config file support.

Original comment by seth.h...@gmail.com on 30 Jan 2010 at 8:08

[GoogleCodeExporter]

It was the Debug treatment of vectors. I just re-wrote the config (recursive) 
reader
to use the same array and avoid copying to a local array; program startup is 
almost
instant now.

Original comment by seth.h...@gmail.com on 1 Feb 2010 at 5:44

[GoogleCodeExporter]

Have to tweak config performance a little before releasing 1.8.

Original comment by seth.h...@gmail.com on 4 Feb 2010 at 3:07

[GoogleCodeExporter]

Finally got our "gigantic" menu to work in Win32; however, a few tweaks are 
needed to
allow it to display Burmese, etc. Added these as a new bug.

Displaying config file settings in this big menu works just fine.

Original comment by seth.h...@gmail.com on 8 Feb 2010 at 7:58

[GoogleCodeExporter]

Added some key blocking issues.

Typing Burglish basically works; got a few more issues to fix before we close 
that bug.

Next up:
  1) Romabama (which is sort of half-keyboard based)
  2) Debug myWin2.2
  3) Add Parabaik keyboard
  4) Generalize keyboards (and their virtual display) and re-enable the Help menu.
  5) Win Innwa, Zawgyi-One keyboards. 

And at some point (after 3, roughly)
  A) Add Padauk(Zawgyi-Padauk?), update our menus.
  B) Add Shan keyboard.
  C) Add Shan Unicode output. (Possibly add Shan ASCII font display for speed).

Original comment by seth.h...@gmail.com on 7 Mar 2010 at 1:12

[GoogleCodeExporter]

Displays are done (but not virtual displays). 
Going to myWin and keyboards before Romabama.

Original comment by seth.h...@gmail.com on 21 Mar 2010 at 9:22

[GoogleCodeExporter]

Added "new version" feature; it makes sense to add it now.

Original comment by seth.h...@gmail.com on 17 Jun 2010 at 9:56

[GoogleCodeExporter]

I'm removing bug 70; Unicode is used internally, but models can still maintain 
their own scratch encoding.

1.8's big contribution in this regard was converting Burglish to Unicode. I'll 
save WaitZar's wordlist for 1.9, since we'll be touching up the WZ wordlist 
anyway for 1.9.

Original comment by seth.h...@gmail.com on 18 Aug 2010 at 7:03

[GoogleCodeExporter]

De-listing issue 88; scons and ibus-waitzar are important, but I don't think I 
can spare time to work on them before 1.8 is released.

Original comment by seth.h...@gmail.com on 18 Aug 2010 at 7:11

[GoogleCodeExporter]

Added a few remaining dependencies. 

Original comment by seth.h...@gmail.com on 18 Aug 2010 at 7:16

[GoogleCodeExporter]

Adding bug 110: split the user's guide into dev. & user.
Removing bug 102: testing pagination with pat-sint shortcuts. I don't think 
it's possible to have these both together at the moment; I'm leaving the bug in 
for future reference, but it's definitely not a problem for 1.8.

Original comment by seth.h...@gmail.com on 18 Aug 2010 at 7:53

[GoogleCodeExporter]

Added split-off bug for highlighting keys.

Original comment by seth.h...@gmail.com on 20 Aug 2010 at 9:45

[GoogleCodeExporter]

Removed a few bugs and put them into the new 1.9 super-bug. 

Things like switching the typing mode for the help keyboard will just take too 
much time right now, with only a minor benefit to WZ.

FYI: Issue 125 is so close to being done...

Original comment by seth.h...@gmail.com on 22 Aug 2010 at 4:06

[GoogleCodeExporter]

New bug reports, with the virtual keyboard.

Original comment by seth.h...@gmail.com on 25 Aug 2010 at 10:40

[GoogleCodeExporter]

The following issues change WZ's functionality; after they have been completed, 
we can work on the user's guide & hacker's guide:
   Issue 90
   Issue 84
   Issue 81
   Issue 67 (Romabama) needs a demo, or something.
   Issue 25
   Issue 13 (hook up user-defined hotkey selection)

Original comment by seth.h...@gmail.com on 15 Sep 2010 at 12:39

[GoogleCodeExporter]

New bug. :(
Regarding the remaining three issues:
  1) Shan now has a tester.
  2) The "sticky" button will be added if one of the easier options works.
  3) Romabama will require a small demo; should be fun.

Original comment by seth.h...@gmail.com on 27 Sep 2010 at 3:33

[GoogleCodeExporter]

New WS bug; will have to re-do some of the config options to fix it.

Original comment by seth.h...@gmail.com on 28 Sep 2010 at 7:31

[GoogleCodeExporter]

Removed the sticky box. I really want to add it, but there's no justification 
for such a new feature right near the feature-freeze. 

Romabama, however, can have definitely make some progress. :)

Original comment by seth.h...@gmail.com on 28 Sep 2010 at 8:52

[GoogleCodeExporter]

Added a bug reported by Lionslayer in ZG.

Original comment by seth.h...@gmail.com on 28 Sep 2010 at 10:02

[GoogleCodeExporter]

All features added (Issue 25 was re-assigned to version 1.9).

There are still plenty of bug reports, but we are now in a feature freeze. 
Also, the config format has been solidifed (for 1.8 at least), so we can now 
update documentation.

Now would be a good time to put out a call for developers.

Original comment by seth.h...@gmail.com on 30 Sep 2010 at 8:31

[GoogleCodeExporter]

Another bug.

Original comment by seth.h...@gmail.com on 27 Oct 2010 at 6:02

[GoogleCodeExporter]

One bug fixed, one bug discovered.

Original comment by seth.h...@gmail.com on 3 Nov 2010 at 10:27

[GoogleCodeExporter]

Needed for a release candidate:
  Issue 103  (Burglish unicode check)
  Issue 126  (all keyboards) 
  Issue 145  (Zawgyi glitch)
  Issue 154  (WZ crashes when running tests)
  Any related issues (link to PDF users' guide, not DOC, etc.)
  Update "Testing Guide" & do it
  Scan bug list for anything 1.8-related.

After that, we can have an RC; while people test this, we can fix up the 
documentation. 

Original comment by seth.h...@gmail.com on 4 Nov 2010 at 6:09

[GoogleCodeExporter]

Minor Uni2Zawgyi issue.

Original comment by seth.h...@gmail.com on 15 Nov 2010 at 6:21

[GoogleCodeExporter]

2 new bugs.

Original comment by seth.h...@gmail.com on 23 Nov 2010 at 10:12

[GoogleCodeExporter]

Rounded up all remaining bugs for 1.8. 
Added a few bugs from routine checks.

Still need to run through the "Testing Guide" and "Compatibility Checklist".

Original comment by seth.h...@gmail.com on 27 Nov 2010 at 4:33

[GoogleCodeExporter]

Bugs left:
Issue 85  - "How to blog". Can be added after the release.
Issue 144, Issue 151 - New words. (Simple check).
Issue 162 - Weird highlighting bug. Have to get a response from one of our 
testers. 

And also:
Need to update & run through the testing guide/compatibility checklist.

Pretty close, actually.

Original comment by seth.h...@gmail.com on 29 Nov 2010 at 7:48

[GoogleCodeExporter]

Burglish crash is still present. Made a new Logging flag to help with this. 
Will run through our compatibility tests AFTER this gets resolved.

Original comment by seth.h...@gmail.com on 30 Nov 2010 at 9:41

[GoogleCodeExporter]

Another romanisation entry to add.

Original comment by seth.h...@gmail.com on 8 Dec 2010 at 3:31

[GoogleCodeExporter]

Another bug. This one's with pat-sint on WZ.

Original comment by seth.h...@gmail.com on 12 Dec 2010 at 10:09

[GoogleCodeExporter]

Normalization issue in most keyboards. 
I'm convinced that we can add some kind of "built-in" normalization to 
KeyMagic. For now, we'll fix it with rules. 

Original comment by seth.h...@gmail.com on 13 Dec 2010 at 9:44

[GoogleCodeExporter]

Added one last myWin bug....

Original comment by seth.h...@gmail.com on 14 Dec 2010 at 6:41

[GoogleCodeExporter]

Remaining issues didn't stop 1.8; removing and marking fixed.

Original comment by seth.h...@gmail.com on 9 Jan 2011 at 1:31

• Changed state: Fixed