search

devhawala / dodo

Xerox Network Services (XNS) implemented in Java
Other
16 stars 1 forks source link

Bug While Exporting on FsUtil #4

Closed iblowmymind closed 3 years ago

iblowmymind commented 3 years ago

Hey, Just wanted to quickly report a bug I found while exporting my VP Applications drawer to inspect it:

(...)
    exporting directory: Index Generator
      exporting directory: INDEX: Exclude These Terms
Exception in thread "main" java.util.DuplicateFormatFlagsException: Flags = '#'
        at java.base/java.util.Formatter$Flags.parse(Formatter.java:4605)
        at java.base/java.util.Formatter$FormatSpecifier.flags(Formatter.java:2800)
        at java.base/java.util.Formatter$FormatSpecifier.<init>(Formatter.java:2854)
        at java.base/java.util.Formatter.parse(Formatter.java:2713)
        at java.base/java.util.Formatter.format(Formatter.java:2655)
        at java.base/java.io.PrintStream.format(PrintStream.java:1209)
        at java.base/java.io.PrintStream.printf(PrintStream.java:1105)
        at dev.hawala.xns.FsUtil.log(FsUtil.java:78)
        at dev.hawala.xns.FsUtil.exportItem(FsUtil.java:214)
        at dev.hawala.xns.FsUtil.exportItem(FsUtil.java:221)
        at dev.hawala.xns.FsUtil.exportItem(FsUtil.java:221)
        at dev.hawala.xns.FsUtil.main(FsUtil.java:402)

The bug happens when you try to export the VP application named "Index Generator" (on the Long Document Options floppy for 8010 VP2.0.6, I believe).

-blw

devhawala commented 3 years ago

Hello blw,

the root cause is fixed, colon-characters in file items are now mapped to underlines when exporting items with FsUtil; the apparent error (leading to the stacktrace) was a missing/incomplete printf format specification when logging the failure to create the directory (due to the colon in the directory name).

Exporting the "Index generator" now gives:

Volume 'fs1' opened

export items from: public!1/VP-LongDocs!1
  exporting directory: VP-LongDocs
    exporting directory: Footnotes
      exporting file: FootnoteFolder.adf
      exporting file: FootnoteFolderConfig.bcd
    exporting directory: Index Generator
      exporting directory: INDEX: Exclude These Terms
        exporting file: Common Proper Nouns.USEnglish
        exporting file: Most-Common Words.USEnglish
        exporting file: Other Common Words.USEnglish
      exporting directory: INDEX: Include These Terms
      exporting file: IndexConfig.bcd
      exporting file: IndexGenerator.adf
    exporting directory: Shared Books
      exporting file: SharedBooks.2.0e.US.messages
      exporting file: SharedBooks.adf
      exporting file: SharedBooksConfig.bcd
    exporting directory: Table of Contents Generator
      exporting file: TOCConfig.bcd
      exporting file: TOCGenerator.adf

volume closed

Greetings Hans

iblowmymind commented 3 years ago

Thanks! I'll test exporting the entire VP Applications now, and tell if I notice any issues.

iblowmymind commented 3 years ago 
(...)
    exporting directory: VP IFM - Financial Solution
      exporting file: ** How to Use VP Integrated Financial Management 2.0**
Exception in thread "main" java.io.IOException: The filename, directory name, or volume label syntax is incorrect
        at java.base/java.io.WinNTFileSystem.createFileExclusively(Native Method)
        at java.base/java.io.File.createNewFile(File.java:1026)
        at dev.hawala.xns.FsUtil.exportItem(FsUtil.java:234)
        at dev.hawala.xns.FsUtil.exportItem(FsUtil.java:222)
        at dev.hawala.xns.FsUtil.exportItem(FsUtil.java:222)
        at dev.hawala.xns.FsUtil.main(FsUtil.java:403)

a similar issue occurs on asterisk signs as well, but this time because of Windows / NTFS limitations. For future guarantee, I think \ / : * ? " < > | characters (the list of characters that aren't allowed in NTFS) and NULL bytes should be replaced with underscores.

iblowmymind commented 3 years ago

Also, a question I had, would it be possible to convert StarFont / NovaFont / NovaWidth / fbBlivetOrigins / fbCoords files to TTF / OTF / X bitmap fonts / whatever else font format there is to inspect/use it (without using XDE Typefounder since that's not archived yet), inspect Icons / Bits files or maybe even disassembling BCD and/or BOOT files? (Since the contents of a BCD contain the source code filenames, which gave me the idea)

When it comes to other various files that can be found; reading .Messages files should be easy enough since the message contents themselves are stored in plaintext, however, there are lots of characters in between the messages which make it hard to read.

And one final question, (I know these are a lot of questions, I'm just curious about how the software works, I apologise for that!) I know Mesa.db is Pilot, MoonBoot.db is Diagnostics, SAx000Initial.db is a rigid initial boot file, .boot files are like initial core programs, but what is the function of a .germ file exactly?

Thanks! -blw

devhawala commented 3 years ago

Hello,

i added more character mappings, so NTFS should be also supported by FsUtil...

For your questions:

OK, have fun experimenting with the new Dodo version. Greetings, Hans

iblowmymind commented 3 years ago

Hello,

Thank you for your latest commit! The extraction happens flawlessly now.

Icon inspection can be done with an Icon Viewer tool in the VP Devtools floppy, I will look more into that. I can (sort of) understand Mesa code now after all of this, (without the official courses I'll probably not be able to understand / write code any further), however I don't understand how the bcd / bootfiles are arranged yet. If I do, there's a really small chance that I can figure out how to disassemble Mesa code, I'll probably fail, but hey, worth a try. So if you know about the structure of compiled Mesa code, please let me know!

Wow, I must've skipped over that Typefounder floppy. I'll try running it now and see if it functions. Also, it seems like this version of Typefounder is actually for XDE 5, since the floppy label says it is. And the header might be corrupted since copying it over from Draco's XDE+Hacks disk gave a Debug.Log and froze the system. Managed to use Duchess to copy the files over to a filing server, and use the File Tool to retrieve them. But, it works great! You can inspect individual characters in many different formats (including Strike, NovaFont and StarFont), make your own fonts etc. There might be a way to export each character into bitmap files, and somehow merge them together on a more modern platform to "port" the font over? I don't know, might look into it later.

For the second point, hm, I do remember seeing a Master file utility BCD file on the VP DevTools floppy.

And for the third point, yeah, I guessed that was the case too.

-blw

iblowmymind commented 3 years ago

I found a Mesa compiler.

Well, technically. It's for the BWS. And only VP 1.0 - 1.1. But, nevertheless, it's still a Mesa compiler. It can compile Mesa code in both simple and VP document formats. Unsure if it can Bind.

It's available in xde5.0utilsProt#11.dmk located in xde_5.0.zip

Will continue to look for more files. -blw

iblowmymind commented 3 years ago

That's sad. It only works with VP1. There were VP1 floppies in Bitsavers, so we could use the same method you used to install VP2 over the provided emulator images to make the Draco pre-built image. Can you tell me how you did it?

-blw

iblowmymind commented 3 years ago

Also found Pacman and Mahjong though. Also XDE File Access. And quite a few BWS Hacks.

devhawala commented 3 years ago

Hello,

i already tried to create a VP 1.1 disk for Draco, but it failed: some important floppy is missing (i don't remember which), leaving the fresh but partially installed disk unbootable (crashed in the boot process, i don't remember at which point).

As Draco does not support floppy boot, the procedure was roughly as follows:

I remember that the document editor is missing in the VP 1.1 floppy set at Bitsavers, as said at least one more important disk is missing. But may be you have more luck installing VP 1.1

Hope that helps Hans

devhawala commented 3 years ago

regarding mesa courses: there is a lot documentation about mesa programming at bitsavers, among that PDF files with the mesa courses, language manuals, ...

For example see:

iblowmymind commented 3 years ago

Document Editor is missing, I noticed that too, however, I don't think that would be necessary as the only purpose of this machine is going to be to run the BWS Mesa Compiler. You can use the Simple Editor to write code that compiles as well.

iblowmymind commented 3 years ago

Other than that, though, all of the floppies required for a core BWS install seems to be intact, other than the Document Editor.

iblowmymind commented 3 years ago

Oh, and also, you don't need to manually search for the floppies. There is a text file containing labels on the same directory, it is as follows:

130k01861   6085 Xerox ViewPoint 1.0.2 (Installer #1) #1 of 12 1Q86
130k01871   6085 Xerox ViewPoint 1.0.2 (BasicWorkstation #1) #3 of 12 1Q86
130k01881   6085 Xerox ViewPoint 1.0.2 (BasicWorkstation #2) #4 of 12 1Q86
130k01891   6085 Xerox ViewPoint 1.0.2 (BasicWorkstation #3) #5 of 12 1Q86
130k01901   6085 Xerox ViewPoint 1.0.2 (BasicWorkstation #4) #6 of 12 1Q86
130k01910   6085 Xerox ViewPoint 1.0.2 (Common Software #1) #9 of 12 1Q86
130k01921   6085 Xerox ViewPoint 1.0.2 (Common Software #2) #10 of 12 1Q86
130k01931   6085 Xerox ViewPoint 1.0.2 (Essential Applications) #12 of 12 1Q86
130k01941   6085 Xerox ViewPoint 1.0.2 VP Object Conversion Utility 1Q86
130k01960   6085 Keyboard, Display and Mouse Diagnostics 1.0 4Q85
130k01981   6085 Xerox Viewpoint 1.0.2 VP Training Exercises #1 1Q86
130k01991   6085 Xerox Viewpoint 1.0.2 VP Training Exercises #2 1Q86
130k02000   6085 Xerox Viewpoint 1.0 File Check #1 4Q85
130k02010   6085 Xerox Viewpoint 1.0 File Check #2 4Q85
130k02020   6085 Xerox Viewpoint 1.0 File Check #3 4Q85
130k12221   6085 Xerox ViewPoint 1.0.2 (Installer #2) #2 of 12 1Q86
130k12231   6085 Xerox ViewPoint 1.0.2 (BasicWorkstation #5) #7 of 12 1Q86
130k12241   6085 Xerox ViewPoint 1.0.2 (BasicWorkstation #6) #8 of 12 1Q86
130k12250   6085 Extended Boot Diagnostics 1.0 4Q85
130k12261   6085 Xerox ViewPoint 1.0.2 VP Training Exercises #3 1Q86
130k12270   6085 Rigid Disk Diagnostics 1.0 4Q85
130k12281   6085 Xerox ViewPoint 1.0.2 (Common Software #3) #11 of 12 1Q86
130k12291   6085 Xerox ViewPoint 1.0.2 Remote System Administration 1Q86
130k16610   6085 Rigid Disk Utilities 1.1 1Q86
130k16620   6085 System Configuration Utility 1.0 1Q86
130k16670   6085 Xerox ViewPoint 1.0.2 VP Training Exercises #4 1Q86
iblowmymind commented 3 years ago

Yep, you were right, the common software for NetCom / RemoteCom / Standalone are missing. Without them the BWS refuses to start.

image

(MP code 0712)

iblowmymind commented 3 years ago

Hello!

My experimentations with throwing whatever was made by Xerox that's remotely related to Star/VP/GV into anything that can run it continues, with this time, running GlobalView for X 1.05!

This version needs to patch and tweak kernel sources and recompile it, so it only supports a very specific architecture and operating system: SunOS 4.1.4 running on SPARC (either sun4u or sun4c, from what I understand.) Luckily, QEMU can emulate the SPARCstation 5 (although a bit broken, I will come to that later), which is a sun4u machine, and it can run SunOS 4.1.4! So I decided to give it a go. Since my experience in UNIX is much better than Windows, I thought it would be easier than installing GVWIN. Note: It wasn't.

The first step was to actually install SunOS onto a virtual hard disk. As I said, the emulation, especially with the CD-ROM access on SunOS, is broken, and the system cannot read CD-ROMs when running this exact configuration. However, there is a workaround: Introducing the CD-ROM image as a secondary hard disk image, which surprisingly works! (However, it doesn't work while mounting regular ISO-9660 drives, more on that) So, that way, by following this guide, which luckily details the whole process, I installed it with a cgthree graphics card to get 1152x900 resolution (which required OFW ROMs and an original cg3 ROM, with OpenBIOS / QEMU's stock ROM it wouldn't work) and 64 MBs of RAM (mainly because the largest disk size that's "pre-defined" by the format program, SUN2.1G (2.1 GBs), only gives 100 MB's of swap space, and I didn't want to screw over partitioning since it works in a sector-size basis (and I didn't want to miscalculate), and the system requires the swap space to be at least as big as the RAM size. (256 MBs, the maximum, would crash the emulator anyway, bug?)

After installing, I tried to mount the GVX 1.05 ISO (which I downloaded from here) using the secondary-hard-disk method, however, failed. It just didn't want to work! Then I noticed: if I set up a TUN/TAP device, connect the network to it, I could (potentially) transfer files to the guest using RCP! So I reinstalled SunOS, this time with networking enabled, created a TAP device and connected to it, assigned all the values, etc. (The networking procedure is also detailed in the wiki page I followed, so that was useful) and bam! I could ping the host! Then, I tried extracting the files from the CD-ROM (using 7-Zip) and repacking it as a tar archive so that I could send it via RCP and extract it on the guest, and everything would theoretically work!

But, of course, it didn't.

Apparently, the ISO contains many files with colons in their filenames, files that have the same names with different capitalizations, etc. Since NTFS (yes, my machine is Windows, and all the time I've been using Windows to run QEMU, the TAP device, etc.) doesn't like colons in filenames, and it's not case-sensitive, so some files got overwritten and the colons got replaced to underscores, which is not what I want, since the way that GVX works is drastically different than how GVWIN works. Apparently, all the files (including all the software, BWS, etc.) are exposed on the root instead of being contained in a disk image, which is the way GVWIN does it. And, instead of running as one fullscreen app, somehow the graphics calls are translated to OPEN LOOK calls, so everything can display on different windows running on the main system desktop! So, critical components, such as GV Write, GV Calc, etc. all had colons in their filenames, and if the installer can't find those and can't install them, what's the point of even installing it in the first place? I had to create a Linux VM, mount the ISO as a virtual CD (which worked!), put everything from there in a tar archive, and transfer it to the host. And thus, now I had a tar file on my hands! And since networking worked, I used rcp to send the files to SunOS... And now I had the tar file on SunOS! And, on top of all that, it successfully extracted!

At first, I couldn't figure out the layout of the disk, but apparently what you need to run is XIST (which is the XSoft Installation Tool). I extracted the archive to directory "XSOFT" under /home (which was the biggest mount, so I chose that), you have to have OPEN LOOK / OpenWindows running (which worked flawlessly under the cg3 adapter on QEMU, and now I had an 1152x900 desktop! Running it natively on built-in VGA is not documented on the wiki though, since it's a bit old, so the only thing you have to add is "-vga cg3 -display sdl -g 1152x900" onto QEMU parameters and it would work. I could, technically, run it on VcXSrv on my host, but that crashes GVX, more on that later) since XIST is a GUI app, and you need to run it from the XIST directory. So, I changed the working dir to /home/XSOFT/XIST, run xist (since . is on PATH by default), and I was good to go! You have to choose "Install / Upgrade" from the left, and "UNIX Directory" on the right, enter the path to the directory "installdir" on the directory, in my case, it was /home/XSOFT/installdir, and then after you click the Install button, a selection comes up with all the software. It even includes Document & File Services and an XNS server, which is very interesting! For now, though I chose GlobalView for X. Choosing it will freeze for a bit, and come up with a list of parameters for installation. I just left everything on default paths (and default kernel parameters), other than disabling network (since I haven't found any way to run Dodo on this, would be interesting if you provided some guidance on how to set it up.) and clicked Next. Now it will come up with a software group selection. I just selected everything (except Document Services Access, since I didn't need it, and Extended Language Options / Keyboard Layouts, etc.), including all the screen fonts and applications, and hit install, and it actually installed. It needs a reboot for the new kernel to load in, and after it booted (while it boots it will load an XNS module, GVX emulation modules and it will wait for an XNS Time Server, even if you disable XNS Networking on configuration, which will (obviously) fail due to the lack of a time server but the system will continue booting) and after logging back in, starting OpenWindows again and launching GVX (using /usr/bin/xerox/gvx) should work! And of course, something had to go wrong. Appearantly, for whatever reason, by default, the hostname you pick isn't added to the xhost access list. So after manually adding it by running "/usr/openwin/bin/xhost +(the hostname you picked)", and starting it again, it worked! Finally, I was at the login options sheet! And it was its whole separate window, with the OPEN LOOK toolkit look & feel since all calls were somehow emulated on Mesa and translated into OPEN LOOK, I still can't figure out how they did that! But, you can't create a user. Unless you start the emulator with "/usr/bin/xerox/gvx V", which will add V to the boot flags, which brings up the Workspace Administration Sheet. And from there you can create the user, and restart without the flag, and log in, and it all worked! I was in the workspace!

So, the Type 4/5 keyboards have these "Sun Keys" which are very much like the Move, Copy, Prop's, Open, Delete, etc. keys on a Xerox workstation. In fact, they have the same purposes, so GVX just uses them for the actions. But, since a proper keyboard emulation isn't done in QEMU, and you can't send raw scancodes, so those keys are rendered unusable (for now), thus, much of the functionality of this amazing system is yet to be fully discovered. And since there's so little documentation that has surfaced (other than the "Online Help and Documentation" package), I don't know what to do! Many of the apps have autorun disabled by default, and right-clicking does in fact come with a menu that has props, move, copy, etc. it's only for icons, and icons in some specific places (such as the Workspace) and not text fields, the loader, etc. So, as I said, much of the functionality limited.

Besides, I couldn't get the Document Services to work, which asks for an "organization password" which you have to obtain from Xerox Customer Services in order to create the first Clearinghouse. Bummer! And, I couldn't figure out how to connect the NetHubGateway to the emulated machine. Would just selecting the same TAP connection which QEMU's connected to work?

And, I still don't know how exactly it works, but from what I saw and said while searching through old mailing list / Google Groups archives for Xerox, it translates BWS calls into OPEN LOOK calls, the files and filesystem is exposed to UNIX, so no volumes, virtual disks, etc. However, the NS Filing volume / Desktops has an interesting structure, (kinda similar to Dodo's filesystem structure) If you want to install an app, you drop it into /bws/newAppls, then the system will move that file into the NSFiling volume. And there isn't much other than that. The documentation included is very limited.

An interesting thing though: The CD-ROM contains a copy of Lotus 1-2-3 v1.2 for SunOS, it also has Interpress Printer Fonts (all of them, not just Classic & Modern) and a 30-day evaluation key file, however, the system runs just fine without it, so I don't know what the point of that is.

I will continue my experiments and report if I find anything interesting. -blw

iblowmymind commented 3 years ago

Here are some screenshots:

image

image The NSFiling volume I was talking about, Desktops has a similar structure.

image The Common BWS files, except for the boot file, germ file and microcode. couldn't find those. Maybe they're embedded into the binaries?

And a note, apparently a GVX system with a built-in clearinghouse running is called a "Genesis" system. And also, apparently, VP 3.x existed! I mean, there are downgraders for it!

image There's a courier server, too!

image xnsstart doesn't work. Still couldn't figure it out!

image And here it is, running.

image Suspending seems to be not implemented.

The only buttons that seem to be working are the Win key (Keyboard), Num Lock (Cursor Keys), and Delete (you guessed it, Delete) right now.

A question: Would it be possible to dissect the binaries, somehow find the germfile, microcode, etc. from them, translate the calls into Swing GUI elements and add GV 1.x as a new target for Dwarf?

iblowmymind commented 3 years ago

Note: there's a utility called xwsft that gets installed that can take a peek at the Desktop volumes, however it's not very usable right now since it wants a clearinghouse on the network to verify the user password. If I can get Dodo to work with this, it could technically work!

iblowmymind commented 3 years ago

I'll move this to its own issue real quick.