packing/unpacking basic types

[tenko]

Consider the use case of receiving binary data in packets from communication and afterwards forming basic types from selected ranges of this data. Also the reverse process of forming binary data packets from basic types.

Currently the predeclared procedures BITS/ORD can be used for this process, but only for INTEGERS (or smaller) and not LONGINT and LONGREAL

REAL can use PACK/UNPK, but not LONGREAL. (Also any special numbers like INFINTY or NAN would not be preserved)

Also I do not find any way to access the upper 32 bits if the LONGINT type. If the set type was larger LONGINT could be covered.

This is just a comment after reading through the language specification.

I believe this was probably solved by procedures in the SYSTEM module in earlier Oberon version?

[rochus-keller]

That's indeed a pending feature; there is even a TODO in the ADOC file; as soon as all bit operation functions also support LONGINT you can use shift operations to access the upper 32 bits; I'm not yet fully happy with the integer type concept inherited from Oberon-2; I think explicit long literals and type casts (instead of long() short()) are necessary; but that's on the list; my current focus is on 32 bit systems.

[tenko]

Excellent. I did not immediate have this use case, but will in the future. Good to know this will eventually be sorted out.

[rochus-keller]

In what kind of applications do you plan to use Oberon+? What languages/frameworks did you use so far for this kind of applications?

[tenko]

I have used Python/PyQT for many years and the last couple of years I switched over to FreePascal and Lazarus due to the way the Python ecosystem of packages have started to mimic Javascript with hundreds of dependencies etc. It is a mess. Probably due to influx of users of these systems.

FreePascal I am fairly happy with, but it has a large baggage and some strange bugs due to backwards combability with TurboPascal/Delphi.

I therefore believe that a modernized Oberon could replace Python(FreePascal with much lower complexity and much improved readability.

The use cases are some internal business related applications (stock keeping, order tracking, database system, report generation etc). This I will almost certain be kept in FreePascal. This is a use case it really shines. Everything is built in and no external dependencies. Everything built to a single binary exe. I understand it is also where Delphi is still in heavy use today.

The second use case is test system hardware and technical products which communicate with PLC, sensors, moto drives etc. User input from touch screens etc. Here I prototype with Micropython (on microcontrollers) and Python with Raspberry PI hardware. In a deployed situation you need a compiled language with static memory (a garbage collector has the risk of fragment the memory over time). Very often you need to write your own drivers for hardware at this level (I2C, SPI, RS485 Modbus etc).

Here I think that Oberon+ can be used and I can avoid the use of the C language directly with all it's built in foot guns. C is the standard on most embedded hardware systems today.

These kind of systems have a very long life span (several decades) and therefore you will have to keep the software updated over time and readability/simplicity/stability is very important.

This is probably the hard lesson many industries now faces. Like the bank/finance/insurance industry with all the complex Cobol code nobody is able to work with or replace. There is no tests or documentation. You would have to replace it, bug for bug, as the bugs have become features. It basically stop all innovation and limit their ability to operate.

They are actually scraping information and simulating input towards terminal screens in order to operate their system. Probably other industries or governments branches have similar issues.

Java/C++/Python systems in use today will probably create similar problems in the future due to the complexity of the languages and frameworks in use.

This needs to be done in a better way for thing to improve.

[rochus-keller]

Interesting, thanks. If you want to use Oberon+ to replace something like Delphi or PyQt a lot of libraries/frameworks have to be developed first; I actually started to implement a code generator which is supposed to be able to generate a decent C API for Qt which then could be directly used from Oberon+; but I also consider completely different solutions like e.g. https://github.com/Immediate-Mode-UI/Nuklear which require less resources and would also be suited for embedded applications.

I actually implemented quite a lot of embedded systems, from microcontrollers like STM32 or ESP32 in C to linux based systems in C++. Oberon like Pascal has the advantage of value arrays and records; to avoid dynamic memory management altogether I thought of VLAs like in C. On Linux based embedded systems one could even run Mono and do remote debugging directly on the board; not much changes would be required for this. If the app is sufficiently stable a C based version of the application could be deployed. I didn't care much yet of multi-threading though; but Oberon+ of course could access threading C libraries; if the GC is not used, this should be not a big deal.

[tenko]

Thanks for the info.

There are not many good options with GUI's from C code. There is an interesting project covering only the QML part of Qt : https://github.com/filcuc/dotherside There where some issue pending last time I looked at it and I understand QML was changed again in Qt6. For embedded use I believe it can actually just be implemented directly with SDL or some small library on top of SDL, which there are many.

Just curious. How would one implement destructors if Qt is used with Oberon+?. Would you not be handling this manually? Could this be tied to the garbage collector at the C level of the bindings?

[rochus-keller]

There are some interesting developments ongoing concerining C GUI libraries; another interesting one is https://github.com/lvgl/lvgl. SDL is already supported by Oberon+, but it doesn't have widgets, though one could implement them in Oberon+ of course.

How would one implement destructors

Oberon+ has no destructors yet; I'm not sure whether it is necessary to integrate descructors; a possible way to do it is demonstrated by Component Pascal; but the lifetime of a QWidget is not the same as a Oberon+ GC object, so we should not wait to delete widgets until the Oberon object is collected; but these are only initial considerations.

[tenko]

I looked at LVGL. It now can also be used on Windows, but it seems to some major redesign efforts under way. Could be interesting option as it covers both the microcontroller without OS and Linux/Windows.

I know PyQt struggled a long time with the garbage collector interacting with Qt. Still there are foot guns to be aware of in order to avoid crashes today. Some object takes owner ship of others etc. It is quite complex to get right it seems.

[tenko]

Perhaps VAL procedure could be used to convert REAL/LONGREAL, like SYSTEM.VAL in Oberon2?

var
  x : LONGREAL
  y : LONGINT
begin
    x := 3.14152
    y := VAL(LONGINT, x)
end

and also similar for the reverse operation.

[rochus-keller]

In this case you could also use y := FLOOR(x). As I said my current focus is on 32 bit, but tuning all built-in functions to long versions of interger and real is on the list.

Btw. there is already a VAL built-in which I added for Blackbox compatibility, but I'm not yet sure about the semantics/limitations; it's also on the list.

[tenko]

For this purpose I am interested in a bit for bit conversion. Ref:

PROCEDURE isNaN (x: REAL): BOOLEAN;  (* by Robert D. Campbell *)
  CONST nanS = {19..30};  (* Detects non-signalling NaNs *)
BEGIN
  RETURN nanS - BITS(SHORT(ASH(SYSTEM.VAL(LONGINT, x), -32))) = {}
END isNaN;

This is from https://github.com/Oleg-N-Cher/OfrontPlus/blob/master/Mod/Lib/MathL.cp which I want to convert. This is Oberon2 I guess and the SYSTEM.VAL procedure is used several places.

[rochus-keller]

It probably makes more sense to add a built-in function isnan() or - like component pascal - a NAN and INF value. Depending on the backend implementation of real/longreal in Oberon is not a good idea from my point of view.

[rochus-keller]

Btw. I just came across this GUI library: https://nappgui.com. I will try to use it from Oberon+.

[tenko]

That looks very promising. It covers much more than a GUI and is more like a lightweight version of QT with abstraction of OS functionality included. Proper documentation is a big plus also.

[rochus-keller]

In case you're interested I created the external library modules for Oberon+ and migrated the Hello example of NAppGui: https://github.com/rochus-keller/Oberon/blob/master/testcases/NAppGUI; works well, looks like a good match.

[tenko]

That is impressive. I will take a look when and test it when Windows support is up and running. Tthe author of NAppGUI quotes N.Wirth in the documentation. I think he got his philosophy right when implementing this library.

[rochus-keller]

Meanwhile I was able to generate the NAppGUI shared libraries on all platforms and do the necessary updates in language, compiler and IDE to make it work. The Linux, Windows and Mac precompiled downloads are ready and the commits are on Github. There is currently one limitation: using the multi-threading functions of NAppGUI doesn't work with Mono; as long as everything is in the same thread and no locking/waiting in place, the app runs fine on Mono (at least on Linux and Windows); if I try to enable more than one thread in the Fractals example Mono crashes on Linux and stops drawing on Windows. There is no such restriction with the generated C code; everything just compiles and runs as it should. I will do research on how Mono can interoperate with threads started in external libraries, but meanwhile we have an MVP version at least for single-threaded NAppGUI applications.

[tenko]

Excellent.

I was able to built NAppGUI shared library with MinGW 64 and tested it successfully with my build of Oberon+. I only made some small changes which could easily be incorporated (Filed an issue for this). I also had to add a few missing libraries to the .pro file (-lGdiPlus -lWSock32 -lUXTheme -lComCtl32 -lws2_32). Probably MSVC include these automatic.

MinGW is a nice alternative to the Microsoft compiler and I can build the IDE with static linking against the QT libraries.

The only problem I encounter is when I try to start the .exe from the shell:

[18:34:20] Starting log for 'Main#'
System.TypeInitializationException: The type initializer for 'Fractals' threw an exception. ---> System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   --- End of inner exception stack trace ---
   at Fractals.ping#()
   at OBX.Runtime.pcall(Command cmd, Boolean report)

This works when starting the .exe from the IDE.

[rochus-keller]

I don't see the issue with my assemblies generated for 32 bit, neither on Linux nor on Windows; ./mono Main#.exe just starts and runs fine (unless I change the Threads combo); could be a 32/64 bit issue; the exception apparently happens when the Fractals.dll is loaded i.e. when it loads the NAppGUI assemblies and the P/Invoke shared libraries; are you sure that all assemblies were generated by the 64 bit version of the IDE and you use the 64 bit Mono version? To avoid the marshalling overhead of P/Invoke I generate all CIL structs with explicit layouts, i.e. these structs are 32/64 bit sensitive if there are pointer type members; I actually check for architecture compatibility in each such assembly and issue an exception if there is a violation, but maybe I missed a loop-hole and Mono got a chance to run incompatible code before I do the check; this of course would cause memory corruption.

[tenko]

My fault. I tried to run the .exe directly. With mono.exe ./Main#.exe it works fine.

[rochus-keller]

Btw. the NAppGUI Fractals example now works on Windows and Linux under Mono too; the problem was that apparently the threding subsystem doesn't work as expected when run from a CIL static constructor; I now moved all module initializers to dedicated methods instead. On Mac there is still an issue with the GUI under Mono, but for another reason.

[rochus-keller]

Update: the macOS issue is solved; it wasn't actually an issue; the GUI worked, but when the application was started from the IDE, the window was burried under all other application windows and didn't have an Apple menu; I now added a fix to NAppGUI (60e980f0553d9d) and commited a new IDE version with some other fixes; the Oberon+ NAppGUI binding is now MPV for CIL and C on all platforms.

[tenko]

That is very impressive. I will then just have to invent a need for an small application and try this out.

[rochus-keller]

I'm thinking of migrating the Bode and Products demo to Oberon+; migrating Hello and Fractals was straight forward; wonder how I can handle resources (https://nappgui.com/en/start/resources.html).

[tenko]

There is some information how FreePascal/Lazarus handle this : Link

It would be nice to be able to include external data as images, long SQL command text etc in a standard way. Perhaps a new statment : RESOURCE SQL := SQLCMDS and then some functions to access this as a byte array in the code. It also has to tie in with NAppGUI in some way. This is probably not easy to get right on multiple platforms, but it seems FreePascal/Lazarus has solved it.

[rochus-keller]

Thanks. Also NAppGUI has its resource concept, but I didn't have a look at it yet. It's alway possible to include binary resources in the source code with a dedicated generator; also Qt does it this way. In Oberon+ byte array literals can be specified in source code (platform independently). SQL commands are just text literals which is no problem.

[tenko]

Yes. That is a possibility. A simple utility could be developed for this kind of embedding of raw data.

I was thinking more in the line with putting static binary data directly into memory section as can be done in c/c++ with optional memory protection (linker .rodata/.text section). This is a relative new feature of CPU's and was not widely available when Oberon was first developed. I think this could be wort while for give an extra layer of protection of an application. Perhaps not a high priority, but something to think about in the future.

[rochus-keller]

Btw. here are examples where I put the png images and icons in the source code as hex strings: https://github.com/rochus-keller/Oberon/tree/master/testcases/NAppGUI/Resources, https://github.com/rochus-keller/Oberon/blob/master/testcases/NAppGUI/HelloGui/Icons.obx. I converted the png included with tha NAppGUI demos using this command:

hexdump -v -e '16/1 "%02X " "\n"' image.jpg

Wirth uses the $$ syntax in his Project Oberon System, but he didn't document it. I added it to the Oberon+ specification.

[tenko]

I like this simple solution. Would it make sense to declare this as const char *MODULE_IMAGE = "XXX";, so that it would be reused, read-only and placed in .rodata section? (it looks like C backend is not implemented yet)

I tested the demos it works fine on Windows 10 64Bit. The library is usable now, there probably are no more show stoppers?

[rochus-keller]

Would it make sense to declare this as const char *MODULE_IMAGE = "XXX";

These kind of literals are translated to an array initializer in C, i.e. something like MODULE_IMAGE[] = { 1, 2, 3.. }; I assume the compiler will automatically place these in read-only sections.

there probably are no more show stoppers?

No that I'm currently aware of; It would be unrealistic to assume that there are no more bugs; as usual I'll remove them when I see them.