Building shared libraries?

[rochus-keller]

This is a great library and there is definitely a need for it; thank you very much for open-sourcing it!

I explored the code and made some experiments with the build system. My approach was to work with the standard features of cmake and build a dynamic library instead of the static ones. However, you seem to have built your own build system with the cmake language, and before I study all the code I wanted to ask first.

Are there provisions to build a shared library on all platforms instead of the static libraries? From an architectural perspective, is there anything against dynamically binding your code to an application?

I also tried to build the libraries without the inet module to get rid of the libcurl dependency; but when deleting the corresponding line in the top level CMakeLists.txt and all exes assumingly depending on it I still get "ERROR- libCURL is required" and "missing: CURL_LIBRARY CURL_INCLUDE_DIR" from cmake.

I was able though to create the attached (quick and dirty) qmake project which generates a shared library without the inet module which I now try to use with my Oberon+ FFI (see https://github.com/rochus-keller/Oberon/); currently it works on Linux i386. But of course it would be more elegant to use your build system. NAppGUI.pro.gz ../todo/res_assert.h: res_assert.h.gz

[frang75]

Hi @rochus-keller and thank you very much for the feedback!

Indeed, libcurl-dev is currently a dependency for compiling NAppGUI on Linux. However, next weekend, I will upload a patch to correct this. When it will be ready you will have to comment/remove the following lines from the main CMakeLists.txt:

staticLib("inet" "core" NRC_NONE) 
desktopApp("Products" "demo/products" "inet" NRC_PACKED) 
desktopApp("UrlImg" "howto/urlimg" "inet" NRC_EMBEDDED)  

It will not be possible to compile the inet library (or projects that depend on it) if libcurl-dev is not installed. But the rest of the libraries and projects could be compiled. More info here: https://nappgui.com/en/start/win_mac_linux.html#h3

On the other hand, support for dynamic multiplatform libraries is scheduled for the middle of next February. We will have a new command for CMake:

dynamicLib("core" "osbs" NRC_NONE)  
dynamicLib("myLib" "core;myOtherLib" NRC_NONE)  

which will work similar to staticLib. Also an executable or dll will be able to combine static and dynamic dependencies.

Regards!

[rochus-keller]

Great, thanks. I try to be patient ;-)

[rochus-keller]

Meanwhile I managed to build a working NAppGUI shared library including all required symbols using this improved qmake project and two helper files: NAppGUI.pro.zip.zip

I also managed to create my first Oberon+ external library module for the NAppCore which works well so far with both my Mono and Cgen code generator of Oberon+: NAppCore.obx.zip

Looks like NAppGUI is well suited OS/GUI abstraction for Oberon+; more to come.

[rochus-keller]

In case you're interested I finished the Oberon+ external library modules and migrated your Hello example; works perfectly on my Linux machine, both as generated CIL as well as C application; see https://github.com/rochus-keller/Oberon/tree/master/testcases/NAppGUI; I will migrate more examples this week and also do experiments on Windows and Mac.

[frang75]

Hi again @rochus-keller !

I've been taking a look at your Oberon project and I find it very interesting. I will delve into it over the next few days. I did not know this programming language, but seeing that it was created by Niklaus Wirth is an extra motivation.

I'm also surpresed at how quickly you have integrated NAppGUI. It is a pleasure to meet such motivated people. Please let me known any issues you encounter within NAppGUI. I will try to solve them as soon as possible. I am sure your suggestions will help me improve the library.

What Linux platform are you using? I would like to test Oberon/NAppGUI using your own system.

I attach the implementation of res_assert.c/h to replace your "dummy". This module is automatically generated by nrc from /src/gui/res_assert folder. nrc is in /prj/script/linux. Surely I will distribute the nrc source code and documentation in future revisions. All resource folders are pre-processed by CMake/nrc and added to the build project automatically (as C source code). https://nappgui.com/en/start/resources.html#h7

This resources are required for the NAppGUI assert dialog: https://nappgui.com/en/start/bugs.html#h3

Keep in touch!

res_assert.zip

[rochus-keller]

Thank you for the files, I will gladly integrate them. Tonight I will try to migrate your Fractals example and then compile your library on Windows and macOS and run the examples on these platforms too.

The language specification and some examples and articles are available under http://oberon-lang.ch (though it's far from the excellent quality of your documentation). Oberon+ unites all features of Oberon-07 and most of Oberon-2 (here are some details: https://oberon-lang.github.io/2021/07/16/comparing-oberon+-with-oberon-2-and-07.html), and it is perfectly legal to use the old syntax with upper case keywords and semicolons, even mixing the old and new syntax.

My major development machine is an old EliteBook 2530p of which I have many copies with an i386 Linux 3 kernel and an Ubuntu derivative distribution; this is a great machine with a long battery life and a good keyboard with trackpoint which I can carry along and do real work e.g. on the train. But I also have macs, windows and some arm machines where I can compile and test; from time to time I release a new pre-compiled version of my IDE for Linux i368, Win32 and mac x64; deployment of pre-compiled Qt apps on Linux is an adventure though, so most people compile it themselves, or they just run the Win32 version under Wine.

[rochus-keller]

Update: the fractals example works on Linux and Windows; here is a pre-compiled Windows version of the IDE and the examples (including the required DLLs): http://software.rochus-keller.ch/OberonIDE_win32.zip

The generated C code works on Linux and Windows with no observed issues so far.

There are still issues with the generated Mono assemblies on Linux and Windows; multi-threading in the external library called via FFI doesn't seem to work; on Windows also the event callbacks don't work yet; the Fractals example works on Linux under Mono as long as not more than one thread is selected.

[frang75]

Hi @rochus-keller!

I can't download the OberonIDE_win32.zip file. On the other hand, if I understand it correctly, the Oberon compiler generates a C file, which is the one that is compiled in the end. Can you pass me the *.c of the examples you are working on? Especially those where multithreading fails.

Mono is the open source .NET? What is FFI?

Thanks!

[rochus-keller]

I can't download the OberonIDE_win32.zip file

Do you have an anti-virus software on your PC preventing the download? Can you try using wget?

Can you pass me the *.c of the examples you are working on?

Here are the C files generated from the Fractals project using the IDE: Fractals_Cgen.zip To compile you need the Boem GC .lib and C headers; to run you need the NAppGUI.dll and the Boehm gc dll. I can compile and run this code both on Linux and Windows with no problems.

Mono is the open source .NET?

Yes. The Mono Common Language Runtime (CLR) is a complete implementation of the Common Language Infrastructure (CLI) as specified in ISO 23271. .NET is a more general term which also includes a lot of classes written on top of the CLR. I'm not using .NET, but only the CLR and the minimal runtime. The CLR is a "managed runtime" with it's own JIT, garbage collector and thread management. I suppose that the call to bthread_wait interferes with the thread management of the CLR; on Linux Mono crashes during this call and on Windows the fractal is just no longer displayed until I reset the number of threads to 1; I don't think that it has anything to do with NAppGUI, since this doesn't happen with the C generated version of the app.

What is FFI?

Foreign function interface.

Note that meanwhile I found the reason for the crashing callbacks and implemented a work-around which I will commit to Github tonight (together with a new precompiled version of the IDE). The reason was the incompatible calling convention; as it turned out Mono on Windows by default generates callback stubs with stdcall (instead of cdecl) calling convention; I was able to extend my CilGen and the Pelib by the required CLR low-level elements to control the stubs generation.

Update: The commit is uploaded to Github; I also uploaded the pre-compiled Win32 version to https://github.com/rochus-keller/Oberon/releases/tag/FFI_MVP (see Assets section) in case you cannot download it from my website.

[rochus-keller]

Update concerning Mac:

I was able to adapt the NAppGUI.pro file to macOS and generate the dylib. The commit is up, and a pre-compiled version of the IDE with all shared libraries is ready here: http://software.rochus-keller.ch/OberonIDE_macOS_x64.dmg and https://github.com/rochus-keller/Oberon/releases/tag/FFI_MVP.

The good news: it works seemlessly with the generated C files.

The bad news: it doesn't seem to work with Mono (i.e. when run from within the IDE); Mono just prints the first log message from the app and then keeps waiting forever; I assume that the Objective-C implementation does some threading which interferes with the Mono task management; the same effect is observable both with Mono 3 and 5. But anyway we can develop an Linux or Windows and still run the resulting C files on Mac; I will have a look at the *.m files anyway and also do research on how libraries accessed over P/Invoke can run their own threads without blocking Mono.

[frang75]

I was able to adapt the NAppGUI.pro file to macOS and generate the dylib.

Great! I'll try to download later and check the .dmg

The bad news: it doesn't seem to work with Mono

I'll take into account for futher revisions. I have an experimental support for C++/CLI .NET dynamic libraries based on NAppGUI. I don't know if a .NET wrapper can help with that.

Thanks again for all your comments and feedback 😃

[rochus-keller]

Btw. I also experience an issue when I want to download files from my website when using the WebKit based browser on Mac; the same also happens when I want to access Stackoverflow; apparently WebKit is extremely overzealous with its security concerns and I didn't find an option yet how to switch off this behaviour; would not be the first time that Apple infantilizes its users.

I'll take into account for futher revisions.

Apparently the problem occurs mainly when I call bthread_wait in the main thread of Mono; just creating threads and letting them run doesn't seem to be a problem; I already did some experiments on Linux. Before you change anything, I should do some more testing and research.

[rochus-keller]

Update: bthread_wait doesn't seem to be the primary problem; now I have evidence that the function handed over to bthread_create_imp doesn't start or causes a deadlock immediately when started; bthread_create doesn't block and runs to bthread_wait, but since the thread function doesn't continue, bthread_wait waits forever; I avoided all data access in the thread function but it still blocks; I will need some time to understand the issue; the cause probably is within Mono or in my generated code, since the issue doesn't happen when running the same program under C.

I opened an issue in the Oberon repository and will track the progress there.

[rochus-keller]

Update: the Mono thread issue is solved; see https://github.com/rochus-keller/Oberon/issues/17#issuecomment-1010366906 for more information; so we now have reached the following achievements:

1. NAppGUI shared library successfully built and tested on Linux, Windows and MacOS.
2. Oberon to C transpiled GUI applications such as the Fractals example compile and work on all three platforms.
3. Oberon to CIL (Mono) compiled GUI applications work on Windows and Linux, but not yet on MacOS.

The issue on MacOS seems to be related to the Cocoa implementation; console applications using NAppGUI (without the GUI functions) work fine; I will further investigate where exactly the code hangs.

[rochus-keller]

The (putative) mac Mono issue is solved!

After an hour in the debugger with the Cocoa implementation of NAppGUI, where everything looked OK, I found out that everything works fine, but the window is just in the background, covered by all other running application windows - silly me; apparently it was just a coincidence that the window was in the foreground when running the C application from the console.

The NAppGUI applications seem not to have the standard Apple menu when run on my Mac and I don't see them in the window list (i.e. cannot ALT-TAB to them and have to click on the window to bring it to the foreground). With the following modification, the window appears in the foreground and is shown in the window list:

File "osapp_osx.m", function "osapp_init_imp", added the following two lines just before return:

[NSApp setActivationPolicy:NSApplicationActivationPolicyRegular];
[NSApp activateIgnoringOtherApps:YES];

This modification worked fine with the Hello and Fractals example; will do some other tests too.

[frang75]

Titanic effort in record time. Congratulations on your progress @rochus-keller!

Yes it's correct. A NAppGUI application does not show any menu by default (not even the Apple standard).

It must be added explicitly. Here there is an example of application with menu: https://nappgui.com/en/demo/products.html

On the other hand, I don't know if I fully understand the Alt+Tab problem. The application appears in the app list when you press [Cmd + Tab] (AppleKey + Tab).

Keep in touch. Greetings.

[rochus-keller]

Thanks. I had a quick look at the demo/die example you reference; from my point of view it has a similar GUI concept as Hello or Fractals, and even though on your screenshot I see the Apple menu on the top left. This didn't appear on my Mac (Catalina); so apparently it has to do with the macOS version too. With my modification it appears.

On the other hand, I don't know if I fully understand the Alt+Tab problem.

The ALT-TAB finding is just a symptom; the major issue was that if I start the application from within my IDE I don't see the application window; that's why I first thought that the application was blocking; there was just a single log output on the terminal but nothing more; only today I discovered that there was indeed an applicaton window, but it was behind all other windows and only appeared when I lowered all other windows (since it didn't appear in the window list and there was no other way to see or bring the window to front otherwise).

What macOS version are you using on the machine you created the screenshot?

[rochus-keller]

Btw. have you seen my DEF file? It might be useful if you want to create DLLs on Windows without changing all function declarations.

[frang75]

Yes! I want to avoid the use of an extra parameter (like DLL_EXPORT) for each function. I will check the support for .def files in CMake and cross-platform.

Thanks @rochus-keller!

[rochus-keller]

In case you're interested: today I officially released the Oberon+ NAppGUI binding, see https://github.com/rochus-keller/Oberon/releases/tag/NAPPGUI_MVP. Of course there is still work to do, but the binding is at least mature enough so the demo apps migrated so far work well. Maybe you're interested to know that instead of using ResPack/ResId (which depends on static C features not available to Oberon+) I can directly add hex-encoded byte array literals to the Oberon+ code. Here is an example; the hex strings were generated using the Linux hexdump -v -e '16/1 "%02X " "\n"' icon.png command.

[frang75]

Hi @rochus-keller NAppGUI 1.3 has been released, with support for dynamic libraries.

https://nappgui.com/en/guide/build.html#h2 https://nappgui.com/en/guide/libraries.html#h2

[rochus-keller]

Great, thanks; reading your release notes I conclude that my existing binding should still work with the new DLL (just witout access to new features like tableview. Or should I regenerate the binding?

[frang75]

The public interface has not change, your binding will not need changes. But you must update it to NAppGUI 1.3 and recompile it.

[rochus-keller]

Thanks. Meanwhile I updated my fork with your most recent main commits and adjusted my BUSY files. I noticed that you removed the nrc source code (utils) which I fetched from the second newest main commit. Everything builds fine, but when I start e.g. fractals, it just prints "Starting log for 'fractals'" to the console and then goes to sleep forever; no window shows up; I'm on Linux x86 in my BUSY2 branch (though I didn't commit to github yet). I will do some research, but unfortunately I don't have much time. In any case I have still the old working version on my OBX branch.

[rochus-keller]

Update: I made yet another attempt but this time didn't merge but directly add the BUSY files (and the utils subdir) to your most recent commit; when I compile and run with your Cmake based system the executable works; when I instead run my BUSY based build (which btw. runs about three times as fast) I see the same effect as described above.

I pushed to https://github.com/rochus-keller/nappgui/tree/BUSY3 what I have so far.

I also did test runs in the debugger with both the new BUSY3 and the old BUSY2 version and stepped down to osapp_run/g_application_run; the executed steps look similar in both versions, but only the old BUSY2 version shows a window when g_application_run is hit. I also tried to find out whether there are some defines I missed from your most recent build system, but didn't find anything so far.

[rochus-keller]

Update: NappGUI up to commit 7012ff0d42576ced1652d6dc6f7c7ee1d8606f0a (2022-11-26, the latest before 1.3 r4146) works fine when compiled with BUSY. The issue only starts with your commit b31d431ab289d704e1073910e99768a4c9a921bf (2022-12-26, 1.3 r4146).

I will now compare these two commits and identify possible reasons by inspecting the changes.

[frang75]

Hi again! I just added the source code of 'nrc' project to GitHub. I forgot do it in the new release. nrc no longer needs nlib, it is self-contained.

With the support for DLLs I had to update the build scripts. For the DLL version of the executables, the RPATH has been configured in (CMakeFunctions.cmake::targetRPath) . These macros are also defined for executables that use the dynamic version of libraries.

#define OSAPP_IMPORT 
#define OSGUI_IMPORT 
#define DRAW2D_IMPORT 
#define GEOM2D_IMPORT 
#define CORE_IMPORT 
#define OSBS_IMPORT 
#define SEWER_IMPORT 
#define GUI_IMPORT 
#define INET_IMPORT

The NAppGUI 1.3 have been tested in Ubuntu 12 x86, using the NAppGUI CMake build system. So far I have not found any issue in demo executables.

[rochus-keller]

Unfortunately I'm not lucky yet with this release; I refactored all my BUSY files so that either static or dynamic libraries are built and used by the examples; the dynamic build considers the IMPORT/EXPORT defines in the *.def files, but I still get missing symbols on Windows and Mac which I have to take care of; I probably have not yet correctly mapped the dependencies between the DLLs; but on Windows with the all-in-one DLL (which doesn't have inter-DLL dependency issues) the linker can't find the kBOX2D_NULLf symbol with my MSVC 2013 toolchain.

The statically linked executables work on Windows; on Mac Fractals and Bode work, but not Die; on Linux none of the executables work. Unfortunately I have to stop here because other projects require my time. Will check back in a few weeks.

Update: just tried a build an a Linux x64 machine (Ubuntu 14.04 LTS, Eol 2024) with exactly the same behaviour of the executables as on the x86 machine; also had a look at the source code changes between 1.3 and the commit before and didn't find the reason yet.

[frang75]

Hi @rochus-keller ! If in Windows dynamic build, functions link but global variables (such as kBOX2D_NULLf) do not, the reason is that GEOM2D_IMPORT was not defined before #include "box2d.h" (__declspec(dllimport) must be active in _geom2d_api extern const Box2Df kBOX2D_NULLf;). This macro must be defined when compile the executable, not the DLL.

[rochus-keller]

Thanks for the hint. I indeed failed to map all the detailed dependencies to the corresponding import definitions, which requires more knowledge about the implementation details than I obviously have. From my point of view the current IMPORT definitions concept is too complicated and error prone.

So I took another run at it, since we have a holiday here anyway, and took the liberty of rearranging the IMPORT concept a bit, as is done with Qt, for example. Only the .def files are affected. With this I was able to simplify both the build and the use of the libraries significantly, since you no longer have to recreate the delicate import dependencies with the defines all exactly. Instead it is sufficient to define NAPPGUI_SHARED.

My build now works on all platforms. The executables all work on Windows and Mac (but the problem on Linux persists).

I would recommend that you adjust the .def files as well; if necessary, you can even use both concepts in parallel without them getting in each other's way so you don't have to change your build.

[rochus-keller]

Heureka! I now also found the reason why the executables didn't work on Linux. Unfortunately I didn't realize that the __GTK3__ defines are used also in other places of the code than just in the GTK specific parts (especially in osqui.c); now since I extended the define to all sources osgui_initialize and osgui_is_initialized do their job and the windows are shown.

My https://github.com/rochus-keller/nappgui/tree/BUSY3 branch is now fully working on all platforms, for both static as well as multi or all-in-one dlls.

[frang75]

Congratulations @rochus-keller!

I'm glad NAppGUI 1.3 works with BUSY. To implement support for DLLs I had to revise the entire project. Static linking allowed me to call internal functions between libraries (draw2d/osgui) that dynamic linking doesn't.

Good thinking, #if defined(__GTK3__) should NOT be present in osgui.c. Differentiation between platforms should be resolved within the implementation of the functions, eg _osgui_is_pre_initialized();. I will review this in future commits.

[rochus-keller]

Thanks. Now since I know that __GTK3__ has to be defined it's no problem anymore. But it would be gread if you could consider the proposal concerning the .def files (see https://github.com/frang75/nappgui_src/issues/4#issuecomment-1373690263 and this commit: https://github.com/rochus-keller/nappgui/commit/82c819e71f579b284c7e1820b410c726de3290b7).

We can close this issue if you want.

[frang75]

Hi! I've been looking at your refactoring of core.def (and others) and it's not entirely clear to me.

In your BUSY branch, to activate __declspec(dllimport), the executable must define NAPPGUI_SHARED, so ALL dependencies of the executable are expected to be dynamic link.

A NAppGUI executable can mix static and dynamic dependencies, so:

• NAPPGUI_SHARED_LIB: Defined to compile a DLL (only in the DLL target).
• CORE_IMPORT (and others) It is defined in a executable target that will use the dynamic version of the core library.
• None define: For build or use the static version of a library.

For example:

Executable 1

// Will use the dynamic version of CORE, but static version of osbs and sewer
#define CORE_IMPORT
#include "coreall.h"
int main(...)
{
....
}

Executable 2

// Will use the dynamic version of CORE, OSBS and SEWER
#define SEWER_IMPORT
#define OSBS_IMPORT
#define CORE_IMPORT
#include "coreall.h"
int main(...)
{
....
}

Executable 3

// Will use the static version of CORE, OSBS and SEWER
#include "coreall.h"
int main(...)
{
....
}

The NAPPGUI_SHARED_LIB, CORE_IMPORT (and others) macros are automatically defined by CMake based on each project's dependencies (created with the staticLib or dynamicLib commands).

[rochus-keller]

NAPPGUI_SHARED has to be defined as soon as the DLL version of NAppGUI is either built or used. If NAPPGUI_SHARED is not defined, all the dllimport/export stuff is ignored.

If NAPPGUI_SHARED is defined, you can control with NAPPGUI_BUILD_X_LIB which library is currently built (the other ones which are not built are automatically imported without explicitly having to take care of all the detailed use/import relationships); e.g. the BUSY file in core defines NAPPGUI_BUILD_CORE_LIB, so that all core functions are compiled with dllexport and all other functions with dllimport.

Mixing static and dynamic libraries is not a good idea because of reduncancies (e.g. you would have to link e.g. sewer and osbs multiple times if they were static libraries, once for each DLL using them and once for the application) and memory management issues; so with my approach (which I actually copied from Qt) this is not supported.

So your Executable 1 is currently not supported by my build system; Executable 2 would just define NAPPGUI_SHARED, and Executable 3 would not define NAPPGUI_SHARED.

If you think mixing static and dynamic libraries is worthwile and feasible, or you don't want to change your build system, both approaches can coexist in the .def files. See this core.def how this can be done: core.def.zip

If you want I can make you an example how e.g. fractals could be built with BUSY if it was a separate project outside of the NAppGUI source tree.

[frang75]

I am going to rethink the .def issue and try to integrate both cases. I would like to leave open the possibility of linking static and dynamic libraries in the same executable. Thank you very much for the feedback @rochus-keller. I am leaving this issue open for now.

[frang75]

Hi @rochus-keller . What do you think of this approach? The idea is:

• Make it generic: BUSY, CMake-NAppGUI, other build systems.
• That it can be used with user libraries, not only with the 9 of NAppGUI. E.g casino library in examples.
• Being able to mix dynamic and static libraries (true that certain combinations may not be valid, but others are).

I have to make a few small changes. In BUSY you would only have to change NAPPGUI_SHARED to USING_SHARED_LIB and NAPPGUI_BUILD_CORE_LIB to BUILD_CORE_LIB (and others). I want to remove the NAPPGUI label for generalizing.

/* Core library import/export */

#if defined(NAPPGUI_IMPORT_CORE_LIB)
    #define IMPORT_DLL
#elif defined(NAPPGUI_SHARED)
    #if defined(NAPPGUI_BUILD_CORE_LIB)
        #define EXPORT_DLL
    #else
        #define IMPORT_DLL
    #endif
#endif

#if defined(__GNUC__)
    #if defined(EXPORT_DLL)
        #define _core_api __attribute__((visibility("default")))
    #else
        #define _core_api
    #endif
#elif defined(_MSC_VER)
    #if defined(IMPORT_DLL)
        #define _core_api __declspec(dllimport)
    #elif defined(EXPORT_DLL)
        #define _core_api __declspec(dllexport)
    #else
        #define _core_api
    #endif
#else
    #error Unknown compiler
#endif

[rochus-keller]

In BUSY you would only have to change NAPPGUI_SHARED to USING_SHARED_LIB and NAPPGUI_BUILD_CORE_LIB to BUILD_CORE_LIB (and others). I want to remove the NAPPGUI label for generalizing.

Let's try it. I'm glad if I don't have to track all dependencies and map them to defines myself, which also in your new approach seems to be ok. I would though be careful with defines without prefix since there is always a risk of unintended interference with other code bases; in case of e.g. IMPORT_CORE_LIB it is not generic anyway.

[frang75]

Please review edit in https://github.com/frang75/nappgui_src/issues/4#issuecomment-1382093317 If I'm not wrong now should work without changes in your actual BUSY. You don't need to track NAPPGUI_IMPORT_CORE_LIB and others. These will define by NAppGUI-CMake in the case of mix static/dynamic dependencies.

[rochus-keller]

I assume the #if defined(NAPPGUI_SHARED) should state #elif.

[frang75]

Yes, its true

[frang75]

Hi @rochus-keller

This commit https://github.com/frang75/nappgui_src/commit/34b3f2551c4b884e65270b0f35f50240a26e0935 fix the .def files to be compatible with BUSY.