HELP required - Not able to use libPGF

[RickCodes1]

Hi! This is Rick. Contact is at the end. Sorry for using this channel for abruptly summarizing something as important as the introduction between 2 people, specially if I am interested in making a collaboration with this real world open source app like digikam which I am a user of and a big thanked fan too, but, despite my years of exprience mostly in web development, apart form just ideas, a heavily significant code contribution from me may arrive a bit later from now since the combination of deep C++ and Qt library is not where I specialize although I have some solid C background, and have used PyQt for some apps. Currently I am trying to finish a task I have assigned to myself months ago and for which I have been preparing reading about some CMake theory also months ago, as this task consists in something similar to this repo: taking advantage of the digikam's PGF thumbnails that already exist for every image registered. And in the end, being able to use them, not in an embedded file of any format but displayed on an application in base64 format once PGF-decoding the loaded blob. Initially I thought the task would require to build a simple, maybe hello-world, C++ Qt app built with CMake and from there build the minimum required namespaces or classes to be able to use not only the libpgf folder's files but also the already coded pgfutils.h and .cpp That namespace and CMake files reconstruction seemed a daunting task so I have been postponing this task for some months. Meanwhile, I tried to gather some people to join and collab and learn together some technologies like deep CMake and others but that friendly recruitment ended up failing. Anyway, now after some months, after noticing the relevance of NodeJS c++ addons, and its benefits, after trying some things, life made me return to Visual Studio and the Windows API scripts I planned to make (cause yes, I am currently on windows 10 64 bit... BTW I have used some linux distros in the past for some time and not in a VM, but that just some time and now I would rely on WSL2 if I got to choose) and that's what made me return (and return more equipped, but again not so so so deeply in c++ yet) to this task of taking advantage of the digikam thumbnails... After noticing the NodeJS C++ addons to enhance a certain NodeJS desktop app, my current objective has changed to: Making a NodeJS C++ addon that uses node_api.h that given a string argument with the PGF encoded blob, could directly use the libpgf folder's files o decode it and then convert it to base64 format before returing it back to the caller app that would display that image. I have checked the downloadable files of libPGF official website and even though its open source, there is not source code abailable there.. just an .exe of PGF console and some other tools... So, only with the file of libpgf folder included in digikam, and even when including the .lib file that could be generated from those files (just in 32 bit arch because the .cpp and .h files use types like INT, BYTE, etc which are not compatible with 64 bit programs), I get the following errors when building my addon with node-yp configure build command, even when put in a x6 Native Tools VS console:

  AddonForPGF.cc
  win_delay_load_hook.cc
     Creating library 
\UtilsFromPGF\build\Release\AddonForPGF.lib and object 
\UtilsFromPGF\build\Release\AddonForPGF.exp

AddonForPGF.obj : error LNK2001: unresolved external symbol "public: void __cdecl CPGFImage::GetBitmap(int,unsigned char *,unsigned char,int * const,bool (__cdecl*)(double,bool,void *),void *)con
st " (?GetBitmap@CPGFImage@@QEBAXHPEAEEQEAHP6A_NN_NPEAX@Z3@Z)
AddonForPGF.obj : error LNK2001: unresolved external symbol "public: void __cdecl CPGFImage::Read(int,bool (__cdecl*)(double,bool,void *),void *)" (?Read@CPGFImage@@QEAAXHP6A_NN_NPEAX@Z1@Z) 
AddonForPGF.obj : error LNK2001: unresolved external symbol "public: void __cdecl CPGFImage::Open(class CPGFStream *)" (?Open@CPGFImage@@QEAAXPEAVCPGFStream@@@Z) 
AddonForPGF.obj : error LNK2001: unresolved external symbol "public: virtual __cdecl CPGFImage::~CPGFImage(void)" (??1CPGFImage@@UEAA@XZ)
AddonForPGF.obj : error LNK2001: unresolved external symbol "public: __cdecl CPGFImage::CPGFImage(void)" (??0CPGFImage@@QEAA@XZ)
AddonForPGF.obj : error LNK2001: unresolved external symbol "class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > __cdecl base64_encode(unsigned char const *,un 
signed __int64,bool)" (?base64_encode@@YA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@PEBE_K_N@Z)
UtilsFromPGF\build\Release\AddonForPGF.node : fatal error LNK1120: 6 unresolved externals 

I think I have orrectly configured my binding.gyp file and coded my AddonForPGF.cc file, so I don't know what else to try, I am really STUCK on this...

I hope you, as the creator of this repo, and a significant coder of this great app called digikam (which I thank for and wish all of the enhancements), could help me on this... I am up to specify any missed detail... An din any case here is my email: rick.codes.1101@gmail.com Thank you!

[cgilles]

Hi,

Your problem is a linking one, probably due to mix C and C++ and/or not exporting the C++ symbols properly. Windows is very sensitive to this stuff...

Without to see all the code, i cannot help.

In other words : create a github repo, push the code and share. Only in this case, i will able to help you...

Best

Gilles Caulier

[RickCodes1]

Sorry for the delay... I have put the code in this repo: https://github.com/RickCodes1/DecodingValidatorForPGF Hope it is possible to assist me soon... Thanks in advance!!

[cgilles]

How did you generate the myLibPGF.lib file ? With MSVC compiler i hope ?

If you take mine, it's cross compiled under Linux with MXE. C++ binary objects are not compatible between MSVC and G++. Do not expect to see Microsoft to be compatible with open Source world. NEVER.

Only C function are retro compatible between compiler due the age of C language. With C++ M$ re)invent the wheel as usual.

So you needs to recompile the libpgf.lib using MSVC, or use the cross compiled version but compile the nodejs code with G++ under Windows.

Definitively more simple solution : use Linux as well, all is already present for the DEVELOPERS... Windows is waste of time and as usual a big puzzle.

Voilà, just my more than 35 years experience as computer science engineer... (:-)))...

Best

Gilles Caulier

[cgilles]

MXE cross compiler for Linux : https://github.com/mxe/mxe

I cross compile the WHOLE digiKam source code plus all dependencies under Linux to create the Windows installer from scratch. There is no Windows system in my CI/CD. It take 4 hours under Ubuntu 18.04.

Gilles Caulier

[RickCodes1]

Hi! I am back here... after making various attempts (or maybe ctually a few big attempts to solve all) and some other thing..

Q: How did you generate the [myLibPGF.lib](https://github.com/RickCodes1/DecodingValidatorForPGF/blob/master/UtilsFromPGF/include/myLibPGF/myLibPGF.lib) file ? **With MSVC compiler i hope ?** A: Yes, I used VS2019, chose an empty project template where on its properties I changed the output to be a .lib instead of a .exe so in that way I only built the files inside the digikam \libpgf folder and not any additional file. Gue to theINT, BYTE, etc types, I got errors when building for x64 platform so I could only get a myLibPGF.lib ile for x86. So, yes. I used MSVC.

Thanks for your suggestions in your lines! Took into account that criteria: libPGF = opensource = made on Linux (x86) = not completely deeply compatible with Windows = for a .lib file a cross compilation is needed as another open source app (digiKam) does it. I say all of this to express that I got the strategy to folloe...

By using Ubuntu 22.04 in WSL2 (also with GUI) without any problems even when moving the unzipped mxe-master folder to /home/user/Desktop , I started trying this cross-compiling idea. Took me a while and various little steps to make the final attempt on that (that ends up failing due to a rare problem) and, in the end, after running these commands to install MXE and its dependencies:

g++ -c Decoder.cpp Encoder.cpp PGFimage.cpp PGFstream.cpp Subband.cpp WaveletTransform.cpp
ar rcs myCCLibPGF.a Decoder.o Encoder.o PGFimage.o PGFstream.o Subband.o WaveletTransform.o

cd ~
wget https://launchpad.net/intltool/trunk/0.51.0/+download/intltool-0.51.0.tar.gz
tar xvf intltool-0.51.0.tar.gz
cd intltool-0.51.0
./configure && make && sudo make install
cd ..
sudo apt install python3-pip
pip install mako-render

sudo apt-get install autopoint bison flex gperf libtool lzip ruby
sudo apt-get install libtool
find /usr -name libtool 2>/dev/null
ls /usr/bin/libtool 2>/dev/null
ls /usr/local/bin/libtool 2>/dev/null
sudo apt-get install libtool-bin
which libtool
cd ./mxe-master
make MXE_TARGETS='i686-w64-mingw32.static'   <------------------------------------------
make MXE_TARGETS="i686-w64-mingw32.static x86_64-w64-mingw32.static" -j 4

export PATH=$PATH:/home/dk-user/Desktop/mxe-master/usr
export PATH=/home/dk-user/Desktop/mxe-master/usr:$PATH

mxe-version

make g++ -x ./libpgf/myCCLibPGF.a
make MXE_TARGETS="i686-w64-mingw32.static x86_64-w64-mingw32.static" -j 4
i686-w64-mingw32.static-ar -x ./libpgf/myCCLibPGF.a
i686-w64-mingw32.static-lib /out:<output_file>.lib <object_file1>.o <object_file2>.o 

P.S: I am not completely sure if the 2 last commands are the proper ones to use if MXE were functioning right now...

In the make MXE_TARGETS=... command I point with the arrow I got these following errors despite solving all the various missing dependency errors I got first:

/mxe-master$ sudo make MXE_TARGETS='i686-w64-mingw32.static'

bash: - : invalid option
Usage:  bash [GNU long option] [option] ...
        bash [GNU long option] [option] script-file ...
GNU long options:
        --debug
        --debugger
        --dump-po-strings
        --dump-strings
        --help
        --init-file
        --login
        --noediting
        --noprofile
        --norc
        --posix
        --pretty-print
        --rcfile
        --restricted
        --verbose
        --version
Shell options:
        -ilrsD or -c command or -O shopt_option         (invocation only)
        -abefhkmnptuvxBCHP or -o option
fatal: not a git repository (or any of the parent directories): .git
[build]       mxe-conf                i686-w64-mingw32.static

Failed to build package mxe-conf for target i686-w64-mingw32.static!
------------------------------------------------------------
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
python --version
bash: line 1: python: command not found
make[1]: *** [Makefile:889: build-only-mxe-conf_i686-w64-mingw32.static] Error 127
make[1]: Leaving directory '/home/dk-user/Desktop/mxe-master/mxe-master'
real    0m1.893s
user    0m1.836s
sys     0m0.074s
------------------------------------------------------------
[log]      /home/dk-user/Desktop/mxe-master/mxe-master/log/mxe-conf_i686-w64-mingw32.static

make: *** [Makefile:888: /home/dk-user/Desktop/mxe-master/mxe-master/usr/i686-w64-mingw32.static/installed/mxe-conf] Error 1

I have python 3.10 installed... I remember that some of the depndencies required it so, changing the version of python may break some things and.. this installation or from its separate dependencies has already been really tricky enough... So I may still significantly delay with this installation of MXE...

So,...

[RickCodes1]

So, while I get rid of this error at installing MXE, is it possible that you could complete my current objective of building the x86 myLibPGF.lib by cross-compiling it with g++ and MXE in Linux? Supposedly that .lib file using only the files inside digikam's /libpgf folder should be enough without any additional file, right? Or an additional implementingPGF.cpp similar to pgfutils.cpp may be required? Any way, I should try without any additional files first I guess...

So, this is what I took away from you r lines: building the .lib file in Linux cross compile it and then use it on Windows and NodeJS windows...

Right now is when I read again and understood that you also suggest to just use g++ on Windows10.. I guess by uising MinGW GCC/G++ Compiler I gues... Which Ok, I will try, But emphazising he open source criteria, i am also interested in trying a .lib file made on Linux that you could help me with since I don't have MXE right now...

Sorry if this was too much text.. Thank you in advance Gilles!

[cgilles]

A: Yes, I used VS2019, chose an empty project template where on its properties I changed the output to be a .lib instead of a .exe so in that way I only built the files inside the digikam \libpgf folder and not any additional file. Gue to theINT, BYTE, etc types, I got errors when building for x64 platform so I could only get a myLibPGF.lib ile for x86. So, yes. I used MSVC.

==> Good point !

Thanks for your suggestions in your lines! Took into account that criteria: libPGF = opensource = made on Linux (x86) = not completely deeply compatible with Windows = for a .lib file a cross compilation is needed as another open source app (digiKam) does it. I say all of this to express that I got the strategy to folloe...

==> I never said that. libpgf is low level C++ code using system base API. It's fully compatible with Windows POSIX API.

!!! In others words do not increase complexity of your workflow. !!!

1/ If you compile libpgf with cmake (if i understand well) under Windows, well you certainly forget to pass some extra options to compile the library

2/ What's the code in NodeJS to play with libpgf API ? It's C++ i hope, because you cannot call C++ from C without to write a C to C++ wrapper for libpgf API (Singleton design pattern). This one do not exists in libpgf as i know. NOTE: you can call C API from C++ as well, it's compatible of course, not the inverse...

** If you are curious, see my bash scripts to install MXE deps for a small library named Exiv2:

https://invent.kde.org/graphics/digikam/-/blob/master/project/scripts/build-mxe-exiv2.sh

When it installed (locally), there is a second script to compile Exiv2 with MXE:

https://invent.kde.org/graphics/digikam/-/blob/master/project/scripts/bootstrap-exiv2.mxe.sh

** The digiKam bundle compilation with MXE is more... complex, but similar than previous scripts for Exiv2. Look in this directory:

https://invent.kde.org/graphics/digikam/-/tree/master/project/bundles/mxe

Gilles Caulier

[RickCodes1]

1) Currently I am not using CMake, I meant that I was noticing that digikam used CMake on its builds with Qt and I thought a similar app would be needed just to decode the PGF images but after discovering NodeJS c++ addons, I am tryin gto use libpgf directly... 2)The code in the NodeJS app and addon are javascript and C++, not C.

And.. At this point, after getting and trying g++ 8.1.0 through MinGW by choosing i686-win32-sjlj from this link: https://sourceforge.net/projects/mingw-w64/files/ And by using these command on Windows 10: >ar rcs myLibPGF.lib Decoder.o Encoder.o PGFimage.o PGFstream.o Subband.o WaveletTransform.o I sitll get the same errors.. I am sure they are the same

  AddonForPGF.cc
  win_delay_load_hook.cc
     Creating library 
\UtilsFromPGF\build\Release\AddonForPGF.lib and object 
\UtilsFromPGF\build\Release\AddonForPGF.exp

AddonForPGF.obj : error LNK2001: unresolved external symbol "public: void __cdecl CPGFImage::GetBitmap(int,unsigned char *,unsigned char,int * const,bool (__cdecl*)(double,bool,void *),void *)con
st " (?GetBitmap@CPGFImage@@QEBAXHPEAEEQEAHP6A_NN_NPEAX@Z3@Z)
AddonForPGF.obj : error LNK2001: unresolved external symbol "public: void __cdecl CPGFImage::Read(int,bool (__cdecl*)(double,bool,void *),void *)" (?Read@CPGFImage@@QEAAXHP6A_NN_NPEAX@Z1@Z) 
AddonForPGF.obj : error LNK2001: unresolved external symbol "public: void __cdecl CPGFImage::Open(class CPGFStream *)" (?Open@CPGFImage@@QEAAXPEAVCPGFStream@@@Z) 
AddonForPGF.obj : error LNK2001: unresolved external symbol "public: virtual __cdecl CPGFImage::~CPGFImage(void)" (??1CPGFImage@@UEAA@XZ)
AddonForPGF.obj : error LNK2001: unresolved external symbol "public: __cdecl CPGFImage::CPGFImage(void)" (??0CPGFImage@@QEAA@XZ)
AddonForPGF.obj : error LNK2001: unresolved external symbol "class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > __cdecl base64_encode(unsigned char const *,un 
signed __int64,bool)" (?base64_encode@@YA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@PEBE_K_N@Z)
UtilsFromPGF\build\Release\AddonForPGF.node : fatal error LNK1120: 6 unresolved externals 

even when running node-gyp configure build using this 32 bit version of the compiler and specifying 32-bit output in binding.gyp:

{
    "targets": [
        {
            "target_name": "AddonForPGF",
            "sources": [ "AddonForPGF.cc" ],
            "include_dirs": [ "C:\\Users\\D\\AppData\\Roaming\\npm\\node_modules\\node-addon-api"
            ,"C:\\Users\\D\\Documents\\D\\include\\libpgf"

            ],
            "libraries": [ "C:\\Program Files (x86)\\Windows Kits\\10\\Lib\\10.0.19041.0\\um\\x64\\WS2_32.Lib"
            ,"C:\\Users\\D\\Documents\\D\\include\\libpgf\\myLibPGF.lib"],
            "cflags": [ "-fno-exceptions" ],
            "arch": "ia32"
        }
    ]
}

So, at this point, I don't consider anymore building another .lib file even with MXE because, although i got rid of compiling errors before these linkin erros, I guess that I am not properly using \libpgf folder's files in my AddonForPGF.cc code(the .cc extension is a format where you can place C++ code without the need of headers, for the C++ addon one can choose to use a .cc file or a .h and .cpp files)...

So, in this situation, could you please review the C++ code included in my AddonForPGF.cc file, and maybe edit it, even in a .h or .cpp if you want, to ensure I am not using incorrectly the library? My goal is to use directly the libpgf functionality to decode the PGF images not to a file but to a buffer to then convert it to base64 format for displaying the image directly, without using PGFConsole.exe app that exports it decode to a file and is not so fast, and even without the use of Qt if possible... In the [DecodingValidatorForPGF repo] (https://github.com/RickCodes1/DecodingValidatorForPGF) only AddonForPGF.cc and binding.gyp need to be edited for the AddonForPGF to work properly. The whole NodeJS app of the repo is built oriented towards that goal... I have included a sample C++ addon PrimeExample which could be comented or configured through binding.gyp... The only facts not mentioned in the repo are that I tested the Node App in 64-bit Windows 10 and used NodeJS v18.12.1 but also the same error is pressent with a Node v14... How possible is it to not use Qt for this?

Hoping you could help me on this, diving deeper into libpgf and my code without using Qt if possible, Best regards, Gilles

[RickCodes1]

As mentioned here in PGFimage.h , there are only 3 methods to decode PGF format so I really wonder what have I done wrong..

//////////////////////////////////////////////////////////////////////
/// PGF image class is the main class. You always need a PGF object
/// for encoding or decoding image data.
/// Decoding:
///     Open()
///     Read()
///     GetBitmap()
/// Encoding:
///     SetHeader()
///     ImportBitmap()
///     Write()
/// @author C. Stamm, R. Spuler
/// @brief PGF main class
class CPGFImage {
public:

[cgilles]

Your problem is a linking stage due to missing export C++ symbols rules, need by MSVC compiler. G++ is lees tolerant with that under Linux.

MSVC is a pain. I lost plenty of working hours about this stuff to compile whole digiKam under MSVC.

https://stackoverflow.com/questions/444356/forcing-symbol-export-with-msvc

These rules are missing in official libpgf code (/h file in declaration of classes).

I patched the headers in digiKam for that (digiKam include libpgf code source, it's not an external deps).

here for ex:

https://invent.kde.org/graphics/digikam/-/blob/master/core/libs/pgfutils/libpgf/PGFimage.h#L55

Look the DIGIKAM_EXPORT macro put before class name. This is very important.

This one is remplace by cmake with right compiler option to export C++ symbols. with G++, it's nothing, with MSVC it's the "__declspec" stuff...

To resume libpgf has never be compiled with MSVC originally, but it can be easily (i do it)

Look how it's done in libraw (not based on cmake) :

https://github.com/LibRaw/LibRaw/blob/master/libraw/libraw_types.h#L106

And classes are exported with "dlldef" like this:

https://github.com/LibRaw/LibRaw/blob/master/libraw/libraw.h#L189

This mechanism exists in digiKam but it's hidden by cmake.

This is what's you ned to patch in libpgf to link fine with MSVC in your nodejs project...

Best

Gilles Caulier

[RickCodes1]

" These rules are missing in official libpgf code (/h file in declaration of classes). " Interesting... That could actually be a fact of why building a working .lib file is actually imposible or not viable (at least at Windows) as I am expriencing ... That makes me focus on the ways that on which it has already been used,, So, despite NodeJS C++ Addons only work with .lib files for building, I must discard completely .lib files in this case... When considering DLLs, they can be called...

I don't expect you to remember precisely but I do remember from a past version that that macro wasn't there in that file. I have seen it in other files but not there until recent versions as you showed... So, apart from the relevance of the macro, the importance of the CMake code may be also high in both past and newer versions.

Reading your following lines that show cases of DLL, I see that, what you described makes sense with having found on my Windows iinstallation of the app, that it contains a DLL in \digiKam\plugins\digikam\dimg\DImg_PGF_Plugin.dll

So, currently, I was about to ask you that... while I check a way to make my JS code faster or to check a C++ addon way, could you check the updates of my repo (that included now ffi-napi module and 4 examples of calling DLLs: 2 for Windows and 2 custom ones made with cl.exe from x64 Native Tools VS cmd) ?

I wonder if in order to end up with a DecodePGF.dll file (that also included base64.h and .cpp) with your CMake code or setup to build a DLL for Windows, you could write a getDecodedStr.h and getDecodedStr.cpp based on pgfutils.cpp (but without writing a file and without using Qt) or that just uses the libpgf method that could store the PGF into a variable either for direct return or returning after converting the buffer to base64?

From JS it can be called like this following line maybe?

...
//read PGF image blob from DB and store it in a a variable
ImgData = blob;
//Importing Custom DLLs with ffi-napi
var DecodePGF = ffi.Library('DecodePGF', {
  'getDecodedStr': [ 'string', [ 'string' ] ]
});
var ConvertedOutput= DecodePGF .getDecodedStr(ImgData );
dataToSend["ConvertedOutput"] = ConvertedOutput;
...

P.S: Notice that I also wonder if string string is the right cmbination since in the updates of my repo I had to know before hand the size of the hardcoded string...

var buffer = new Buffer.alloc(17.3*1024);
b64cpplib.get_b64_string(buffer);
var actualString = buffer.toString('utf-8');

which is not so much desirable...

BTW, in my repo, the ouput of the called DLLs are: