search

conda-forge / omniorb-feedstock

A conda-smithy repository for omniorb.
BSD 3-Clause "New" or "Revised" License
0 stars 9 forks source link

windows marshalling issue #37

Closed JeanLucPons closed 1 year ago

JeanLucPons commented 1 year ago

Solution to issue cannot be found in the documentation.

Issue

Hello, It seems that there is a marshalling issue with 4.2.5 conda forge build (Windows 64bit). More details in the following issue: https://gitlab.com/tango-controls/cppTango/-/issues/1066 Thanks

Installed packages

(base) N:\tango\pytango\jlptest>conda list
# packages in environment at C:\ProgramData\miniconda3:
#
# Name                    Version                   Build  Channel
boost                     1.78.0          py310h220cb41_4    conda-forge
boost-cpp                 1.78.0               h9f4b32c_1    conda-forge
brotlipy                  0.7.0           py310h2bbff1b_1002
bzip2                     1.0.8                he774522_0
ca-certificates           2022.12.7            h5b45459_0    conda-forge
certifi                   2022.12.7          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h2bbff1b_3
charset-normalizer        2.0.4              pyhd3eb1b0_0
colorama                  0.4.6           py310haa95532_0
conda                     23.1.0          py310haa95532_0
conda-content-trust       0.1.3           py310haa95532_0
conda-package-handling    2.0.2           py310haa95532_0
conda-package-streaming   0.7.0           py310haa95532_0
console_shortcut_miniconda 0.1.1                haa95532_1
cpptango                  9.4.1                h643942d_0    conda-forge
cryptography              38.0.4          py310h21b164f_0
idna                      3.4             py310haa95532_0
intel-openmp              2023.0.0         h57928b3_25922    conda-forge
libblas                   3.9.0              16_win64_mkl    conda-forge
libcblas                  3.9.0              16_win64_mkl    conda-forge
libffi                    3.4.2                hd77b12b_6
libhwloc                  2.8.0                h039e092_1    conda-forge
libiconv                  1.17                 h8ffe710_0    conda-forge
libjpeg-turbo             2.1.4                hcfcfb64_1    conda-forge
liblapack                 3.9.0              16_win64_mkl    conda-forge
libsodium                 1.0.18               h8d14728_1    conda-forge
libxml2                   2.10.3               hc3477c8_0    conda-forge
libzlib                   1.2.13               hcfcfb64_4    conda-forge
menuinst                  1.4.19          py310h59b6b97_0
mkl                       2022.1.0           h6a75c08_874    conda-forge
numpy                     1.24.2          py310hd02465a_0    conda-forge
omniorb-libs              4.2.5                hb04206d_4    conda-forge
openssl                   1.1.1t               hcfcfb64_0    conda-forge
packaging                 23.0               pyhd8ed1ab_0    conda-forge
pip                       22.3.1          py310haa95532_0
pluggy                    1.0.0           py310haa95532_1
powershell_shortcut_miniconda 0.0.1                haa95532_1
psutil                    5.9.4           py310h8d17308_0    conda-forge
pthreads-win32            2.9.1                hfa6e2cd_3    conda-forge
pycosat                   0.6.4           py310h2bbff1b_0
pycparser                 2.21               pyhd3eb1b0_0
pyopenssl                 22.0.0             pyhd3eb1b0_0
pysocks                   1.7.1           py310haa95532_0
pytango                   9.4.0           py310hdd9df50_0    conda-forge
python                    3.10.9               h966fe2a_0
python_abi                3.10                    2_cp310    conda-forge
requests                  2.28.1          py310haa95532_0
ruamel.yaml               0.17.21         py310h2bbff1b_0
ruamel.yaml.clib          0.2.6           py310h2bbff1b_1
setuptools                65.6.3          py310haa95532_0
six                       1.16.0             pyhd3eb1b0_1
sqlite                    3.40.1               h2bbff1b_0
tbb                       2021.7.0             h91493d7_1    conda-forge
tk                        8.6.12               h2bbff1b_0
toolz                     0.12.0          py310haa95532_0
tqdm                      4.64.1          py310haa95532_0
tzdata                    2022g                h04d1e81_0
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
urllib3                   1.26.14         py310haa95532_0
vc                        14.2                 h21ff451_1
vs2015_runtime            14.34.31931         h4c5c07a_10    conda-forge
wheel                     0.37.1             pyhd3eb1b0_0
win_inet_pton             1.1.0           py310haa95532_0
wincertstore              0.2             py310haa95532_2
xz                        5.2.10               h8cc25b3_1
zeromq                    4.3.4                h0e60522_1    conda-forge
zlib                      1.2.13               hcfcfb64_4    conda-forge
zstandard                 0.18.0          py310h2bbff1b_0
zstd                      1.5.2                h12be248_6    conda-forge

Environment info

(base) N:\tango\pytango\jlptest>conda info

     active environment : base
    active env location : C:\ProgramData\miniconda3
            shell level : 3
       user config file : C:\Users\pons\.condarc
 populated config files :
          conda version : 23.1.0
    conda-build version : not installed
         python version : 3.10.9.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=11.1=0
                          __win=0=0
       base environment : C:\ProgramData\miniconda3  (writable)
      conda av data dir : C:\ProgramData\miniconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\ProgramData\miniconda3\pkgs
                          C:\Users\pons\.conda\pkgs
                          C:\Users\pons\AppData\Local\conda\conda\pkgs
       envs directories : C:\ProgramData\miniconda3\envs
                          C:\Users\pons\.conda\envs
                          C:\Users\pons\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/23.1.0 requests/2.28.1 CPython/3.10.9 Windows/10 Windows/10.0.19045
          administrator : True
             netrc file : None
           offline mode : False
sdebionne commented 1 year ago

Is the CMake port/patch used for the Windows build in 47e1caadf0f2152a5504fda0cb7e3d28c4725858 coming from CaeruleusAqua/omniORB-cmake ?

beenje commented 1 year ago

As this repo isn't maintained anymore, I created a fork: https://github.com/beenje/omniORB-cmake I updated it for 4.2.5.

Issue could indeed come from here. It could also be some compilation flags specific to conda-forge?

sdebionne commented 1 year ago

I try to compare the autotools build system and the CMake one but so far nothing... any progress on your side?

I would be really surprised if the compiler activation script of conda forge set compiler options that would break the build of the project.

For now, a safety measure would be to mark the Windows package as broken .

beenje commented 1 year ago

Sorry I haven't had time to check yet. Will try to investigate in the coming days.

beenje commented 1 year ago

To give some update. I spent quite some time on this issue but still haven't found a fix.

I decided to give up on the cmake fork (which requires some effort to maintain) and to use the default build system. Unfortunately I haven't managed to compile yet in a conda env. I posted on the omniORB mailing list but haven't got any feedback yet. ESRF will ask Duncan to have a look.

sdebionne commented 1 year ago

@beenje in mk/win32.mk you need to replace /manifest by -manifest and /outputresource by -outputresource.

beenje commented 1 year ago

@beenje in mk/win32.mk you need to replace /manifest by -manifest and /outputresource by -outputresource.

Yes, thanks, I did (not for /outputresource but for /manifest). I also had to replace DUMPBIN.EXE /SYMBOLS with DUMPBIN.EXE -SYMBOLS. Otherwise the / is replaced with the path of the conda env...

I opened a draft PR (in link). I compiled locally with python 3.11 and got it working. Still having some issues with other python versions that I have to investigate. Another issue is that it's not ABI compatible with the previous build. Not sure what to do about that.

sdebionne commented 1 year ago

I found out that you made a lot of progress but after investigating your issue on the mailing list... great job!

Regarding the ABI incompatibility in 

public: __cdecl CORBA::TypeCode_member::~TypeCode_member(void) __ptr64

is the __ptr64 the issue?

sdebionne commented 1 year ago

Looking at the dll generated (e.g. omniDynamic425_vc16_rt.dll) with the official build system (that I also built locally), I get:

image

It would be interesting to check with dependency walker why the tango.dll complains (and if it loads the proper version of the omniorb dll).

beenje commented 1 year ago

Reynald found the -manifest trick on Monday and I did some progress since. You can see the full patch I applied on the 4.2.5 sources here: https://github.com/conda-forge/omniorb-feedstock/pull/38/files#diff-20ae539d30792fad4cddd68d7138913e44b8c460d3ec228d57119b01d826fb10

It includes some modifications to keep the same DLL name as in the previous build (without _vc16 and only the major version number). The -MACHINE:X64, I took from https://github.com/tango-controls/omniorb-windows-ci/blob/master/appveyor.yml

sdebionne commented 1 year ago

Here are the missing symbols in omniorb4_rt.dll:

CORBA::Any::~Any(void)
CORBA::TypeCode_member::~TypeCode_member(void)
void CORBA::IDLType_Helper::release(class CORBA::_objref_IDLType *)

This is probably due to a change of c++ standard, /std:c++14 being the default since msvc 2017. I guess cpptango needs to be rebuilt with the same compiler options...

beenje commented 1 year ago

I rebuilt omniorb as well and cpptango (9.3.6 and 9.4.1) and pytango (9.3.6 and 9.4.1). @sdebionne could you please try? You'll need the latest build from each package.

I also tested TangoTest (didn't have to recompile it).

sdebionne commented 1 year ago

Thanks for the update, everything looks good to me! This issue was painful for Tango / Conda / Windows users so I am really grateful that it's been fixed.

For the records, just updating pytango wont work, it would not update cpptango and omniorb-libs automatically. Since the versions are not binary compatible (e.g. it would segfault), incrementing the build number is probably not enough...

>mamba update omniorb-libs cpptango pytango
>mamba list
# Name                    Version                   Build  Channel
cpptango                  9.4.1                h7f68e99_2    conda-forge
pytango                   9.4.1           py311h44fd347_1    conda-forge
omniorb-libs              4.2.5                h27dd91c_6    conda-forge

@conda-forge-admin The build system used for building omniorb has been updated and the resulting builds are binary incompatible with the previous ones. Just incrementing the build number would not prevent to mix incompatible binaries. What would be the proper way to handle that issue?