search

JuliaInterop / CxxWrap.jl

Package to make C++ libraries available in Julia
Other
419 stars 67 forks source link

Possible bug in 0.8.2: ::Core.Compiler.Argument #151

Closed TeroFrondelius closed 3 years ago

TeroFrondelius commented 5 years ago

Something has changed, see https://travis-ci.com/JuliaFEM/MFrontInterface.jl/jobs/217771744

Error message: 

  Got exception outside of a @test
  MethodError: no method matching load(::String, ::String, ::Core.Compiler.Argument)
  Closest candidates are:
    load(::AbstractString, ::AbstractString, !Matched::MFrontInterface.behaviour.Hypothesis) at none:0
    load(!Matched::Union{SmartPointer{T2}, T2} where T2<:MFrontInterface.behaviour.FiniteStrainBehaviourOptions, ::AbstractString, !Matched::AbstractString, !Matched::MFrontInterface.behaviour.Hypothesis) at none:0

With previous 0.8.1 version the master is working: https://github.com/JuliaFEM/MFrontInterface.jl

/CC @thelfer

thelfer commented 5 years ago

Hi, probably a duplicate of Issue 36...

wbhart commented 5 years ago

We are seeing something that looks vaguely similar with Julia 1.3 and nightlies:

ERROR: LoadError: InitError: MethodError: no method matching nInitChar(::Tuple{Core.Compiler.UseRef,Nothing}, ::Ptr{Nothing})
Closest candidates are:
  nInitChar(!Matched::Singular.libSingular.n_coeffType, ::Ptr{Nothing}) at none:0
Stacktrace:
 [1] Singular.Integers() at /home/travis/build/oscar-system/Singular.jl/src/number/NumberTypes.jl:17
 [2] __init__() at /home/travis/build/oscar-system/Singular.jl/src/Singular.jl:62
 [3] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:692
 [4] _require_from_serialized(::String) at ./loading.jl:743
 [5] _require(::Base.PkgId) at ./loading.jl:1034
 [6] require(::Base.PkgId) at ./loading.jl:922
 [7] require(::Module, ::Symbol) at ./loading.jl:917
 [8] include at ./boot.jl:328 [inlined]
 [9] include_relative(::Module, ::String) at ./loading.jl:1105
 [10] include(::Module, ::String) at ./Base.jl:31
 [11] include(::String) at ./client.jl:432
 [12] top-level scope at none:6

I note that issue 36 is no longer open. We are using version 0.8.2 of CxxWrap.

The problem can be reproduced by running the tests for Singular.jl [1] with Julia-1.3 or nightlies. It doesn't occur for any earlier version of Julia. I apologise that we haven't cut it down to a smaller example.

This issue also doesn't occur for version 0.8.1 of CxxWrap.

[1] https://github.com/oscar-system/Singular.jl

wbhart commented 5 years ago

@thelfer Could you possibly say why you think issue 36 is relevant here? It looks like that issue leads to missing symbols for the linker, whereas here one gets a missing Julia function, and Compiler internals leak into user code. They don't seem related to me.

I'm just trying to clarify whether our issue is related to this one or not (we certainly link against the right libraries).

@TeroFrondelius are you still having this issue, or was it resolved for you? Does it occur with Julia-1.2 or below?

TeroFrondelius commented 5 years ago

@wbhart I am still stuck to the old version. I updated my pull request and the travis is running here: https://github.com/JuliaFEM/MFrontInterface.jl/pull/5, let's see what will happen.

TeroFrondelius commented 5 years ago

@wbhart interestingly Julia v1.0, v1.2, and nighly fails but v1.3 passes.

thelfer commented 5 years ago

@wbhart I let @TeroFrondelius handle the Julia part, so he is more capable of following this topic.

wbhart commented 5 years ago

@TeroFrondelius Thanks. I will open a separate ticket for our issue then, as it is the other way around for us, so perhaps a different issue.

barche commented 5 years ago

I have changed the code quite a lot between 0.8.2 and current master, could you try with CxxWrap master and libcxxwrap-julia 0.6.2 (which should be installed automatically when updating CxxWrap to master, I think)?

wbhart commented 5 years ago

At least in our case, we get a symbol lookup error, so it doesn't look like it automatically installs the libcxxwrap-julia 0.6.2. Sorry to ask a really ignorant question, but how would I check that? It's not listed in the manifest for our project.

/home/wbhart/julia-1.3.0-rc3/bin/julia: symbol lookup error: /home/wbhart/.julia/dev/Singular/local/lib/libsingularwrap.so: undefined symbol: _ZN5jlcxx12gc_index_mapEv
ERROR: Failed to precompile Singular [bcd08a7b-43d2-5ff7-b6d4-c458787f915c] to /home/wbhart/.julia/compiled/v1.3/Singular/9fz0y_e1zOn.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1283
 [3] _require(::Base.PkgId) at ./loading.jl:1024
 [4] require(::Base.PkgId) at ./loading.jl:922
 [5] require(::Module, ::Symbol) at ./loading.jl:917
wbhart commented 5 years ago

Actually, I found the libcxxwrap-julia and it does seem to be 0.6.2. I think what is happening is that our project is still linked against the old version. I'll try rebuilding.

wbhart commented 5 years ago

The issue seems to be some changed definition:

In file included from /home/wbhart/.julia/dev/Singular/deps/src/singular.cpp:2:0:
│ /home/wbhart/.julia/dev/Singular/deps/src/includes.h:47:20: error: ‘IsBits’ is not a class template
│  template <> struct IsBits<n_coeffType> : std::true_type {
│                     ^~~~~~
│ /home/wbhart/.julia/dev/Singular/deps/src/includes.h:47:57: error: explicit specialization of non-template ‘jlcxx::IsBits’

which prevents it building libsingularwrap.so.

Has the usage of IsBits changed?

TeroFrondelius commented 5 years ago

could you try with CxxWrap master and libcxxwrap-julia 0.6.2

I immediately had a build issue with mgis:

In file included from /workspace/srcdir/MFrontGenericInterfaceSupport/bindings/julia/src/behaviour-module.cxx:17:0:
/workspace/srcdir/MFrontGenericInterfaceSupport/bindings/julia/include/MGIS/Julia/JuliaUtilities.hxx:32:10: error: 'IsBits' is not a class template
   struct IsBits<mgis::behaviour::Variable::Type> : std::true_type {};
          ^~~~~~

Here is the full log: https://travis-ci.org/TeroFrondelius/mgisBuilder/builds/596516710

/CC @thelfer @barche

barche commented 5 years ago

Ah, yes, sorry, I put a list of breaking changes at the bottom of the CxxWrap readme, for IsBits and IsImmutable you should replace them with IsMirroredType.

thelfer commented 5 years ago

@barche Any macro to test when one shall make the change (in order to keep backward compatibility ) ?

wbhart commented 5 years ago

Making the changes relating to IsMirroredType, we are now able to build everything, but when we start up our package, we get the following message:

ERROR: LoadError: LoadError: Type Ph has no Julia wrapper
Stacktrace:
 [1] register_julia_module at /home/wbhart/.julia/packages/CxxWrap/IYyY2/src/CxxWrap.jl:284 [inlined]
 [2] readmodule(::String, ::Symbol, ::Module) at /home/wbhart/.julia/packages/CxxWrap/IYyY2/src/CxxWrap.jl:570
 [3] wrapmodule(::String, ::Symbol, ::Module) at /home/wbhart/.julia/packages/CxxWrap/IYyY2/src/CxxWrap.jl:576
 [4] top-level scope at /home/wbhart/.julia/dev/Singular/src/LibSingular.jl:5
 [5] include at ./boot.jl:328 [inlined]
 [6] include_relative(::Module, ::String) at ./loading.jl:1105
 [7] include at ./Base.jl:31 [inlined]
 [8] include(::String) at /home/wbhart/.julia/dev/Singular/src/Singular.jl:1
 [9] top-level scope at /home/wbhart/.julia/dev/Singular/src/Singular.jl:110
 [10] include at ./boot.jl:328 [inlined]
 [11] include_relative(::Module, ::String) at ./loading.jl:1105
 [12] include(::Module, ::String) at ./Base.jl:31
 [13] top-level scope at none:2
 [14] eval at ./boot.jl:330 [inlined]
 [15] eval(::Expr) at ./client.jl:433
 [16] top-level scope at ./none:3
in expression starting at /home/wbhart/.julia/dev/Singular/src/LibSingular.jl:5
in expression starting at /home/wbhart/.julia/dev/Singular/src/Singular.jl:110
ERROR: Failed to precompile Singular [bcd08a7b-43d2-5ff7-b6d4-c458787f915c] to /home/wbhart/.julia/compiled/v1.3/Singular/9fz0y_e1zOn.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1283
 [3] _require(::Base.PkgId) at ./loading.jl:1024
 [4] require(::Base.PkgId) at ./loading.jl:922
 [5] require(::Module, ::Symbol) at ./loading.jl:917

I don't know what type it is referring to. A grep of our code base doesn't seem to include the string Ph anywhere.

barche commented 5 years ago

Any macro to test when one shall make the change (in order to keep backward compatibility ) ?

@thelfer I wasn't planning that, this will all be part of CxxWrap 0.9, so package versioning should deal with it as expected?

barche commented 5 years ago

I don't know what type it is referring to. A grep of our code base doesn't seem to include the string Ph anywhere.

This can be decoded using:

c++filt -t Ph

This turns out to be unsigned char*, which means this is my fault. In what context are you using an unsigned char*? I probably forgot an automatic pointer type creation somewhere.

wbhart commented 5 years ago

We are using char quite a bit, for paths, version strings, error messages and so on. But I don't see any instances of unsigned char .

It's possible one of the functions that is "automatically" wrapped (i.e. without using lambdas) is using it somewhere. But there's hundreds of functions, so it would really take a lot of time to track it down.

How critical is it to know?

barche commented 5 years ago

The problem is that I have to identify the pathway taken to arrive at this error, normally pointer types are created automatically when functions and types are wrapped. If a function uses an unsigned char* directly, this already works.

In the mean time, I discovered that if a function takes a pointer to a function that contains an unwrapped type, this is not detected. So maybe you are passing a function pointer to a function returning unsigned char* or taking it as a parameter? If so, that case is fixed on libcxxwrap master, but not in the binaries yet.

wbhart commented 5 years ago

We are using function pointers, so this could certainly be it. I'll try to investigate a little more thoroughly. But I'm not sure I can do it this week as I have a visitor.

wbhart commented 5 years ago

I had the idea to bisect our C++ code by commenting out parts of it to refine which function was causing the problem. It seems there are numerous problems.

The following causes a message about no wrapper for Pi:

Singular.method("syBetti_internal", [](void * ra, int len, ring o) {
        const ring origin = currRing;
        rChangeCurrRing(o);
        int      dummy;
        intvec * iv = syBetti(reinterpret_cast<resolvente>(ra), len, &dummy,
                              NULL, FALSE, NULL);
        rChangeCurrRing(origin);
        int  nrows = iv->rows();
        int  ncols = iv->cols();
        auto betti = (int *)malloc(ncols * nrows * sizeof(int));
        for (int i = 0; i < ncols; i++) {
            for (int j = 0; j < nrows; j++) {
                betti[i * nrows + j] = IMATELEM(*iv, j + 1, i + 1);
            }
        }
        delete (iv);
        return std::make_tuple(betti, nrows, ncols);
    });

So I guess it accepts (void, int, ring) and returns a tuple with (int, int, int). The ring type should be a pointer to a struct (it's probably typedef'd).

The function that seems to yield the Ph error has prototype:

auto rDefault_long_helper(coeffs                        cf,
                          jlcxx::ArrayRef<uint8_t *>    vars,
                          jlcxx::ArrayRef<rRingOrder_t> ord,
                          int *                         blk0,
                          int *                         blk1,
                          unsigned long                 bitmask)

I think it returns a ring (which is probably not relevant).

The other odd issue is that if I comment out these functions I then get a message about numberRef not being defined. But this is strange given that I am doing:

    Singular.add_type<snumber>("number");

which I would have thought would also define numberRef. Though perhaps we focus on resolving the above two issues first, and we'll see if that problem goes away once I uncomment everything.

barche commented 4 years ago

OK, I pushed some changes to the master branches of CxxWrap and libcxxwrap-julia, so to test you need to compile it.

wbhart commented 4 years ago

Is it enough to just do Pkg.update("CxxWrap"); Pkg.build("CxxWrap") ? I have nothing in my environment to do with CxxWrap. Or are there some shell commands I should issue.

At present I don't see any difference with the errors I was getting.

wbhart commented 4 years ago

I also dev'd CxxWrap, which surely rebuilt it. Then I rebuilt our wrap .so and checked it was actually linked against the dev'd version of libcxxwrap_julia.so.0. But I still get the Ph error.

wbhart commented 4 years ago

Does the stack trace help?

ERROR: LoadError: LoadError: Type Ph has no Julia wrapper
Stacktrace:
 [1] register_julia_module at /home/wbhart/.julia/dev/CxxWrap/src/CxxWrap.jl:305 [inlined]
 [2] readmodule(::String, ::Symbol, ::Module) at /home/wbhart/.julia/dev/CxxWrap/src/CxxWrap.jl:596
 [3] wrapmodule(::String, ::Symbol, ::Module) at /home/wbhart/.julia/dev/CxxWrap/src/CxxWrap.jl:602
 [4] top-level scope at /home/wbhart/.julia/dev/Singular/src/LibSingular.jl:5
 [5] include at ./boot.jl:328 [inlined]
 [6] include_relative(::Module, ::String) at ./loading.jl:1105
 [7] include at ./Base.jl:31 [inlined]
 [8] include(::String) at /home/wbhart/.julia/dev/Singular/src/Singular.jl:1
 [9] top-level scope at /home/wbhart/.julia/dev/Singular/src/Singular.jl:110
 [10] include at ./boot.jl:328 [inlined]
 [11] include_relative(::Module, ::String) at ./loading.jl:1105
 [12] include(::Module, ::String) at ./Base.jl:31
 [13] top-level scope at none:2
 [14] eval at ./boot.jl:330 [inlined]
 [15] eval(::Expr) at ./client.jl:433
 [16] top-level scope at ./none:3
in expression starting at /home/wbhart/.julia/dev/Singular/src/LibSingular.jl:5
in expression starting at /home/wbhart/.julia/dev/Singular/src/Singular.jl:110
barche commented 4 years ago

No, but the bisection did. I'm pretty sure the Ph error is fixed in master, but you need to build libcxxwrap and then link the C++ part of your wrapper against the new wrapper (see the libcxxwrap readme). I'm still going over the other issues and will then do a new libcxxwrap-julia release with binaries.

wbhart commented 4 years ago

I've managed to build libcxxwrap according to the instructions and build CxxWrap to use that (although the build seems to take only a fraction of a second). However, linking our code against it seems to be complicated. Simply setting JlCxx_DIR doesn't seem to be enough (we were already doing this, so I simply changed the directory name to the new location).

That sets off errors from CMake about CMake module files that it wants supplied, and other things which are not possible for me to debug (I don't know anything about CMake).

We are going to have to wait for binaries to try this out, unfortunately. It seems whoever set this up in the first place, at our end, didn't go for a straightforward installation.

barche commented 4 years ago

OK, the new libcxxwrap-julia binaries are out now, usable with CxxWrap master.

jstrube commented 3 years ago

Looks like this can be closed?

barche commented 3 years ago

Yes, I think so too, I haven't seen this happen with the new binaries.