search

JuliaParallel / PETSc.jl

Julia wrappers for the PETSc library
Other
121 stars 38 forks source link

Automatically wrapping with `Clang.jl` #149

Open jkozdon opened 3 years ago

jkozdon commented 3 years ago

Experimenting to play with #147

Todo

jkozdon commented 3 years ago

Doesn't work with everything yet but you can call initialize and finalize.

jkozdon commented 3 years ago

Input:

# For some reason you need to initialize MPI before using multiple libraries.
# Need to debug this
using MPI
MPI.Initialized() || MPI.Init()
using PETSc
for petsclib in PETSc.LibPETSc.petsclibs
    println(splitpath(petsclib.petsc_library)[end])
    @show PETSc.initialized(petsclib)
    @show PETSc.initialize(petsclib)
    @show PETSc.initialized(petsclib)
    @show PETSc.finalized(petsclib)
    @show PETSc.finalize(petsclib)
    @show PETSc.initialized(petsclib)
    @show PETSc.finalized(petsclib)
    println()
end

Output:

libpetsc_double_real_Int64.dylib
PETSc.initialized(petsclib) = false
PETSc.initialize(petsclib) = nothing
PETSc.initialized(petsclib) = true
PETSc.finalized(petsclib) = false
PETSc.finalize(petsclib) = nothing
PETSc.initialized(petsclib) = false
PETSc.finalized(petsclib) = true

libpetsc_single_real_Int64.dylib
PETSc.initialized(petsclib) = false
PETSc.initialize(petsclib) = nothing
PETSc.initialized(petsclib) = true
PETSc.finalized(petsclib) = false
PETSc.finalize(petsclib) = nothing
PETSc.initialized(petsclib) = false
PETSc.finalized(petsclib) = true

libpetsc_double_complex_Int64.dylib
PETSc.initialized(petsclib) = false
PETSc.initialize(petsclib) = nothing
PETSc.initialized(petsclib) = true
PETSc.finalized(petsclib) = false
PETSc.finalize(petsclib) = nothing
PETSc.initialized(petsclib) = false
PETSc.finalized(petsclib) = true

libpetsc_single_complex_Int64.dylib
PETSc.initialized(petsclib) = false
PETSc.initialize(petsclib) = nothing
PETSc.initialized(petsclib) = true
PETSc.finalized(petsclib) = false
PETSc.finalize(petsclib) = nothing
PETSc.initialized(petsclib) = false
PETSc.finalized(petsclib) = true
jkozdon commented 3 years ago

I guess I still need to sort out the types (But I think it should be possible with more walking of the syntax tree).

psanan commented 3 years ago

Lazy question, but what's the high-level procedure here? What is now automatic and what still has to be maintained "by hand"? Are the specific things here related to various PETSc components all of the "by hand" flavor, or are those things that have been generated automatically and are then checked in?

vchuravy commented 3 years ago

I think everything in lib is autogenerated.

jkozdon commented 3 years ago

As @vchuravy said, the file in lib is generated from the PETSc header files with the code in gen. These produce pretty raw wrappers, that I am then expanding on to make it all a little more friendly to work with.

There is a bunch of stuff we do not currently wrap (look at generator.toml), some of it will need to be sorted out and some of it won't (I think I know / have a plan for how to support most of what will be needed, just waiting to fix it until I am ready to use it / write the test).

There probably some gotchas left in there with respect to system / build dependent types that will need to be sorted out, but right now it does support multiple PETSc builds living side-by-side in Julia.

@psanan shoot me an email if you'd like to VTC to chat about what I am doing, I'd be more than happy to have feedback.

psanan commented 3 years ago

Docs suggestion, re our chat our yesterday. Perhaps somewhere like here one could add language like "PETSc.jl's high-level, easy-to-use interface exposes only some of the functionality of PETSc. However, PETSc.jl includes an automatically-generated wrapper layer which directly exposes more of the PETSc API, which may be of use to existing expert users. See (link to minimal example/test of using the wrappers, directly) for an example".

jkozdon commented 3 years ago

Thanks for the recommendation. I've put it as part of the PR task list.

ViralBShah commented 1 month ago

It would be really cool to get this working and merged. I have no idea what the challenges were (and plan to dive in), but is it possible that improvements in Clang.jl will help us get this done?

boriskaus commented 1 month ago

I think there were no real challenges, but it was probably mainly caused by the main developer wanting things to be perfect before merging it back into main.

Clang already used to do a good job in wrapping the functions. Making a nice julia-like interface around this would be good to have, but that had to be done manually which is what took a long time (and was never finished). I guess Clang can now even take the help into account in the documentation

ViralBShah commented 1 month ago

I feel that it would be nice to get it refreshed and merged and it may help others to contribute if all the auto-wrapping is in place. Kind of imagining moving things one step forward.

Since the package is not even loading right now, once that is fixed, this sort of thing may help us find other maintainers.

boriskaus commented 1 month ago

Another thing was the interaction with the GC. There was a presentation about how this was solved in python (for firedrake) at this year’s PETSc conference, so I suppose the same method can be used in julia

boriskaus commented 1 month ago

I fully agree with you. If the basic functions are wrapped and we have some basic documentation it would motivate others to help. It probably won’t take professionals that long. I, however, don’t belong to that group. Happy to help with usage examples etc.

ViralBShah commented 1 month ago

I'll see how I can help with getting the package back up and running and get this PR merged.

boriskaus commented 1 month ago

That would be fantastic!

ViralBShah commented 3 weeks ago

Just leaving this one open to be able to drive work on it.

ViralBShah commented 3 weeks ago

@boriskaus This has a pretty large set of conflicts. Any suggestions on how to get it rebased to main quickly? Are some of these just auto-generated files?

ViralBShah commented 3 weeks ago

Tried to update this branch to Julia 1.11. The package loads fine in Julia 1.11 with some minor tweaks (I checked in my commits).

The clang autowrap stuff prints a few errors (naturally it's been a long time), but it does write a petsc_library.jl. I see this:

┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESObjectiveComputeFunctionDefaultFD, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESObjectiveComputeFunctionDefaultFD(SNES, Vec, Vec, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESComputeJacobianDefault, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESComputeJacobianDefault(SNES, Vec, Mat, Mat, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESComputeJacobianDefaultColor, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESComputeJacobianDefaultColor(SNES, Vec, Mat, Mat, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESPicardComputeFunction, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESPicardComputeFunction(SNES, Vec, Vec, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESPicardComputeMFFunction, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESPicardComputeMFFunction(SNES, Vec, Vec, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESPicardComputeJacobian, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESPicardComputeJacobian(SNES, Vec, Mat, Mat, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:TSComputeRHSFunctionLinear, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) TSComputeRHSFunctionLinear(TS, PetscReal, Vec, Vec, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:TSComputeRHSJacobianConstant, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) TSComputeRHSJacobianConstant(TS, PetscReal, Vec, Mat, Mat, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
ViralBShah commented 3 weeks ago

Part of the issue is that the generator seems to be for the default Float64/Int32 combination, and I wonder if it needs to be extended to generate all the type combinations.

boriskaus commented 3 weeks ago

I think we can first focus on getting this working before doing the rest. Part of the problem is likely that some of the PETSc functions have been renamed.

ViralBShah commented 3 weeks ago

I'm going to try to just get this branch working without worrying about rebasing. I am already able to generate the full petsc_library.jl but it doesn't load yet. You can try it out on this branch by running generator.jl.

I don't think major changes have happened since this branch was created and now.

boriskaus commented 3 weeks ago

fine with me; ideally we would change the existing tests that are currently in main to only use the auto-generated wrappers, which would simplify maintaining it.

boriskaus commented 3 weeks ago

We don't have to strictly follow what was done before, since Clang.jl has presumably improved a lot over the last few years.

If I simplify generator.toml to:

[general]
library_name = "$petsc_library"

use_julia_native_enum_type = true
use_deterministic_symbol = true

prologue_file_path = "./prologue.jl"
# epilogue_file_path = "./epilogue.jl"

auto_mutability = true              # Create mutable structures:
auto_mutability_with_new = true     # Allow creating empty structs
auto_mutability_ignorelist = []
auto_mutability_includelist = []

extract_c_comment_style = "doxygen"

export_symbol_prefixes = ["CX", "clang_"]

It essentially compiles the whole library with a few exceptions:

julia> include("generator.jl")
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscbag.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsys.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscconf.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscconf_poison.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscfix.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscmacros.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscversion.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsc/private/petscadvancedmacros.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsystypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscmath.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscerror.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscviewertypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscoptions.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsclog.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsctime.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscbt.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscstring.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsclogtypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsclogdeprecated.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscmatlab.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdraw.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdrawtypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdevice.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdevicetypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsf.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsftypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscvec.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscis.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsectiontypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscistypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscviewer.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsection.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmda.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdm.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscmat.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscmatcoarsen.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmtypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscfetypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdstypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmlabel.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdt.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmdatypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscpf.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscao.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscfe.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscspace.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdualspace.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmcomposite.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmpatch.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmplex.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscpartitioner.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmplextypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscfv.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscfvtypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmfield.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmplextransform.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmplextransformtypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmredundant.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmshell.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmsliced.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmswarm.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmstag.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscdmproduct.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscds.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsccharacteristic.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscts.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsnes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscksp.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscpc.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscpctypes.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscconvest.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsctao.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsctaolinesearch.h
[ Info: Found dependent header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsctao_deprecations.h
[ Info: Parsing headers...
/Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petscsys.h:98:6: error: "PETSc was configured with Open MPI but now appears to be compiling using a non-Open MPI mpi.h"
[ Info: Processing header: /Users/kausb/.julia/artifacts/a69b8f699a39f274f95752394759a3e9973e02b0/lib/petsc/double_real_Int64/include/petsc.h
[ Info: Building the DAG...
┌ Warning: [CollectDependentSystemNode]: found symbols in the system headers: [:MPI_Info, :MPI_Op, :MPI_Status, :MPI_Status, :MPI_Datatype, :off_t, :__darwin_off_t, :MPI_Comm, :MPI_Aint, :MPI_File, :ADIOI_FileD, :MPI_Request, :MPI_Group, :MPI_Offset, :MPI_Win]
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/passes.jl:95
[ Info: Emit Julia expressions...
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESObjectiveComputeFunctionDefaultFD, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESObjectiveComputeFunctionDefaultFD(SNES, Vec, Vec, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESComputeJacobianDefault, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESComputeJacobianDefault(SNES, Vec, Mat, Mat, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESComputeJacobianDefaultColor, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESComputeJacobianDefaultColor(SNES, Vec, Mat, Mat, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESPicardComputeFunction, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESPicardComputeFunction(SNES, Vec, Vec, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESPicardComputeMFFunction, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESPicardComputeMFFunction(SNES, Vec, Vec, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:SNESPicardComputeJacobian, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) SNESPicardComputeJacobian(SNES, Vec, Mat, Mat, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:TSComputeRHSFunctionLinear, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) TSComputeRHSFunctionLinear(TS, PetscReal, Vec, Vec, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
┌ Error: missing implementation for ExprNode{Clang.Generators.FunctionDefault, Clang.CLFunctionDecl}(:TSComputeRHSJacobianConstant, Clang.Generators.FunctionDefault(), CLCursor (Clang.CLFunctionDecl) TSComputeRHSJacobianConstant(TS, PetscReal, Vec, Mat, Mat, void *), Expr[], Expr[], Int64[]), please file an issue to Clang.jl.
└ @ Clang.Generators ~/.julia/packages/Clang/uJQ2L/src/generator/codegen.jl:716
[ Info: [ProloguePrinter]: print to /Users/kausb/.julia/dev/PETSc.jl/gen/../lib/petsc_library.jl
[ Info: [GeneralPrinter]: print to /Users/kausb/.julia/dev/PETSc.jl/gen/../lib/petsc_library.jl
[ Info: [EpiloguePrinter]: print to /Users/kausb/.julia/dev/PETSc.jl/gen/../lib/petsc_library.jl
[ Info: Done!
"/Users/kausb/.julia/dev/PETSc.jl/gen/../lib/petsc_library.jl"
boriskaus commented 3 weeks ago

@ViralBShah I've looked at the tests that are marked with FIXME, and most of the changes are due to PETSc changing the function calls (slightly) during more recent versions. We can port that manually but it probably makes more sense to try and autowrap the full library and use that instead (surely easier to maintain).

Also, this PR is very large. Perhaps it is simpler to start a new one from main?

ViralBShah commented 3 weeks ago

Yes, the simpler thing would be to start the generator on a new PR in main, and just get that running first. And then piecemeal port things over to the new generated API. But not being a user of PETSc, that is where I will start running into my limitations here. Let's push one step forward at least.

I do feel that on main, it would be nice to fix all the FIXME things first so that everything that was working in the past is working again now, and tagged - giving a solid baseline for the new stuff.

boriskaus commented 3 weeks ago

I’m a PETSc user (well at least of part of the library). Yet I’m only a beginner when it comes to using clang to port it all. I did notice in a recent project that it is possible to automatically make julia docs out of doxygen strings (which every routine in petsc has). It would be fantastic if we can automatically provide help for all of the routines (even when the help is strictly speaking only valid for the c routines)

If you leave out the functions that have ccalls in the current main, there are only a limited number of additional functions remaining that are mainly there to make using petsc more julia-like. Those ones could be gradually expanded to include more functionally, but as long as we call autowrapped routines throughout it will be much easier to maintain this.

ViralBShah commented 3 weeks ago

So with your generated library, did you try doing include petsc_library.jl? Getting it to load cleanly is the first step. I assume you used the generator.toml that is in this branch (which has a lot of stuff to exclude and such).

I see what you are saying about the auto-generated comments for the wrappers. Wouldn't we still need the docs that are in the Julian interface as it stands right now - or are you thinking those could be completely replaced with this auto-generated stuff?

boriskaus commented 3 weeks ago

So with your generated library, did you try doing include petsc_library.jl? Getting it to load cleanly is the first step. I assume you used the generator.toml that is in this branch (which has a lot of stuff to exclude and such).

I didn’t manage to load it just now; I was using a simpler generator.toml shown in one of the comments

I see what you are saying about the auto-generated comments for the wrappers. Wouldn't we still need the docs that are in the Julian interface as it stands right now - or are you thinking those could be completely replaced with this auto-generated stuff?

I think it can be replaced with autogenerated stuff. Multiple dispatch will take care of the rest.

boriskaus commented 1 week ago

@ViralBShah, @vchuravy

One of the reasons for the wrapper file to be rather complex is that the different versions of the PETSc libraries have different definitions of PetscInt, PetscScalar, etc. which can be 32bit, 64bit etc. In the main branch, we handle this by interpolating the values of $PetscInt, $PetscScalar into function calls as can be seen here

Yet, this doesn't work for structures, which are instead defined as:

mutable struct DMSTAGSTENCIL{PetscInt}
    loc::DMStagStencilLocation
    i::PetscInt
    j::PetscInt
    k::PetscInt
    c::PetscInt
end

At the time the initial wrapping was done, it wasn't possible to automatically wrap C structures to give something like above. Instead Clang gives something like:

mutable struct DMStagStencil
    loc::DMStagStencilLocation
    i::PetscInt
    j::PetscInt
    k::PetscInt
    c::PetscInt
    DMStagStencil() = new()
end

Question: do you know whether there is a way to automatically add types to the structures using Clang? If yes, it would greatly simplify the generator.toml file.

EDIT: @albert-de-montserrat pointed out that this is actually done already in wrap.jl. The disadvantage is that it has to be specified manually for every struct as:

(x, s) -> add_struct_type(x, s, ("MatStencil",), ("PetscInt",)),
boriskaus commented 3 days ago

@ViralBShah it now loads again and some of the tests work. Quite a bit of work remains to automatically wrap the structs (if that is possible at all).

boriskaus commented 3 days ago

In principle yes, structs do get wrapped automatically with Clang, Yet, I had to deactivate a large number of them to get the package to load (plus all functions that call unwrapped structs). I suppose that this is what was done before as well. Having a way to automatize this process would certainly help.

ViralBShah commented 2 days ago

Perhaps @gnimuc can chime in, if they have the time to look at this.