JuliaInterop / Clang.jl

C binding generator and Julia interface to libclang
https://juliainterop.github.io/Clang.jl/
MIT License
223 stars 68 forks source link

Macro help #255

Closed bramtayl closed 3 years ago

bramtayl commented 4 years ago

This is mostly just a request for help. I'm trying to resurrect Sodium.jl. I've got a binary version build via BinaryBuilder so you can now access it via using libsodium_jll.

I'm trying to use Clang to automatically generate a wrapper, but I'm running into a number of issues. The most common one seems to be SODIUM_MIN is a commonly used macro in the codebase, but Clang doesn't seem to be able to parse the macro. Is there way to register the macro (e.g. map it to the julia function min)?

You can see my progress at my branch here https://github.com/bramtayl/Sodium.jl

This is the output I get when I run generate.jl:

WARNING: redefining constant LIBSODIUM_HEADERS
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/core.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_aead_aegis256.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_aead_aes256gcm.h ...
┌ Warning: Skipping struct: "CLCursor (CLStructDecl) crypto_aead_aes256gcm_state_" due to unsupported field: CLCursor (CLUnexposedAttr) 
└ @ Clang ~/.julia/packages/Clang/QLEoj/src/wrap_c.jl:156
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_aead_chacha20poly1305.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_aead_xchacha20poly1305.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_auth.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_auth_hmacsha256.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_auth_hmacsha512.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_auth_hmacsha512256.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_box.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_box_curve25519xchacha20poly1305.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_box_curve25519xsalsa20poly1305.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_core_ed25519.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_core_hchacha20.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_core_hsalsa20.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_core_ristretto255.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_core_salsa20.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_core_salsa2012.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_core_salsa208.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_generichash.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_generichash_blake2b.h ...
┌ Warning: Skipping struct: "CLCursor (CLStructDecl) crypto_generichash_blake2b_state" due to unsupported field: CLCursor (CLUnexposedAttr) 
└ @ Clang ~/.julia/packages/Clang/QLEoj/src/wrap_c.jl:156
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_hash.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_hash_sha256.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_hash_sha512.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_kdf.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_kdf_blake2b.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_kx.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_onetimeauth.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_onetimeauth_poly1305.h ...
┌ Warning: Skipping struct: "CLCursor (CLStructDecl) crypto_onetimeauth_poly1305_state" due to unsupported field: CLCursor (CLUnexposedAttr) 
└ @ Clang ~/.julia/packages/Clang/QLEoj/src/wrap_c.jl:156
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_pwhash.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_pwhash_argon2i.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_pwhash_argon2id.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_pwhash_scryptsalsa208sha256.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_scalarmult.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_scalarmult_curve25519.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_scalarmult_ed25519.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_scalarmult_ristretto255.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_secretbox.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_secretbox_xchacha20poly1305.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_secretbox_xsalsa20poly1305.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_secretstream_xchacha20poly1305.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_shorthash.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_shorthash_siphash24.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_sign.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_sign_ed25519.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_sign_edwards25519sha512batch.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_stream.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_stream_chacha20.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_stream_salsa20.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_stream_salsa2012.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_stream_salsa208.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_stream_xchacha20.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_stream_xsalsa20.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_verify_16.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_verify_32.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/crypto_verify_64.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/export.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/randombytes.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/randombytes_internal_random.h ...
┌ Warning: not wrapping CLCursor (CLVarDecl) randombytes_internal_implementation
└ @ Clang ~/.julia/packages/Clang/QLEoj/src/wrap_c.jl:443
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/randombytes_sysrandom.h ...
┌ Warning: not wrapping CLCursor (CLVarDecl) randombytes_sysrandom_implementation
└ @ Clang ~/.julia/packages/Clang/QLEoj/src/wrap_c.jl:443
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/runtime.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
[ Info: wrapping header: /home/brandon/libsodium/src/libsodium/include/sodium/utils.h ...
[ Info: writing /home/brandon/.julia/dev/Sodium.jl/src/libsodium_api.jl
Gnimuc commented 4 years ago

I cannot open that link, is that a private repo?

The most common one seems to be SODIUM_MIN is a commonly used macro in the codebase, but Clang doesn't seem to be able to parse the macro. Is there way to register the macro (e.g. map it to the julia function min)?

Is that macro a true public API of libsodium? These kinda macros are usually for implementing C functions as C doesn't support polymorphism, but is it really used when calling the API functions? The only use-case I can imagine is that it will reduce efforts when translating C code. Anyway, this can be done by manually define const SODIUM_MIN = min in Julia.

However, it's tricky to let Clang.jl do this kinda thing automatically because it's not smart enough to understand why the SODIUM_MIN should be mapped to min. To tell this info to Clang.jl, you can either use a rewriter or overload wrap_c, please refer to this post for further details.

bramtayl commented 4 years ago

Sorry I gave up and deleted it. I decided just to use ccall because I just needed one function from libsodium. It doesn't seem to have worked correctly either; the code for that is here:

https://github.com/bramtayl/OnlinePackage.jl/blob/master/src/OnlinePackage.jl

Gnimuc commented 4 years ago

OK, I just did this https://github.com/Gnimuc/Sodium.jl ;)

Gnimuc commented 4 years ago

And I didn't see sodium_init() is called anywhere in your package. https://download.libsodium.org/doc/usage

bramtayl commented 4 years ago

Wow thanks!!!

bramtayl commented 4 years ago

Thanks for the advice, and also, for the package! Since sodium is being used by github, I bet a lot of people will find the package useful.

I added a call to sodium_init to my code, and it still isn't working :( I also tried using the types listed in the package (Ptr{Cuchar}, Ptr{Cuchar}, Culonglong, Ptr{Cuchar}) for cryptobox seal instead of what I have now (Ptr{UInt8}, Cstring, Cint, Ptr{UInt8}) and that didn't work either. What's puzzling me is that after I run the function, the resulting string contains many NUL characters (well at least, findall(x .== 0, raw_encoded)) returns many values, and if I'm understanding correctly, NUL should signify the end of a string?

bramtayl commented 4 years ago

The resulting raw_encoded is also different every time I run the function, which doesn't seem right...

Gnimuc commented 4 years ago

Could you share an MWE so I can debug? This test works fine.

bramtayl commented 4 years ago

Omigosh you're the best! I posted a MWE on my discourse post.

bramtayl commented 4 years ago

Well, I tried using the package you wrote and updated online package based on the test you linked. And I'm still getting the same error...which suggests to me this isn't a problem with ccall after all. Something else must be wrong :( Anyway, thanks so much for your help! I don't know if @amitmurthy is still around but if he is maybe he can accept your rewrite of Sodium as a PR?

Gnimuc commented 4 years ago

I can transfer the package to him or JuliaWeb. I'd like to keep this issue open since we do need to add enhancement for dealing with function-like macros.

bramtayl commented 4 years ago

@Gnimuc ok. Let me know where your sodium code ends up. I've added it as a GitHub dependency to OnlinePackage for now