Open jeromegn opened 6 years ago
Hello,
The parser.files
section shouldn't need memory
, as its a system header (Not memory.h
, the C++ stdlib headers don't have any ending). That section is only needed to tell Bindgen (Actually the clang parser part) where to find the declarations. If a normal C++ program only needs to #include <v8.h>
, then that's all you need too.
The error looks like the clang
part doesn't find its own system headers. Make sure that you don't move (copy, rename, etc.) anything related to clang (Both in your systems clang and bindgen).
Assuming you didn't move files, you can dive into what's going on by setting the VERBOSE
environment variable when running bindgen: VERBOSE=1 lib/bindgen/tool.sh v8.yml
That'll then proceed to tell you the (possibly long) command it's running to call out to the clang
part. With that in hand, you can then check why it's not finding its system headers. That command should understand the standard clang
arguments.
A quick try would be to add the path to your system includes using -I
.
This is however an artifact of bindgen not having a MacOS port. Afaik, the clang
on MacOS wants some additional include paths (compared to on Linux), which the build script doesn't know about. The build script for that is clang/find_clang.cr
.
On that note, does it work at all? Is the test suite running (Standard crystal spec
for that) fine?
This is what I get when I run it with VERBOSE=1
Runner command: /Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/src/bindgen/parser/../../../clang/bindgen /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.tfMeLu -m "" -f "" -- -x c++ -std=c++11 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS
In file included from /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.tfMeLu:2:
/usr/local/include/v8.h:21:10: fatal error: 'memory' file not found
#include <memory>
^~~~~~~~
I can't run using the lib/bindgen/tool.sh
directly because it somehow is broken. Maybe just a macos thing.
Error: File ./lib/bindgen/src/bindgen.cr does not exist
As for the spec, I got this:
$ crystal spec
.................................................................................................................................................................................................................................................................................................................................In file included from /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.FBLx6t:1:
/Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/spec/integration/arguments.cpp:1:10: fatal error: 'string' file not found
#include <string>
^~~~~~~~
arguments.cpp:22:32: error: no viable conversion from 'int' to 'std::string' (aka 'basic_string<char, char_traits<char>, allocator<char>
>')
return _self_->defaultString(str);
^~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/string:763:5: note: candidate
constructor not viable: no known conversion from 'int' to 'const std::__1::basic_string<char> &' for 1st argument
basic_string(const basic_string& __str);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/string:768:5: note: candidate
constructor not viable: no known conversion from 'int' to 'std::__1::basic_string<char> &&' for 1st argument
basic_string(basic_string&& __str)
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/string:778:31: note: candidate
constructor not viable: no known conversion from 'int' to 'const value_type *' (aka 'const char *') for 1st argument
_LIBCPP_INLINE_VISIBILITY basic_string(const value_type* __s);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/string:810:5: note: candidate
constructor not viable: no known conversion from 'int' to 'initializer_list<value_type>' (aka 'initializer_list<char>') for 1st
argument
basic_string(initializer_list<value_type> __il);
^
../arguments.cpp:21:33: note: passing argument to parameter 'str' here
int defaultString(std::string str = "Okay") {
^
1 error generated.
Build step failed!
Directory: /Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/spec/integration/tmp
Command: c++ -std=c++11 -c -o arguments.o arguments.cpp -I.. -Wall -Werror -Wno-unused-function
FIn file included from /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.f4tuQd:1:
/Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/spec/integration/basic.cpp:1:10: fatal error: 'cstdio' file not found
#include <cstdio>
^~~~~~~~
.In file included from /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.uopdlq:1:
/Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/spec/integration/c_only.cpp:1:10: fatal error: 'stdarg.h' file not found
#include <stdarg.h>
^~~~~~~~~~
.In file included from /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.3EH4zC:1:
/Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/spec/integration/c_wrapper.cpp:1:10: fatal error: 'cstdlib' file not found
#include <cstdlib>
^~~~~~~~~
.In file included from /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.T6tOpl:1:
/Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/spec/integration/containers.cpp:1:10: fatal error: 'vector' file not found
#include <vector>
^~~~~~~~
containers.cpp:9:10: error: no viable conversion from returned value of type 'std::vector<int>' to function return type 'int'
return _self_->integers();
^~~~~~~~~~~~~~~~~~
containers.cpp:13:10: error: no viable conversion from returned value of type 'std::vector<std::string>' (aka 'vector<basic_string<char,
char_traits<char>, allocator<char> > >') to function return type 'int'
return _self_->strings();
^~~~~~~~~~~~~~~~~
containers.cpp:17:22: error: no viable conversion from 'int' to 'std::vector<double>'
return _self_->sum(list);
^~~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/vector:530:5: note: candidate
constructor not viable: no known conversion from 'int' to 'initializer_list<value_type>' (aka 'initializer_list<double>') for 1st
argument
vector(initializer_list<value_type> __il);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/vector:542:5: note: candidate
constructor not viable: no known conversion from 'int' to 'const std::__1::vector<double, std::__1::allocator<double> > &' for 1st
argument
vector(const vector& __x);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/vector:548:5: note: candidate
constructor not viable: no known conversion from 'int' to 'std::__1::vector<double, std::__1::allocator<double> > &&' for 1st
argument
vector(vector&& __x)
^
../containers.cpp:14:34: note: passing argument to parameter 'list' here
double sum(std::vector<double> list) {
^
3 errors generated.
Build step failed!
Directory: /Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/spec/integration/tmp
Command: c++ -std=c++11 -c -o containers.o containers.cpp -I.. -Wall -Werror -Wno-unused-function
F...
Failures:
1) the argument translation functionality works
Failure/Error: tool.run!.should eq(0)
Expected: 0
got: 1
# spec/integration/spec_helper.cr:72
2) container instantiation feature works
Failure/Error: tool.run!.should eq(0)
Expected: 0
got: 1
# spec/integration/spec_helper.cr:72
Finished in 16.44 seconds
329 examples, 2 failures, 0 errors, 0 pending
Failed examples:
crystal spec spec/integration/arguments_spec.cr:4 # the argument translation functionality works
crystal spec spec/integration/containers_spec.cr:4 # container instantiation feature works
Pretty good!
/Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/src/bindgen/parser/../../../clang/bindgen /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.tfMeLu -m "" -f "" -- -x c++ -std=c++11 -DSTDC_CONSTANT_MACROS -DSTDC_LIMIT_MACROS
That whole thing there is the invocation command of the clang part. Instead of
/var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.tfMeLu
you can also just give it the path to your v8.h
. Run that command, the error output should be the same what you got before.
If you can get that command to run, you'll get bindgen to work. So that's what you want to focus on. You can check clang/find_clang.cr
on how it dissects your systems clang to get the information it needs. The gist is this:
.cpp
, or use an existing .cpp
file. You just need a file ending in .cpp
.clang -"###" the_file.cpp
. With that invocation, clang will dump the internal process invocations."/usr/bin/clang-6.0" "-cc1" "-triple" ...
. That's what you're interested in. Copy it into an editor for easier reading.clang/bindgen
command from before, and append all those to the invocation. Fire it off.With that we could come up with what to do next.
Ok, adding the resource-dir option fixed it, but the JSON is full of empty objects:
$ ./lib/bindgen/clang/bindgen /usr/local/include/v8.h -m "" -f "" -- -x c++ -std=c++11 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -resource-dir /usr/local/Cellar/llvm/5.0.1/lib/clang/5.0.1
{"enums": {}, "classes": {}, "macros": []}
With that invocation that's an expected output. Bindgen doesn't "grab everything", only what you want it to. Please check the TEMPLATE.yml
for further information.
Oh nice, adding one class produced a big JSON output.
I had to change the runner.cr to pass this argument I need to make it work with the usage from the README. I'm not sure where I should add it in the build process so I don't have to change the bindgen package.
It seems to work, there are a lot of coupling between classes in v8, I'm trying to find a simple class to get a "win" 😄Unfortunately I'm mostly getting errors likes:
Runner command: /Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/src/bindgen/parser/../../../clang/bindgen /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.DRxuMl -c v8::Locker -m "" -f "" -- -x c++ -std=c++11 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -resource-dir /usr/local/Cellar/llvm/5.0.1/lib/clang/5.0.1
Invalid alias name "v8::Locker" at V8::Binding::v8::Locker
Invalid alias name "v8::Isolate" at V8::Binding::v8::Isolate
Argument 1 has unreachable type v8::Isolate at V8::Binding#initialize(isolate)
Result type v8::Locker is unreachable at V8::Binding#initialize(isolate)
Argument 1 has unreachable type v8::Isolate at V8::Binding.IsLocked(isolate)
Argument 1 has unreachable type Binding::v8::Isolate at V8::Locker#initialize(isolate)
Argument 1 has unreachable type Binding::v8::Isolate at V8::Locker.IsLocked(isolate)
Result type Binding::v8::Locker is unreachable at V8::Locker#to_unsafe()
Argument 1 has unreachable type Binding::v8::Locker at V8::Locker#initialize(unwrap)
Found 9 errors. Aborting.
Most types are going to be void pointers I believe. I tried inspiring from the qt5 shard you made, but I don't fully grasp how things are parsed yet.
I added a bunch of ignored types in the v8.yml and brought it down to 2 errors, but it's hard to know what to do here.
Result type Binding::v8::Locker is unreachable at V8::Locker#to_unsafe()
Argument 1 has unreachable type Binding::v8::Locker at V8::Locker#initialize(unwrap)
Found 2 errors. Aborting.
I feel like it might be easiest to write a C bridge! Although Qt must've been a massive effort and you seem to have done it.
I had to change the runner.cr to pass this argument I need to make it work with the usage from the README.
The README is fun and all, but the main documentation is in the TEMPLATE.yml
. Don't try to guess all of the functionality just from qt.yml
. The TEMPLATE.yml
is the definitive authority on what works, and how it works. In your case, that'd be the parser.flags
array. This is a temporary fix though to make it work on MacOS. A proper fix would incorporate this ideally into the clang/find_clang.cr
so it'd just workâ„¢ on MacOS.
I don't know anything about v8 (beyond its existence), so I don't know how it's meant to be used. But I recommend adding the absolutely necessary classes, that you'll actually interact with, first. Ignore all other types, do so explicitly where necessary (You did this already it seems). And then iteratively add classes as you need them, cleaning up ignored types as you go.
Invalid alias name "v8::Locker" at V8::Binding::v8::Locker
Qt doesn't use namespacing, so that case may not be handled nicely yet. That error basically warns that bindgen (with that configuration) was about to generate a Crystal line like alias v8::Locker = Void
. That wouldn't compile anyway. You can 1) just ignore the type explicitly, or 2) add the class to the classes
list, or 3) rename the type by setting the binding_type
and crystal_type
(I think it was) fields in the type configuration to provide a sane name.
The CrystalBinding processor is responsible for generating these aliases in any case.
Argument 1 has unreachable type Binding::v8::Isolate at V8::Locker#initialize(isolate)
Is an artifact of the error above, but do have have explicitly ignored the locker type, while having it in the classes list? If so, then yeah it's unreachable, as it doesn't exist on the binding level. Locker by name, and the API snippets from above, make it seem like it's a RAII locker class? That'll be fun, I'd ignore it for now. RAII doesn't translate well to Crystal. You probably want to half-manually bind it in the long run to provide a Crystal native API on top of it instead of passing it through directly. I do this in a few places in Qt as well.
I feel like it might be easiest to write a C bridge!
Yeah, that's what bindgen does for you. Take a look in the generated .cpp file (Just remove the sanity_check
processor for a moment). That's all the code that basically you would have to write. Add the Crystal API, too. If you don't fancy that, then you now understand why I built bindgen :P
However, if there is a good C bridge, you could bind to that instead. Bindgen also has facilities to deal with C libraries. Consult the TEMPLATE.yml
on that.
Thanks, that helps!
Looks like it's working well. The TEMPLATE has great comments, but I didn't know what I didn't know before, meaning I didn't know what to make of a lot of the comments in there. Now they make more sense.
While I've got you here, I'm not sure how to add a container like v8::Local<T>
. Here's the class definition for reference: https://v8docs.nodesource.com/node-9.3/de/deb/classv8_1_1_local.html
I've added:
containers:
- class: v8::Local
type: Sequential
classes:
# ...
v8::Local: Local
types:
# ...
v8::Local:
crystal_type: Local
binding_type: Local
This template can contain a lot of different types, most commonly it will be a v8::Value
.
I got the error:
Runner command: /Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/src/bindgen/parser/../../../clang/bindgen /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.Ww3X99 -c v8::Isolate::CreateParams -c v8::HeapStatistics -c v8::Local -e v8::MemoryPressureLevel -m "" -f "" -- -x c++ -std=c++11 -resource-dir /usr/local/Cellar/llvm/5.0.1/lib/clang/5.0.1 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS
clang/bindgen failed to execute. (Exception)
from lib/bindgen/src/bindgen/parser/runner.cr:38:11 in 'run'
from lib/bindgen/src/bindgen/parser/runner.cr:45:9 in 'run_and_parse'
from lib/bindgen/src/bindgen/tool.cr:130:7 in 'parse_cpp_sources'
from lib/bindgen/src/bindgen/tool.cr:48:7 in 'run_steps'
from lib/bindgen/src/bindgen/tool.cr:40:15 in 'run!'
from lib/bindgen/src/bindgen.cr:58:1 in '__crystal_main'
from /usr/local/Cellar/crystal-lang/0.24.1_2/src/crystal/main.cr:11:3 in '_crystal_main'
from /usr/local/Cellar/crystal-lang/0.24.1_2/src/crystal/main.cr:112:5 in 'main_user_code'
from /usr/local/Cellar/crystal-lang/0.24.1_2/src/crystal/main.cr:101:7 in 'main'
from /usr/local/Cellar/crystal-lang/0.24.1_2/src/crystal/main.cr:135:3 in 'main'
Is there a way to get more info from the error?
The template instantiation support is quite in its beginnings, as I simply didn't need it much with Qt (Qt was the driving factor behind bindgen after all). A sequential
container instantiation expects the C++ class to respond like a sequential STL container. The Local<T>
class there looks like a smart pointer? That won't cut it :|
While that feature would be neat, I don't have the time right now to work on it.
What you can do however is providing custom conversion code to go from a Local<Value>
to a Value
(Or whatever makes sense). Qt uses this to bind QString
in C++-land to String
in Crystal-land. Maybe that's a workable solution?
Oh and when you add Foo: Bar
to classes
, it'll automatically set the crystal_type
and binding_type
for you. So in your paste above, you can omit the whole v8::Local
block.
In general, that Value
class sounds like it wants to have a proper Crystal API (To e.g. marshal Crystal values easily). You can use QString
as starting point. If it is a container for an (basically) arbitrary value like QVariant
, then you could look there too.
Basically a fully custom conversion goes like this:
struct
into the generated lib Binding
block by simply re-opening it in a non-generated file.W.r.t. 5), Set the cpp_type
to the name of your conversion structure, binding_type
to Binding::YourConversionStructure
, and crystal_type
to the user-facing Crystal class (or struct). Add functionality to that latter class/struct as you like. Then use the from_cpp
, to_cpp
and the from_crystal
, to_crystal
type configuration to register your conversion functions. The QString
type in Qt may be helpful for this as real-world example.
That's so complex that I don't think a wrapper generator could auto-generate a good API that people would like. Bindgen can help you get the tedious bits done, though for such complex cases, you'll end up writing a few lines.
Yes, I think a converter would do the trick.
I think there are issues with the v8 namespacing. If I don't manually write those types
in addition to the classes
, it doesn't seem to be able to reach them.
I wasn't expecting perfect bindings generation, this shard already surpasses my wildest dreams ;) I have a feeling it's generating the right structs and functions. We'll see once I can actually try it out (I can't until I have v8::Platform done at least, which is fairly contained, so it should be quick enough.)
Any tips for std::unique_ptr<T>
? v8 seems to be using those quite a bit.
Any tips for std::unique_ptr
? v8 seems to be using those quite a bit.
Oh shit. Well, that's going to be nasty. First there's the issue with there not being proper template instantiation support that's going to bite you here. (Although I think for this having a "smart pointer" specialization as there is for sequential containers would be neat, as that idiom is all over the place in C++).
That aside, the bigger issue is mixing the ownership model of C++ (Well, std::unique_ptr
in this case) with Crystals (Pure garbage collection). Going from C++ to Crystal, I think it'd be reasonable to transfer ownership (as std::unique_ptr
suggests) to Crystals GC. But what about the other way around? There's really no notion of "ownership" in Crystal, as there is no need for it (Memory management-wise).
You'd have to rip out the object pointer from the Crystal container class, and then in the #finalize
(And actually, in every other method) check if the @ptr
is null or not. That's not only super nasty, prone to error, but also (and IMHO the biggest issue) surprising to the end-user.
We can fix/add stuff to deal with smart pointers, maybe even special rules for shared/unique smart pointers. But we can't fix the user. I'm honestly not sure how to properly bridge this.
Do you know how other bindings (maybe for other GC'd languages) deal with this?
There are basic golang bindings here (they work well): https://github.com/augustoroman/v8
As far as I understand, he wraps those pointers in its own pointers and then passes those back and forth. Go is GCed, but it doesn't do bindings the same way that Crystal does.
That's the C bridge I might decide to use. Except I may just use that package and golang instead of writing bindings at all!
Would using a C bridge similar to https://github.com/augustoroman/v8 help? I expect I'd run into the same problems with pointers getting freed.
However, it probably would be a simpler bridge given the nicer bindings API in Crystal.
As long no one feels like implementing smart pointer support, using a wrapper that "hides" them will help. Bindgen has facilities to wrap C libraries back into classes.
Note: I've successfully got bindgen to work under macos. I had to modify the scripts a bit, but I was able to compile clang and everything. I just couldn't get
-ltinfo
, I just removed it for now.I don't think I'll succeed, but I'm trying to generate bindings for libv8.
One of the first things it includes is
<memory>
. Which is part of the stdlib. My .yml config looks like this:I get the following error:
How can I make this work? I tried adding
memory.h
to my files and it didn't work.