[RFE] Installation of libguile-gi header files

[Bob131]

PyGObject, for instance, installs a pkg-config file and headers, exposing the type conversion glue in a way that's usable for other applications. This is quite handy in the case of embedding the Python interpreter in programs that make heavy use of GObject and (for one reason or another) can't use GI directly.

Looking forward to a v0.1 ;)

[LordYuuma]

I'm not really sure about the use case here. Most of our conversion functions require a GISomethingInfo somewhere along the line, the sole exception being types, which only need the GType. However, a type without any associated functionality is next to meaningless, unless you plan to define the type itself in C and then only use inherited functionality in Scheme, which should work on its own, but may fail in your specific case because your type is missing in gig_type_transfer_object.

This issue can be addressed in (at least) two ways without needing to expose internals that may be subject to change. The first would be to handle the failure in gig_type_transfer_object more robustly, perhaps defining the object type as needed or downcasting it. The second way would be to write an "API" header, which merely defines your types and create a typelib from that. Perhaps install it in a special directory and use the GI functions to add that to the search path in your program only.

I'm currently working on #29 and I think, I might have to make gig_type_transfer_object more robust when people can partly load a typelib. (E.g. they import only <GtkWindow>, but they create a <GtkApplicationWindow> somehow). I'm not sure, whether that will actually happen, though.

Can you give us more specific information about your program and why exactly you can't use GI with it, but do want to expose it to Guile using libguile-gi?

[spk121]

As I understand it, PyGObject has made the choice to write their own code to call C functions directly using their foreign function interface. So its glue code could, I suppose, be useful to include in other projects if you are looking for a more complete foreign function interface implementation.

But, so far, guile-gi is using function calling mechanism provided by the GObject Introspection library, and not Guile's FFI. So to even call a C function with guile-gi, you already need to have GObject Introspection libraries and *.gir files. The glue we have isn't very useful outside of our context.

[LordYuuma]

I don't think, that makes much of a difference. Whether we need the function info to call a function or simply to define a function, that may be called later, is largely irrelevant, because the calling code is never public to begin with. However, the defining function could be, regardless of what we do internally. Here, the question we need to ask, is "is it useful?". Since we need GI to define the function (whether we do so with the FFI or not is irrelevant – we could even write a "compiler", which compiles typelibs to scheme code with (system foreign), but the interface wouldn't change), it does not make sense to expose this function for the sake of usage without GI. It would for instance make sense, if one wanted to preload certain libraries, e.g. write an application that automatically has GLib and Gtk bound to glib:: and gtk:: respectively. However, it is not really clear, whether that (i.e. the preloading) makes sense.

Going back to types, I think the problem arises when mixing basic FFI with GI. Assume for instance, that you SCM_DEFINE a function that takes some arguments, some of which may actually be wrapped GObjects. Knowing that, you want to extract the object and do some stuff with it. I really don't think, that's good design, but whatever floats your boat, I guess? Not really sure, whether we want to support that, though.

[spk121]

The question of whether it is useful was the question I was trying to address. The compiled library of PyGObject, outside of of its usage with gobject-introspection, does provide some useful standalone functionality, particularly by trying to make its Python-to-GObject type conversion routines useful standalone as well as some FFI functionality that is useful standalone. libguile-gi.so without any of the (gi) scheme files, outside of its intended usage with typelibs, provides very little, and its SCM-to-GObject type conversion routines, at the moment, don't have an architecture to be useful standalone and it provides no FFI functionality that is useful standalone or that is better than Guile provides.

[LordYuuma]

I think you are conflating "useful" and "useful on its own". No one will ever use PyGObject without Python or GObject. In a similar manner, no one will use this library without Scheme or GI (or if we weaken the requirements Scheme or GObject), but there may still be functionality in here, that might be useful to applications that make use of both. The question is, which functionality is so useful, that we should export it. Currently, neither of us see much use beyond the Scheme API. However, @Bob131 may have opened this issue thinking of particular functionality that he wished was available for him to use in C code, that maybe only requires minimal use of GI. In fact, they specifically asked for the "type conversion glue", meaning some of the gig_type_* functions.

I have not yet written any C application, that mixes GObject and Guile using Guile-GI. I have however already written a Vala application, that boots Guile and runs a GApplication within it using some quickly hacked together vapis and other stuff that I am not proud of. The reason I am working on this project is so that I can one day implement the same thing in a cleaner way and perhaps in Scheme. However, even if I do so, applications not written in Scheme don't cease to exist, but some of them might want to use Scheme, maybe even with this library as a base.

It should be noted, that for extending GApplications, libpeas should be preferred over any ad-hoc solution. However, Peas has no loader for Guile and will not have one until one of the many libraries, that aims to make GObject usable in Guile, is at least to some extent completed and applicable in their context. Guile-GI is the most promising one I currently know of, if I do say so myself, and at some point I think we should aim for Peas.

[Bob131]

(Sorry if this is a little verbose; there are a couple of interacting factors in my use-case that make typelib generation painful, so I'm just going to dump a bunch of specifics in your lap)

I have a Processing-inspired framework for doing visualisations written in Vala. The concept is that you have a single Vala file implementing the simulation/visualisation/etc and a compiler driver handles compilation and runs the result. It's quite convenient to be able to make adjustments to program behaviour with some sort of live-scripting, so the supporting library features some wrapper types for embedding and interacting with a Python interpreter.

A couple of avenues are available for generating GIRs:

• Writing a header/GIR file explicitly: Appealingly straightforward, but unfortunately this conflicts with the convenience-oriented nature of the use case. The main idea is to be able to provide fast prototypes of single-use code without having to manage multiple source files or think about APIs.
• Use libvala to write an analysis pass for type extraction: Using such an approach precludes the exporting of lambdas since their symbol identifiers are computed when lowering the AST to C, making it information available only within the compiler and unavailable to users of the parsing/AST library.
• Add a pass to the Vala compiler: Probably the Right Way(tm) to do it, but valac doesn't support plugins so it would require a modification of the compiler itself.

However, even if there was a nice way to get GI to work as-intended, there's still one element of my use case still wanting: the current set up allows evaluation of arbitrary expressions, like an Expr object representing some Python code that given some parameters returns a result. In my case it's implemented using Vala generics (syntactic sugar for GValues), the heavy lifting for which is handled by PyGObject.

The only real sore-point I've noticed from lack of typelibs (in terms of user experience, in reality the PyGObject support for this use case is limited and so the code is pretty ugly) is the lack of support for native closures of arbitrary arity, but unary functions get the job done and everything else is probably preferable in its current state.

I understand this use case is perhaps ultra-specific and unusual, but I promise it does exist! :)

[LordYuuma]

Long story short: You have an application, that does stuff and extend it with an embedded Python interpreter using PyGObject. You wish to also run Scheme code in it using Guile-GI.

I think your architecture might benefit from a split and clean interfaces. Put the code that the Python/Scheme side needs into a library and use the Vala compiler to generate a typelib. If there are lambdas, that you really, really need in there, define them as functions instead. Otherwise, Vala lambdas can only be used in places where you'd have callbacks, which should map to Python function objects/Guile procedures when passing through the respective language boundaries (not quite sure about the latter, needs confirmation). Put the rest (setup code for your application and the interpreters) in the application and link it against that library. I'm pretty sure, that your interpreter only needs some basic functionality to interact with the application and none of the "single-use" code.

As far as live-scripting is concerned, I think there may be some features missing in Guile-GI, that you do have in PyGObject. (If you do find someting missing, don't hesitate to report it.) However, as long as you are simply extending rather than entirely rewriting, Guile-GI should hopefully be fine.

[Bob131]

I'm pretty sure, that your interpreter only needs some basic functionality to interact with the application and none of the "single-use" code.

Unfortunately this is incorrect. The supporting libraries themselves already have typelibs, that's no problem; my use of PyGObject's C API is specifically for the purpose of functions/callbacks/native closures defined in the single-use code, exposed for the use of live-coded script doodads. I should clarify that the callbacks themselves aren't exported via FFI GI but rather embedded in a C-defined callable Python object which handles marshalling/unmarshalling of parameters and return values. PyGObject comes into play helping to marshal individual values between GValue and PyObject and for converting Python errors to GError.

[LordYuuma]

Sounds hacky – and I'm pretty sure that's not just a single use. Could you show me some of that code?

[Bob131]

Sorry, I should clarify again: The mechanism for exporting these functions (the C-defined callable PyObject) is part of the supporting library. It facilitates the export of functions from the single-use code to the interpreter.

That said, it most certainly is hacky (the source file from the library is called python-hack.c ;). Is that the bit you wanted? I can upload it as an attachment, but I should warn you that there is an embarrassing lack of care put into writing such an eyesore.

[LordYuuma]

So, in principle you have something like

// support.vala
void export(PythonCallable callable, ...);

// something.vala
export((obj) => { foo(obj); bar(obj); baz(obj); }, ...);

with support.vala being introspected and something.vala not being introspected? Do you by any chance also generate lambdas on the fly?

[Bob131]

Yep, that's pretty close. As for 'generate lambdas on the fly', do you mean things like Python code passing callables back into native code? I don't do that just because I haven't found the need, but thinking about it I can't imagine how I'd make that workable without GI anyway.

If you were just after what the Vala interface looks like, it's something like:

// python.vala
public class Expr : PythonObject {
  public string code {set; get;}
  public T eval<T> (Namespace locals);
}
public class Namespace : PythonObject {
  public void add (PythonObject object);
}
public class NativeClosure<T,U> : PythonObject {
  public delegate T Func<T,U> (U parameter);
  public NativeClosure.named (string name, Func<T,U> func);
}

// something.vala
var locals = new Namespace ();
locals.add (new NativeClosure<int,int>.named ("sq", (a) => a**2));
var expr = new Expr ();
expr.code = "sq(5) + sq(7)";
var result = expr.eval<int> (locals);

[LordYuuma]

No, I meant like generating new Namespaces and NativeClosures in Vala as the situation demands. Are your namespaces (in Vala code) statically known or dynamically introduced?

[Bob131]

The latter, dynamically introduced.

[LordYuuma]

Okay, I think I figured out, how you can do this in Guile with Guile-GI. (You should also be able to do this with Python code).

// vala
public class Environment
{
    public int n_functions;
    public string[] function_names { get; set; }
    public Callback[] functions { get; set; }
}

public class Evaluation<T>
{
    public Environment env {get; set;}
    public string code {get; set;}
    public T return_value {get; set;}
}

public void
push_evaluation(Evaluation e);

public void
pop_evaluation(out Evaluation e);

public T eval<T>(Environment env, string code)
{
    Evaluation e = new Evaluation();
    e.env = env;
    e.code = code;
    push_evaluation(e);
    scm_call_0(eval_request);
    return e.return_value;
}

// scheme
(define (eval-request)
 (let ((e (pop-evaluation)))
  (for-each (lambda (n f) (module-define! (current-module) n f))
   (function-names (env e)) (functions (env e)))
  (set! (return-value e) (eval code))))

Now, obviously, there are some caveats here. The biggest problem is, that you can't do this with threaded code. You need mutexes and condition variables to work around that restriction. (Should not really be an issue with Python due to the GIL, but in Guile you can make things more efficient with MT.) Secondly, I'm not doing this in a clean scheme environment, but you can and you should – it's just a bit of extra typing. And last, but not least, you may also want other stuff than callbacks in your environments, but this is somewhat simplified.

Now, there is one even bigger problem, if you want to introduce callbacks to Guile in any way, and that is, that we probably don't convert them to scheme functions. However, as I said, you should be able to do this in Python, where callback conversion should work. So try to rewrite your Python code using this thunk trampoline and tell me how it went.

[Bob131]

I'm having a bit of trouble figuring out what's going on. Is the idea that Evaluation is exported via GI, and assignment to a GValue field from within the interpreter coaxes PyGObject/guile-gi/etc to automatically box it?

[LordYuuma]

At least guile-gi implements property setting via GValues. However, there is a little problem, in that templates are actually not implemented as GValues in Vala, so you would have to rewrite your code to actually use GValues. Also, you would have to explicitly set the value, the way I've written it now, it would try to construct a value to store in the GValue, which is an unnecessary level of nesting (and also not supported, we only have actual value types like INT). However, we could easily define GValues as functions with setters. Then you only need an additional level of brackets around (return-value e), i.e. (set! ((return-value e)) (eval code)).

On the Python side, there should be a function to set a GValue from a Python object (hopefully). Use that instead of the set! here.

The basic idea is the following: You want to marshall some arguments and returns, but you can't, because we're not exporting that functionality. So instead, you define a void function without arguments and pass the actual arguments through a "side channel", which for our purposes could be a GLib.Queue<Evaluation>. You push a new Evaluation to that queue and make an evaluation request. That request pops the first item from the queue and evaluates it, storing the result in the return value. (In an MT setting, you also want a CV to wait on and have the request signal it as soon as evaluation is finished.) Such code also works very nice with worker threads and GLib.ThreadPool, which is where one would typically encounter this pattern.

[Bob131]

So I've just dug through PyGObject again, and it doesn't seem like it supports GValue-typed properties (nor G_TYPE_POINTER, which is presumably what a generic-typed property would be). Something that seems to work is passing Python code a GValue initialised to typeof (T) and using the set_value method supplied by PyGI. Only thing is, there are no GTypes corresponding to e.g. tuples, so the conversion routine just bails out with an error. Using the C API means support can be hacked in for things like tuples whilst letting the library handle the majority of cases.

[LordYuuma]

Or you can just convert tuples to arrays in Python code. Any extraneous python types can be converted to the expected one in the "eval-request" function, which is implemented in Python/Scheme/whatever. Having to initialize the GValue gives you some type safety, which is nice, even in the example you've given. You don't want "2+2" to return a string, for instance.

Alternatively to that, you could use callbacks, aka. delegates. The delegate would look roughly like delegate void Evaluator(Namespace space, string code, GValue *value) and you wouldn't have to go through the trouble of pushing and popping environments. Guile-GI supports procedure to callback conversions and PyGObject should too.

[spk121]

As an aside... In PyGObject, as I recall, there are no GValue-based properties for GObject classes defined in Python. Instead, there was a single property that was a Python hash table that stored properties for an object, and there were getter/setter functions that would handle finding the right properties, be they C or Python or stored in the current or parent object. You can see remnants of that approach in very early commits of guile-gi, as well. (It was the INST_DICT in the old GObject implementation)

[LordYuuma]

After some working on dev-value, I think there might be some uses for a public API. Basically, what I've been doing there was extending the type system with some useful stuff – for instance, <GValue> can be used as an applicable struct with setter.

Looking at @Bob131's use case, we could construct something quite similar but with different code. Basically, it would be something like this:

// something.vala
public class Closure
{
    public GLib.Closure real_closure {get; construct;}
    public Type[] arg_types {get; construct;}
    public Type[] return_type {get; construct;}

    public void apply(out Value retval, Value[] args) { ... }
}
// something.c
gig_register_extended(SOMETHING_TYPE_CLOSURE, scm_list_2(gig_x_type, scm_procedure_type), ...);
// something.vala
(define-method (initialize (closure <SomethingClosure>) initargs)
  (next-method)
  (slot-set! closure 'procedure (lambda args #| pack to values, invoke closure, unpack |#))

As far as I'm aware this is not possible with GClosures and good luck trying to infer anything from an arbitrary GCallback. You would still have to work with GIRepository most of the time, because that's how this library rolls, but you could do without if you're just using types, and you could control aspects of the bindings even if you're using GIRepository for everything else. Right now, you could also redefine the classes themselves, but GOOPS will soon no longer support this for "basic" classes and I'm honestly looking forward to that, because it makes our type infrastructure much more stable.

I'm still not sure about which features to make public, but having at least something would be an improvement. However, given the internal work that's ahead of us for even basic type support (#25, #31), this is something for a later release.