libffi support

[GoogleCodeExporter]

I've been working on a FPC libffi unit, and some of the basic cases seem to 
work just fine. (See below)

My question is, how exactly do you want this integrated in lape?
Do you want to generate CIF's automatically when someone uses a specific 
keyword? The same goes for function wrappers, do you want to generate those 
automatically for lape functions? (You can even make a 
call-this-function-with-any-arguments-wrap-all function with libffi, or so I 
have read)

----------------------------------------
var
  Cif: TFFICif;
  s: TFFIStatus;
  ret: ptruint;
  int_type: TFFIType;     

begin
  writeln(fpgetpid());

  s := ffi_prep_cif(@cif, FFI_DEFAULT_ABI, 0, @ffi_type_uint64, nil);
  writeln('S:' + inttostr(PtrUInt(s)));

  if s = FFI_OK then
  begin
    writeln('Woo');
    ffi_call(@cif, @fpgetpid, @ret, nil);
    writeln('Result: ' + inttostr(ret));
  end;
end;

Results in:
$ ./project1
5853
S:0
Woo
Result: 5853

Original issue reported on code.google.com by merlijnw...@gmail.com on 26 Apr 2012 at 3:08

[GoogleCodeExporter]

I don't think I can change the type of the issue; please change to something 
appropriate.

Original comment by merlijnw...@gmail.com on 26 Apr 2012 at 3:09

[GoogleCodeExporter]

Looks promising!
I'm not sure how I want to integrate this, but I don't want Lape to rely on it. 
It should be an add-on which gives access to native functions.

One to way, to stay compliant with Pascal (Script), is to use the external 
keyword for importing. I'm not so sure about wrappers for Lape functions; 
perhaps that would be better off using an explicit function like getNative(), 
getExternal(), funcAddr() or something along those lines. Did you have anything 
in mind?

Anyway, a libffi FPC unit is awesome and even without syntactic sugar it would 
be a great addition to Lape/FPC. Much appreciated!

Original comment by niels....@gmail.com on 26 Apr 2012 at 5:29

• Changed state: Accepted
• Added labels: Priority-Low, Type-Enhancement
• Removed labels: Priority-Medium, Type-Defect

[GoogleCodeExporter]

You can find the ffi.pas unit here: 
https://github.com/MerlijnWajer/Simba/blob/master/Projects/ffi/ffi.pas
It's not yet complete, but I plan to add most of the other functionality later.

As for suggestions, I made this for two main reasons:
- Calling plugins / external functions.
- Callback support. (Python does this a lot in ctypes too and it is pretty 
convenient)

As for external functions; we can go two (maybe both) ways:

- external keyword; part of the language:

function puts(s: PChar); external 'libc.so';

We'd also need different calling conventions (and they should be integrated as 
well), since not every function is using cdecl or stdcall.

- export the libffi api to lape scripts so they can generate their own CIF's 
and call them.

This is simply a ctypes-api like many existing languages already have.

As for callback support:

- We could add something that is part of the syntax, some keyword to 
automatically generate a wrapper for the function we want to use as a callback. 
I'm not sure how easy this would be technically - but you could simply pass the 
pointer of the function if the function being called is an external one, and 
otherwise just pass the internal lape function.

procedure hello(sender: tobject); callback;
begin
  writeln('Hello');
end;

someform.onsomeevent := @hello;

- Add a procedure which creates a (new, libffi) function from an existing lape 
function:

procedure hello(sender: tobject);
begin
  writeln('Hello');
end;

var hell_callback: TSomething;
hell_callback := nativeify(@hello);
someform.onsomeevent := @hell_callback;

Original comment by merlijnw...@gmail.com on 26 Apr 2012 at 8:47

[GoogleCodeExporter]

I've been looking into this a bit and (since the general feel seems to be avoid 
modifying lape and adding external dependencies) I believe it's entirely 
possible to implement this with the API lape already has. 

With the Merlijn's `nativeify` that english speakers would more naturally call 
`natify`, one could implement a function in native that used lape api to invoke 
a procedure by name (PROVIDED, of course, that this api exists***) where the 
only complication might come from determining the argument list, but lape 
should*** have some functionality implemented for this, if it doesn't already. 
Then the matter simplifies to importing the `natify` function into lape 
(functionality which exists). `natify` would take the function name as a string 
(or PERHAPS lape could automatically convert a function pointer to a string 
name on native call, or at least have api to turn an "internal function 
pointer" into a string on native call) and return an actual native pointer by 
way of making an ffi closure, using lape API  to get the argument list, around 
another native method that uses lape API invoke the method by name (ffi has api 
specifically for this, so only one static native method would be needed).

As for calling plugin functions (i.e. functions that are from plugins into 
pascalscript in Simba currently), my suggestion would be use the lape api for 
this, but for the implementing application (i.e. simba) make a closure around a 
method using lape api to invoke the actual native method in a sane way 
(basically auto generating wrappers).

Calling external functions with the external keyword from non-plugin libraries 
will be very difficult without incorporating ffi into lape's core, but that's 
not a significant issue, since the first two will solve the immediate problems. 

Original comment by benland...@gmail.com on 27 Apr 2012 at 3:58

[GoogleCodeExporter]

ffi.pas unit 
(https://github.com/MerlijnWajer/Simba/blob/master/Projects/ffi/ffi.pas) now 
supports closure API. At least the simple cases work, so I assume all cases 
work. (Example below)
I'd love some feedback on platforms other than x64 GNU/Linux. I've tried to 
port the TFFIClosure struct as close as possible; but the FFI_TRAMPOLINE_SIZE 
needs to be tried on other platforms. It won't work at all when 
FFI_EXEC_TRAMPOLINE_TABLE is used; but luckily that was not the case on my 
platform. I guess this can turn out to be quite the nuisance to add support for 
all platforms. (For a C project this is simple, they just import the ffi.h file 
and they everything they need at disposal. (Mainly, a structure with the 
appropriate size...)

Feedback very welcome. I believe some of the arguments could be made to look a 
bit prettier.

Note the difference in the bound_puts variable, the first time we pass the 
reference of the reference to it (because we're going to write it), the second 
time just the function reference.

----------------------------------------
procedure puts(var cif: TFFICif; var ret: cuint; args: TPointerArray; userdata: 
Pointer) cdecl;
var
  s: PChar;
begin
  s := PChar(args[0]);
  writeln(s);
  ret := strlen(s);
end;

procedure testclosure;
var
  cif: TFFICif;
  s: TFFIStatus;
  args: PPFFIType;
  closure: PFFIClosure;
  bound_puts: function (s: pchar): PtrUInt;
  rc: integer;
begin
  writeln('Size:', sizeof(bound_puts));
  closure := ffi_closure_alloc(sizeof(TFFIClosure), @bound_puts);

  if assigned(closure) then
  begin
    args := GetMem(sizeof(PFFIType) * 1);
    args[0] := @ffi_type_pointer;
    s := ffi_prep_cif(cif, FFI_DEFAULT_ABI, 1, @ffi_type_uint32, args);
    if s = FFI_OK then
    begin
      s := ffi_prep_closure_loc(closure, &cif, @puts, nil, bound_puts);
      if s = FFI_OK then
      begin
        rc := bound_puts('Hello World!');
        writeln('Return value:', rc);
      end else writeln('ffi_prep_closure_loc failed', s);
    end else writeln('ffi_prep_cif failed', s);
  end else writeln('Closure not assigned');

  ffi_closure_free(closure);
end;  

Original comment by merlijnw...@gmail.com on 27 Apr 2012 at 10:19

[GoogleCodeExporter]

I'm 90% through writing the libffi front end for lape in simba that will allow 
it to invoke regular native methods from plugins (no modifications to plugins 
necessary), and I was basically wondering if lape had parsing capabilities 
exported and if so how would I use them. Basically the end goal I want is to 
know at least the size of the arguments passed, but preferable would be getting 
the exact mappings from the pascal types to the ffi types (see Merlijn's 
ffi.pas). (I'm not up to date on my calling conventions, but sometimes more 
than size makes a difference on how the arguments are passed). If lape could 
read a function header and spit out the argument types in some sane way that 
would be excellent. I'm sure it exists, I just don't know where to look for 
such functionality (the source tree is quite large lol), and I'm not sure it's 
exported for external use.

See for progress: 
https://github.com/BenLand100/Simba/commit/fb624cba60fdb7449450c7135048f63686ea6
fe0

Original comment by benland...@gmail.com on 28 Apr 2012 at 2:53

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Looking good!

I started a ffi unit for Lape (lpffi.pas), which should be of help. I added a 
LapeHeaderToFFICif function for you which converts a string header to a FFICif 
struct. The resulting structure is wrapped in a class to handle the memory 
issues, but the raw Cif object is still available.

An example comparable to Merlijn's code looks like this:

Cif := LapeHeaderToFFICif(Compiler, 'function(var Arg1, Arg2: record x, y, z: 
Int64; end): Int32', FFI_STDCALL);
try
  Cif.Call(@testInc, @Res, [@Arg1, @Arg2]);
  WriteLn('Result: ', Res);
finally
  Cif.Free();
end; 

Original comment by niels....@gmail.com on 28 Apr 2012 at 5:31

[GoogleCodeExporter]

Fantastic, that worked like a charm. See 
https://github.com/BenLand100/Simba/commit/c1c3aab20ee346e4779f963a9afb89d05a00d
e53 for implementation details, and some concerns. 

In short, the proliferation of calling conventions on x86 is going to screw us 
over, but everything works fine on x64. See the comment on that commit for my 
suggestions.

Original comment by benland...@gmail.com on 28 Apr 2012 at 9:09

[GoogleCodeExporter]

Alright, looking good.

I've created a wrapping class for FFI closures and added functions for the 
creation of closures for Native->Lape and Lape->Native. Simple examples in the 
following gist: https://gist.github.com/2560391

You're right about the calling convention. Libffi supports fastcall since the 
last version, but this is not compatible with the FPC register calling 
convention (plus it's not available on Linux). Cdecl is a logical solution.

FPC makes the conversion to Cdecl pretty easy with the {$CALLING} directive, 
but Delphi doesn't seem to have such a convenient feature. I will look at this 
later.

Original comment by niels....@gmail.com on 30 Apr 2012 at 5:57