Wrong method signature generated

andre2007 commented 5 years ago

For the project https://github.com/apache/arrow/tree/master/c_glib (Release 0.15.0 zip file) I generated the gir files Arrow-1.0.gir and Parquet-1.0.gir.

APILookup.txt

outputRoot: generated

lookup: APILookupArrow.txt lookup: APILookupArrowParquet.txt

APILookupArrow.txt

wrap: arrow file: Arrow-1.0.gir

APILookupArrowParquet.txt

wrap: parquet file: Parquet-1.0.gir

A D source file arrow/Array.d is genererated with a method signature like this:

        /**
     *
     * Params:
     *     targetDataType = A #GArrowDataType of cast target data.
     *     options = A #GArrowCastOptions.
     * Returns: A newly created casted array on success, %NULL on error.
     *
     * Since: 0.7.0
     *
     * Throws: GException on failure.
     */
    public Array cast(DataType targetDataType, CastOptions options)
    {
        GError* err = null;

This causes syntax errors

Gir files: gir.zip

MikeWey commented 5 years ago

I've added 'cast' to the list of keywords, that should fix this issue.

andre2007 commented 5 years ago

@MikeWey thanks a lot. I have some more findings:

Also 'new' needs to be added to the list of keywords.
in dub.json of glibd (https://github.com/gtkd-developers/GlibD/blob/master/dub.json) an import path for 'generated' is missing: "importPaths": ["src/","generated"] otherwise you get this error message source/arrow/c/types.d(3,15): Error: module `types` is in file 'gio/c/types.d' which cannot be read

If have some more errors but I need to analyze them in detail:

source/arrow/FileT.d(53,14): Error: cannot implicitly override base class method gio.InputStream.InputStream.isClosed with arrow.InputStream.InputStream.FileT!(GArrowInputStream).isClosed; add override attribute

source/arrow/InputStream.d(46,2): Error: mixin arrow.InputStream.InputStream.FileT!(GArrowInputStream) error instantiating

source/arrow/SeekableInputStream.d(28,27): Error: function arrow.SeekableInputStream.SeekableInputStream.getStruct does not override any function

source/arrow/BufferInputStream.d(26,27): Error: function arrow.BufferInputStream.BufferInputStream.getStruct does not override any function

source/arrow/CompressedInputStream.d(28,27): Error: function arrow.CompressedInputStream.CompressedInputStream.getStruct does not override any function

source/arrow/GIOInputStream.d(28,27): Error: function arrow.GIOInputStream.GIOInputStream.getStruct does not override any function

source/arrow/GIOOutputStream.d(14,8): Error: arrow.OutputStream.OutputStream at source/arrow/OutputStream.d(16,8) conflicts with gio.OutputStream.OutputStream at /home/user/.dub/packages/glibd-2.1.0/glibd/generated/gio/OutputStream.d(52,8)

source/arrow/GIOOutputStream.d(57,9): Error: arrow.OutputStream.OutputStream at source/arrow/OutputStream.d(16,8) conflicts with gio.OutputStream.OutputStream at /home/user/.dub/packages/glibd-2.1.0/glibd/generated/gio/OutputStream.d(52,8)

source/arrow/GIOOutputStream.d(78,22): Error: arrow.OutputStream.OutputStream at source/arrow/OutputStream.d(16,8) conflicts with gio.OutputStream.OutputStream at /home/user/.dub/packages/glibd-2.1.0/glibd/generated/gio/OutputStream.d(52,8)

source/arrow/ListArray.d(100,27): Error: function DataType arrow.ListArray.ListArray.getValueType() does not override any function, did you mean to override GArrowType arrow.Array.Array.getValueType()?

source/arrow/MemoryMappedInputStream.d(30,27): Error: function arrow.MemoryMappedInputStream.MemoryMappedInputStream.getStruct does not override any function

andre2007 commented 5 years ago

Thanks a lot . I was able to solve all issues manually. At 1 place override was missing and at the other place it was added but not expected.

The other issues was that there is s symbol OutputStream in gio and in arrow. My gut feeling is, girtod got confused here but I am not 100% sure.

Now I get a lot linker errors although the dev libraries for Arrow and Parquet are installed. I am not sure what I have to do now, pragma(lib, ...) did not help. Do I have to create manually a file like https://github.com/gtkd-developers/GlibD/blob/master/src/gtkd/Loader.d?

MikeWey commented 5 years ago

What linker errors are you getting, the generated binding should be using the loader from glibd.

andre2007 commented 5 years ago

For every single Arrow and Parquet function from the c/functions.d modules there is an undefined reference error during dub build on ubuntu 18.04. I will check the glibd loader.

MikeWey commented 5 years ago

You will need to link to the Arrow and Parquet libraries in the dub.json or pass --use-runtime-linker to girtod when generating.

andre2007 commented 5 years ago

Thanks a lot, that solved the issue.

jcrapuchettes commented 4 years ago

@andre2007 is there any way I could get access to your APILookup files? I'm trying to use parquet in D too and am running into some problems. Specifically the OutputStream issue.

andre2007 commented 4 years ago

@jcrapuchettes I cannot use the tool gir-to-d due to its license and therefore switched to DPP to generate header files. Works like a charm and reading parquet files is working fine.

FROM dlang2/ldc-ubuntu:1.21.0 as ldc

COPY install.sh /
RUN /install.sh

RUN apt-get install -y clang-9 libclang-9-dev
RUN ln -s /usr/bin/clang-9 /usr/bin/clang
COPY parqueth.dpp /tmp/

RUN DFLAGS="-L=-L/usr/lib/llvm-9/lib/" dub run dpp -- /tmp/parqueth.dpp \
    --include-path /usr/include/glib-2.0 \
    --include-path /usr/lib/x86_64-linux-gnu/glib-2.0/include \
    --include-path /usr/include/arrow-glib \
    --include-path /usr/include/parquet-glib \
    --preprocess-only

The dockerfile will generate you the header file. You have to do this only once.

install.sh

apt-get update && apt-get install -y -V lsb-release wget
if [ $(lsb_release --codename --short) = "stretch" ]; then
  tee /etc/apt/sources.list.d/backports.list <<APT_LINE
deb http://deb.debian.org/debian $(lsb_release --codename --short)-backports main
APT_LINE
fi
wget https://apache.bintray.com/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-archive-keyring-latest-$(lsb_release --codename --short).deb
apt install -y -V ./apache-arrow-archive-keyring-latest-$(lsb_release --codename --short).deb
apt update && apt install -y -V libparquet-glib-dev libarrow-glib-dev

parqueth.dpp

#include <parquet-glib/arrow-file-reader.h>
#include <parquet-glib/arrow-file-writer.h>

ximion commented 4 years ago

I cannot use the tool gir-to-d due to its license

You know that output of (L)GPL licensed tools isn't subjected to the terms of these tools? (otherwise any document you edit in LibreOffice would become GPL-licensed, which would be a bit insane, and GCC couldn't be used to build proprietary software). So there shouldn't be a reason to avoid this tool, unless it's used in company tooling which has one of these crazy "no-copyleft-whatsoever-anywhere" policies. It is pretty neat that dpp works though, it has come a long way!

MikeWey commented 4 years ago

The generated code depends on GlibD which might be an issue.

andre2007 commented 4 years ago

@ximion you are totally right, but as the example from MikeWey shows, there are still a lot of traps. As an employee I really do not like to get in the position to be responsible for a case against the company I work for, therefore I avoid using anything gpl / lgpl as much as possible.

gtkd-developers / gir-to-d

Wrong method signature generated #27