xndtools fails to generate xnd_structinfo.c

[costrouc]

Leaving here for discussion tomorrow xnd meeting. Most likely this is due to me passing in the incorrect include files.

c_utils.py

...
    for stmt in block[1:-1].split(';')[:-1]:
        stmt = stmt.strip()
        print('========')
        print(stmt)
        if stmt.startswith('PyObject_HEAD'):
            items.append('PyObject_HEAD')
            stmt = stmt.split(None, 1)[1]
        m = union_match(stmt)
        if m is not None:
            key,  = m.groups()
            items.append(('union',_get_block_items(blocks[key], blocks)))
            continue
        m = struct_match(stmt)
        if m is not None:
            key, name = m.groups()
            items.append(('struct',_get_block_items(blocks[key], blocks), name))
            continue
        size = None
        if stmt.endswith(']'): # <typespec> <name>[<size>]
            i = stmt.index('[')
            size = stmt[i+1:-1].strip()
            stmt = stmt[:i]
        print('========')
        print(stmt, size)
        name = rname_match(stmt[::-1]).group(0)[::-1].strip()
        assert name is not None, repr(stmt)
        typespec = stmt[:-len(name)].strip()
        if typespec.startswith('alignas'):
            typespec = typespec.split(' ', 1)[1]
        typespec = _normalize_typespec(typespec)
        items.append((typespec,name,size))
...

I have added print statements here in the file.

With the following error when running

structinfo_generator structinfo_config.py

int64_t datasize
========
int64_t datasize None
========
uint16_t align
========
uint16_t align None
========
#ifdef _MSC_VER
    __int64 volatile refcnt
========
#ifdef _MSC_VER
    __int64 volatile refcnt None
========
#else
    _Atomic int64_t refcnt
========
#else
    _Atomic int64_t refcnt None
========
#endif

    union @@@49@@@
========
#endif

    union @@@49@@@ None
Traceback (most recent call last):
  File "/nix/store/gpmracibiqixbw73r0imv4adjfscxwb0-python3.7-xndtools/bin/.structinfo_generator-wrapped", line 74, in <module>
    main()
  File "/nix/store/gpmracibiqixbw73r0imv4adjfscxwb0-python3.7-xndtools/bin/.structinfo_generator-wrapped", line 71, in main
    args.func(args)
  File "/nix/store/gpmracibiqixbw73r0imv4adjfscxwb0-python3.7-xndtools/lib/python3.7/site-packages/xndtools/structinfo_generator.py", line 126, in generate
    structs = c_utils.get_structs(source)
  File "/nix/store/gpmracibiqixbw73r0imv4adjfscxwb0-python3.7-xndtools/lib/python3.7/site-packages/xndtools/c_utils.py", line 179, in get_structs
    structs[name] = _get_block_items(remove_comments(blocks[key]), blocks)
  File "/nix/store/gpmracibiqixbw73r0imv4adjfscxwb0-python3.7-xndtools/lib/python3.7/site-packages/xndtools/c_utils.py", line 139, in _get_block_items
    name = rname_match(stmt[::-1]).group(0)[::-1].strip()
AttributeError: 'NoneType' object has no attribute 'group'

I understand that the regex is not matching but I am not too familiar with xndtools so I am not sure if it should have even gotten this output.

[saulshanabrook]

We have to likely debug this file: https://github.com/plures/xndtools/blob/master/xndtools/c_utils.py

If you pip install xndtools from source then you can edit and try running that command again.

[costrouc]

Yes that is exactly where I am right now. I have xndtools installed and working. Trying to debug where the parsing error occurs

[saulshanabrook]

Sweet. If you make any progress, I would publish to a branch of that and switch numba-xnd to install from that branch till your PR is merged.

Once we can get this to parse the new XND code at least, then we can get started on getting it up to date, starting with the nrt branch. I will put some time into working on that this weekend or early next week, to get the repo in a place where things are at least working and can be extended.

[costrouc]

Further progress I have isolated the issue to xndtools not handling c directives. Take this simple example.

from xndtools import c_types

source = '''
struct _ndt {

    enum ndt tag;
    uint16_t align;

#ifdef _MSC_VER
    __int64 volatile refcnt;
#else
    _Atomic int64_t refcnt;
#endif

    union {
        struct {
            char *name;
            const ndt_t *type;
        } Module;
    };
};
'''

c_utils.get_structs(source)

It will fail. This is due to the directives being added to ndtypes.h see git blame it was added 1 month ago. https://github.com/plures/ndtypes/blame/c4cf463cb67e0d71ccb613388b04cac7d960eb1d/libndtypes/ndtypes.h.in#L358

[saulshanabrook]

See https://github.com/aguinet/dragonffi/issues/13 for another approach which removes necessity of xndtools

[costrouc]

@saulshanabrook when https://github.com/plures/xndtools/pull/11 is merged numba-xnd will no longer be blocked by this issue and can be closed. I went with a solution that allows growth (default naive preprocessor with option to use compiler preprocessors such as gcc). I have checked that structinfo_generator structinfo_config.py will work.

[saulshanabrook]

Great! Thanks for solving this