Implement parsing to a bit vector in C++

[HackerFoo]

Bitstream generation with xc-fasm uses a significant amount of time, most of which is spent parsing FASM text with textX.

Given that the FASM format is fairly simple, and the input is in the megabytes, this could be much faster if written in a low level language such as C++.

While the FASM format is well established at this point, it would be nice to keep the same Python interface to allow easy integration and filtering of the FASM input and bit vector output.

Plan to use C++ within Python

There are a number of ways to interface Python and C: cython, Boost Python, ctypes, and many more, several of which are compared in this article.

I have chose ctypes for the following reasons:

• The interface should be very simple (filename or string in, byte vector out), so there is no need to model C++ objects.
• ctypes comes with Python, so there is no extra dependencies required.

I have written some test code that approximates how this will work: https://github.com/HackerFoo/cpyo

So then, kernel.cpp will be replaced with code that converts FASM input to a bit vector, reading from prjxray as needed, and I will use ctypes to call it from xc-fasm in place of fasm_assembler from prjxray.

[litghost]

Given that your own data shows that 99% of the time is spent in textx, why would you convert the fasm_assembler stuff to C++? The vast majority of the gain is from avoiding textx for the parsing tree, not in the python used to assemble the bit vector?

[HackerFoo]

The assembler isn't very large or complex (201 lines for fasm_assembler.py, and 128 for bitstream.py, about half of which is comments), and doing only parsing in C++ would require passing a lot more data across the FFI than necessary.

[litghost]

You also need to manage reading segbits, and pathing to segbit data.

[HackerFoo]

@litghost @mithro I have created a document here with more details, which I will keep updated.