Move away from numpy

[pepijndevos]

On an Intel Linux machine you can just install numpy binary packages and live happily. However, there has been a slow trickle of requests to get rid of C dependencies that are hard to bundle and compile in various contexts.

This mainly comes from the direction of oss-cad-suite (@mmicko) and yowasp (@whitequark) which are uh somewhat unconventional builds. It's not a problem I personally have, but if it makes it easier to install an FPGA toolchain that seems like a positive thing.

Apicula doesn't actually do any linear algebra, we just use Numpy as a matrix of bits. We use a few packing/unpacking operations and that's it for the most part.

@yrabbit suggested we might try to use https://github.com/wadetb/tinynumpy but I actually think that's not super helpful because we don't need the linear algebra and it doesn't have the bit manipulation stuff.

@whitequark mentioned Glasgow has a bits type that we could steal, which is more in line with what we actually use Numpy for I believe, although only one-dimensional: https://github.com/GlasgowEmbedded/glasgow/blob/main/software/glasgow/support/bits.py

Maybe we can steal code from Glasgow to implement packbits in tinynumpy or something like that? Or literally just port everything to plain python lists.

So in conclusion, it would be a welcome change that is fairly mechanical, but would require some work to port/implement our bit manipulation operations.

Our numpy usage:

$ git grep "np\."
apycula/bslib.py:    return np.fliplr(np.array(bitmap)), hdr, ftr
apycula/bslib.py:    new_bs = np.vstack((bs, bsram_init))
apycula/bslib.py:    bs = np.fliplr(bs)
apycula/bslib.py:    pad = np.ones((bs.shape[0], padlen), dtype=np.uint8)
apycula/bslib.py:    bs = np.hstack([pad, bs])
apycula/bslib.py:    bs=np.packbits(bs, axis=1)
apycula/bslib.py:        lst, _ = np.histogram(bs, bins=[i for i in range(256)])
apycula/bslib.py:            data=np.packbits(data, axis=1))
apycula/chipdb.py:    template: np.ndarray = None
apycula/chipdb.py:    res = np.zeros((db.height, db.width), dtype=np.uint8)
apycula/fuse_h4x.py:    tile = np.zeros((h, w), np.uint8)#+(255-ttyp)
apycula/fuse_h4x.py:    bitmap = np.zeros((height, width), np.uint8)
apycula/fuse_h4x.py:    res = np.zeros((height, width), dtype=np.uint8)
apycula/fuse_h4x.py:    rows, cols = np.where(tile==1)
apycula/gowin_pack.py:        bsram_init_map = np.zeros((256 * len(db.simplio_rows), db.template.shape[1]), dtype=np.uint8)
apycula/gowin_pack.py:    loc_map = np.zeros((256, 3 * 60), dtype = np.int8)
apycula/gowin_pack.py:    loc_map = np.flipud(loc_map)
apycula/gowin_pack.py:    bs = np.fliplr(bs)
apycula/gowin_pack.py:    bs=np.packbits(bs)
apycula/gowin_pack.py:    bb = np.array(bs)

[whitequark]

This mainly comes from the direction of oss-cad-suite (@mmicko) and yowasp (@whitequark) which are uh somewhat unconventional builds.

What's unconventional about oss-cad-suite?

[pepijndevos]

I guess for the purpose of this issue "unconventional" is anywhere where you can't just pip install apycula and it'll wheel out numpy in a matter of seconds.

I don't know their exact build process but I think they statically compile everything into a nice bundle, and apparently installing numpy for them isn't as easy as pip install numpy.

[whitequark]

Ah, I see. I would actually call Apicula itself unconventional because it's the furthest thing from normal to ship a bitstream packer as a Python application with library dependencies that require network access (and as such it causes no end of issues). No other toolchain does this.

[pepijndevos]

Well I'm open to someone to Rewrite it in Rust

[whitequark]

That would solve basically all issues I've ever had with Apicula as a packager (and to a lesser extent as a user). I've considered it at one point but I did not have enough time to justify starting it.

[uis246]

In C(++) that uses vendor db directly with headers recreated from debuginfo

[whitequark]

Thanks again for this!