On Wrapping CBash

[GandaG]

I'd like to bring forth a proposal - this is obviously a long-term goal but from short discussions with @Infernio I believe it's something achievable (since part of it already has been) and can be considered. Wrapping CBash in Rust, RBash.

TL;DR - Memory-safe with no loss of performance wrapper. Python 3 64bit only. Long term goal. Example. Built wheel.

CBash is written in C++ and afaik I believe none of the current devs have the skill in it needed to tackle working on CBash. The biggest draw of C/C++ is the perfomance, mainly via the low-level access, which let's you bypass some slower, safer abstractions to achieve it. It also benefits from being an old language, pretty much everything has been done with C++ and you can leverage that. And if there is something you need that hasn't, you can just use C, which was created around the time the primordial dust was gathering into what would one day become Earth.

Rust, on the other hand, is the hipster newcomer. Version 1 in 2015(?), with a new edition in 2018. A memory-safe systems programming language with influence from functional programming. No more segmentation faults or memory leaks. It seems to be more and more talked about as a credible alternative to C/C++ but it could just be confirmation bias on my end. Pretty easy to learn, too. Biggest drawback is its age - the existing material for it is in no way comparable to the others above. BUT, much like C++, we can just use C behind the scenes and hope it doesn't segfault or memory leak there :P In terms of performance I've seen benchmarks that placed it above C++ and below C, and I know that benches are not real life situations but the point is that performance shouldn't be an issue.

Rust also allows us to create a python extension module from code completely in rust. In the example repo I'm using pyo3, although there is another crate who can also create python extensions. This can be discussed later if this proposal is accepted.

As far as specific goals for the wrapper, I'd like to divide them into 3 stages. These are all long term goals that will be done slowly, over time.

• Stage 1 - Create a simple wrapper around CBash public API and expose it to python. Essentially a replacement for ctypes in cint. This should already be accomplished in this repo, but I'm sure I forgot something or there's some mistake, can't really test right now.

• Stage 2 - Consume cint. Slowly take over cint, start moving the python code there into rust. The templates/macros in rust should also help with reducing the monstruosity that is that module right now and provide a better API. At the end of this stage the public RBash API should be stabilized.

• Stage 3 - Consume CBash. This is a very long term goal, but if done slowly over time we could do it. Not much to say here, CBash is ridiculously huge, who knows what secrets it hides.

Over these stages, we'll never lose access to CBash and indeed we'll be using as RBash's own backend, so there's never any "big" breaking step. Keep in mind this is just a proposal and nothing is really set in stone.

EDIT: A handy flowchart to help with deciding :P

Hope this gets greenlit!

[Utumno]

That's a nice idea - some random thoughts: would be nice to have an automated way of creating records based say on the xEdit defs (I would throw this as an idea to the modding community). And/or we could consider maybe cython and get us rid of cbash entirely. Haven't looked at it but maybe worth a look. If performance (memory and speed) is comparable we could stay with a python/cython patcher. Finally, comparing the P and C patchers we might come up with a new patcher routine. In general, if we find a way of having the logic and the records (reworked so we can generate them automatically from common defs like the xEdit) in python (cython) we might be able to get best of both worlds - python ease of coding and c performance

[GandaG]

Disclaimer - I intensely dislike Cython:

• It makes excellent complex wrappers, as for when a backend lib wasn't designed with python at all in mind and you have to actually make more than a thin wrapper - look into lxml's source, brilliant design.

• "Naive" Cython (regular python with typed variables) is quite faster than Python but still not in the same range as C. That performance can be achieved with "optimized" Cython, which at that point is just C code with a crap syntax.

• As said above, Cython is C, not C++ - we'd be losing access to all those goodies it brings, hash maps, vectors, templates. I haven't looked further than the topmost levels of CBash but I've already seen heavy usage of these. If we want a list in Cython we could either go with the slow Python lists, a pre-allocated array or numpy arrays (which I've never used but heard they're really good for simple numeric types).

Cython would be absolutely fine up to stage 2, it's its specialty. But we'll suffer from the same problem we currently have - the C++ -> Cython API has to be "dumbed down" to C. Neither CPython or Cython understand the C++ goodies so we have to pass everything as opaque structs and functions.

With Rust we lose nothing - all the Rust high-level abstractions and goodies that are used for the backend (currently the CBash wrapper and in stage 3 the actual code) can and are used to build the Python wrapper.

The Rust solution isn't perfect either, pyo3 has a severe lack of documentation and doesn't support everything we could do with Python, but it's honestly getting there. For example, with cython we could use enums while with pyo3 we have to (for now) use this monstruosity ("enum variants" are global variables in modules imported into the main module).

Just my two cents on this. If we're willing to consider "rewriting" CBash, we should do it correctly from the start, half-assed solutions will just land us here in another few years. Cython is a good wrapper, but I believe we'd be losing value if moving CBash into it. Not saying that Rust is the answer, just that I don't think Cython is.