Update bindings for Capstone 4.0.x

[aquynh]

So far, we have around 20 bindings for Capstone 3.x, but after updating to v4.0 (see https://github.com/aquynh/capstone/releases/tag/4.0), except Python binding, most become broken due to some changes in cs_insn structure.

Besides that, v4 added few architectures and some more APIs, that most bindings are missing now.

Finally, v4 added instructions & stuff for most architectures, but that would be handled if you are using https://github.com/aquynh/capstone/blob/master/bindings/const_generator.py.

To be sure, you can diff include/capstone/*.h between v3.0.5 & v4.0, and see the details.

Below is summary of what is broken, and new features that bindings should update to support.

Broken

---

[ x86 ]

• cs_x86 added new fields: xop_cc & encoding, eflags/fpu_flags
• cs_x86.disp type changed from int32_t to int64_t

[ mips ]

• add mode CS_MODE_MIPS2

[ PowerPC ]

• change cs_ppc_op.imm type from int32_t to int64_t.

[ Sparc ]

• change cs_sparc_op.imm type from int32_t to int64_t.

New features

---

• new APIs: cs_regs_access(). see http://www.capstone-engine.org/op_access.html
• new options for cs_option(): CS_OPT_MNEMONIC & CS_OPT_UNSIGNED & CS_OPT_SYNTAX_MASM.
• add 4 new architectures: EVM, M68K, M680X & TMS320C64x.
• add new group types: CS_GRP_PRIVILEGE & CS_GRP_BRANCH_RELATIVE.
• add new error types: CS_ERR_X86_MASM.
• new instructions for most architectures.

Good bindings for Capstone 4.0.x

---

Here is the list of bindings that fully support Capstone 4.0.x.

• Python (maintain in source code tree)
• .NET
• D

[aquynh]

cc @bnagy, @radare, @parasyte, @zer0mem, @Dax89, @capstone-rust, @9ee1, @kevemueller, @ibabushkin, @collarchoke, @beatcracker, @FuzzySecurity, @firodj, @bohlender, @williballenthin, @jingtaozf, @cagyirey, @stievie, @t00sh

[9ee1]

I've been working on updating Capstone.NET to support Capstone 4, along with some general improvements to Capstone.NET's codebase. I am hoping to release by the end of the week. I'll post here to let you know once I do.

[bohlender]

I've updated the D bindings to the v4.0 API. Not all of the new architectures are supported yet, but will be once I've slimmed down the template usage in the codebase.

[disconnect3d]

Not sure if this is proper issue to post this in but since there is a 4.0.1 version can you release Python bindings for it on PyPi?

[aquynh]

yes i pinged @rhelmot, who is the lord in charge of Pypi package, on v4.0.1 release. lets wait for her reply.

[aquynh]

Pypi package for v4.0.1 is out now. thank you, @rhelmot!

[disconnect3d]

Thanks a lot for this 🍷 !

However, I found one minor bug:

root@618558d983a5:/t# python -c 'import capstone; print(capstone.__version__)'
4.0.0
root@618558d983a5:/t# python -c 'import pkg_resources; print(pkg_resources.get_distribution("capstone").version)'
4.0.1

So the version in setup.py is correct (it is fetched from pkgconfig.mk (https://github.com/aquynh/capstone/blob/4.0.1/bindings/python/setup.py#L30-L57) but the __version__ attribute set in the capstone/__init__.py is incorrect: https://github.com/aquynh/capstone/blob/4.0.1/bindings/python/capstone/__init__.py#L130-L139

Maybe it should be set via pkg_resources.get_distribution("capstone").version? Afair this would require setuptools to be installed but it's probably not a big deal?

[aquynh]

sorry that i forgot to bump Python version up: https://github.com/aquynh/capstone/commit/f52532a0028dadd9b8cd12c5c1471177405bfbbb

[aquynh]

@disconnect3d sounds reasonable, please send pull req

[9ee1]

Capstone.NET has been (finally) updated to support Capstone 4, along with a bunch of new goodies. Get the code here: https://github.com/9ee1/Capstone.NET/tree/V2.0.0, or get the Nuget package here: https://www.nuget.org/packages/Gee.External.Capstone/2.0.0

[aquynh]

thanks for your great work, @9ee1!

i updated the first post, with the list of bindings fully support Capstone 4.0.x (that includes D, Python & .NET for now). please let me know if any other bindings get updated.

[david942j]

The Ruby binding crabstone has no update (and no response to issues/prs) since 2015. I'm willing to fork it and upgrade it to support Capstone4. (I'm also the owner of crabstone on RubyGems.org)

[aquynh]

@david942j, i remember that @bnagy was looking for helper, or even new maintainer for Crabstone, so you can discuss with him.

[david942j]

When trying to develop the Ruby binding for Capstone 4.x I found there're changes of cs_insn and cs_detail in master (committed after the 4.0.1 release): https://github.com/aquynh/capstone/blob/c25414d1fb61587a150e95cc15331def2681b41a/include/capstone/capstone.h#L336 The size of cs_insn#bytes is changed from 16 to 24.

This change will affect most bindings, will it be included in version 4.x? If so, bindings will need check the (minor) version of Capstone and decide which structure of cs_insn and cs_detail should be used, which makes code ugly.

Besides, this kind of change may lead bindings hard to debug. For example, since the array size of cs_insn#bytes is changed, for FFI-based bindings with wrong structure may face heap overflow, which will be hard to figure out why that happens. Is there an easier way for bindings to not face such issues? Like, maybe Capstone could provide APIs for bindings can somehow check if the structures changed.

[aquynh]

oh i missed that change!

@erique, so m68k really needs to extend the size of bytes field to 24?

[aquynh]

@david942j, that is a good question. except by documenting all the important changes, i am not sure how providing an API can help bindings, since we need the semantics as well, not just the structure size.

[david942j]

Yap I agree this problem is hard to deal with. I decided to write scripts in crabstone for parsing capstone.h to generate structures into Ruby form, and use different structure definition according to the output of cs_version.

One thing Capstone can do might be having a README for third-party binding developers, remind them must check the value of cs_version and suggest them to generate all constants/structures automatically.

[aquynh]

I am thinking about bumping the next version to 5.0, but not 4.1, since this breaks bindings now

[erique]

@aquynh (sorry for the slow response - travelling atm) It's necessary if the intent is to keep the entire instruction within cs_insn.bytes as the biggest instruction for the 68000 family (according to the programmers reference, and also tested in the small testbed in the PR) is 11 words, i.e. 22 bytes. I opted on making it 24 bytes due to alignment.

The other change in that PR suffers from the same issue, where I had to bump the size of affected registers, as there are 68000 instructions that affect all GPRs (8 data + 8 address registers).

I agree it's an API breaking change and thus should probably be part of a major version update. Sorry for not making that statement in the original PR..

[bohlender]

I am thinking about bumping the next version to 5.0, but not 4.1, since this breaks bindings now

In my opinion this is the way to go. That's exactly how semantic versioning is supposed to be implemented.

[aquynh]

David, feel free to update bindings/README with instructions for binding authors.

[aquynh]

FYI, i just bumped master version to 5.0. this will be released with major updates on all architectures, and we will get some more new architectures, too.

[knightsc]

Working to get 4.0.x support into the Golang bindings (Gapstone) here:

https://github.com/bnagy/gapstone/pull/23

[david942j]

@aquynh @bnagy I have upgraded the Ruby binding (Crabstone) to support Capstone4 and the gem has been pushed to rubygems.org as well.

[XVilka]

@aquynh @kabeor probably could be closed too?

[ddkwork]

where is bind for golang?

[Rot127]

The bindings for golang haven't been updated for a while. Contributions are very much welcome for this! Also related to https://github.com/capstone-engine/capstone/issues/2089