search

rizinorg / rizin

UNIX-like reverse engineering framework and command-line toolset.
https://rizin.re
GNU Lesser General Public License v3.0
2.66k stars 357 forks source link

Python 3.11, and 3.12 versions bytecode support #3130

Open XVilka opened 1 year ago

XVilka commented 1 year ago

See also dis module from the corresponding Python versions: https://docs.python.org/3/library/dis.html

The changes are possibly required in these files:

3.10

CPython bytecode changes The MAKE_FUNCTION instruction now accepts either a dict or a tuple of strings as the function’s annotations. (Contributed by Yurii Karabas and Inada Naoki in bpo-42202.)

3.11

CPython bytecode changes

The bytecode now contains inline cache entries, which take the form of the newly-added CACHE instructions. Many opcodes expect to be followed by an exact number of caches, and instruct the interpreter to skip over them at runtime. Populated caches can look like arbitrary instructions, so great care should be taken when reading or modifying raw, adaptive bytecode containing quickened data.

New opcodes

Replaced opcodes

Replaced Opcode(s) | New Opcode(s) | Notes -- | -- | -- BINARY_*INPLACE_* | BINARY_OP | Replaced all numeric binary/in-place opcodes with a single opcode CALL_FUNCTIONCALL_FUNCTION_KWCALL_METHOD | CALLKW_NAMESPRECALLPUSH_NULL | Decouples argument shifting for methods from handling of keyword arguments; allows better specialization of calls DUP_TOPDUP_TOP_TWOROT_TWOROT_THREEROT_FOURROT_N | COPYSWAP | Stack manipulation instructions JUMP_IF_NOT_EXC_MATCH | CHECK_EXC_MATCH | Now performs check but doesn’t jump JUMP_ABSOLUTEPOP_JUMP_IF_FALSEPOP_JUMP_IF_TRUE | JUMP_BACKWARDPOP_JUMP_BACKWARD_IF_*POP_JUMP_FORWARD_IF_* | See 3; TRUE, FALSE, NONE and NOT_NONE variants for each direction SETUP_WITHSETUP_ASYNC_WITH | BEFORE_WITH | with block setup
3

All jump opcodes are now relative, including the existing JUMP_IF_TRUE_OR_POP and JUMP_IF_FALSE_OR_POP. The argument is now an offset from the current instruction rather than an absolute location.

Changed/removed opcodes

  • Changed MATCH_CLASS and MATCH_KEYS to no longer push an additional boolean value to indicate success/failure. Instead, None is pushed on failure in place of the tuple of extracted values.

  • Changed opcodes that work with exceptions to reflect them now being represented as one item on the stack instead of three (see gh-89874).

  • Removed COPY_DICT_WITHOUT_KEYS, GEN_START, POP_BLOCK, SETUP_FINALLY and YIELD_FROM.

CPython bytecode changes The bytecode now contains inline cache entries, which take the form of the newly-added [CACHE](https://docs.python.org/3.11/library/dis.html#opcode-CACHE) instructions. Many opcodes expect to be followed by an exact number of caches, and instruct the interpreter to skip over them at runtime. Populated caches can look like arbitrary instructions, so great care should be taken when reading or modifying raw, adaptive bytecode containing quickened data. New opcodes [ASYNC_GEN_WRAP](https://docs.python.org/3.11/library/dis.html#opcode-ASYNC_GEN_WRAP), [RETURN_GENERATOR](https://docs.python.org/3.11/library/dis.html#opcode-RETURN_GENERATOR) and [SEND](https://docs.python.org/3.11/library/dis.html#opcode-SEND), used in generators and co-routines. [COPY_FREE_VARS](https://docs.python.org/3.11/library/dis.html#opcode-COPY_FREE_VARS), which avoids needing special caller-side code for closures. [JUMP_BACKWARD_NO_INTERRUPT](https://docs.python.org/3.11/library/dis.html#opcode-JUMP_BACKWARD_NO_INTERRUPT), for use in certain loops where handling interrupts is undesirable. [MAKE_CELL](https://docs.python.org/3.11/library/dis.html#opcode-MAKE_CELL), to create [Cell Objects](https://docs.python.org/3.11/c-api/cell.html#cell-objects). [CHECK_EG_MATCH](https://docs.python.org/3.11/library/dis.html#opcode-CHECK_EG_MATCH) and [PREP_RERAISE_STAR](https://docs.python.org/3.11/library/dis.html#opcode-PREP_RERAISE_STAR), to handle the [new exception groups and except*](https://docs.python.org/3.11/whatsnew/3.11.html#whatsnew311-pep654) added in [PEP 654](https://peps.python.org/pep-0654/). [PUSH_EXC_INFO](https://docs.python.org/3.11/library/dis.html#opcode-PUSH_EXC_INFO), for use in exception handlers. [RESUME](https://docs.python.org/3.11/library/dis.html#opcode-RESUME), a no-op, for internal tracing, debugging and optimization checks. Replaced opcodes Replaced Opcode(s) New Opcode(s) Notes BINARY_* INPLACE_* [BINARY_OP](https://docs.python.org/3.11/library/dis.html#opcode-BINARY_OP) Replaced all numeric binary/in-place opcodes with a single opcode CALL_FUNCTION CALL_FUNCTION_KW CALL_METHOD [CALL](https://docs.python.org/3.11/library/dis.html#opcode-CALL) [KW_NAMES](https://docs.python.org/3.11/library/dis.html#opcode-KW_NAMES) [PRECALL](https://docs.python.org/3.11/library/dis.html#opcode-PRECALL) [PUSH_NULL](https://docs.python.org/3.11/library/dis.html#opcode-PUSH_NULL) Decouples argument shifting for methods from handling of keyword arguments; allows better specialization of calls DUP_TOP DUP_TOP_TWO ROT_TWO ROT_THREE ROT_FOUR ROT_N [COPY](https://docs.python.org/3.11/library/dis.html#opcode-COPY) [SWAP](https://docs.python.org/3.11/library/dis.html#opcode-SWAP) Stack manipulation instructions JUMP_IF_NOT_EXC_MATCH [CHECK_EXC_MATCH](https://docs.python.org/3.11/library/dis.html#opcode-CHECK_EXC_MATCH) Now performs check but doesn’t jump JUMP_ABSOLUTE POP_JUMP_IF_FALSE POP_JUMP_IF_TRUE [JUMP_BACKWARD](https://docs.python.org/3.11/library/dis.html#opcode-JUMP_BACKWARD) [POP_JUMP_BACKWARD_IF_*](https://docs.python.org/3.11/library/dis.html#opcode-POP_JUMP_BACKWARD_IF_TRUE) [POP_JUMP_FORWARD_IF_*](https://docs.python.org/3.11/library/dis.html#opcode-POP_JUMP_FORWARD_IF_TRUE) See [3](https://docs.python.org/3.11/whatsnew/3.11.html#bytecode-jump); TRUE, FALSE, NONE and NOT_NONE variants for each direction SETUP_WITH SETUP_ASYNC_WITH [BEFORE_WITH](https://docs.python.org/3.11/library/dis.html#opcode-BEFORE_WITH) [with](https://docs.python.org/3.11/reference/compound_stmts.html#with) block setup [3](https://docs.python.org/3.11/whatsnew/3.11.html#id4) All jump opcodes are now relative, including the existing [JUMP_IF_TRUE_OR_POP](https://docs.python.org/3.11/library/dis.html#opcode-JUMP_IF_TRUE_OR_POP) and [JUMP_IF_FALSE_OR_POP](https://docs.python.org/3.11/library/dis.html#opcode-JUMP_IF_FALSE_OR_POP). The argument is now an offset from the current instruction rather than an absolute location. Changed/removed opcodes Changed [MATCH_CLASS](https://docs.python.org/3.11/library/dis.html#opcode-MATCH_CLASS) and [MATCH_KEYS](https://docs.python.org/3.11/library/dis.html#opcode-MATCH_KEYS) to no longer push an additional boolean value to indicate success/failure. Instead, None is pushed on failure in place of the tuple of extracted values. Changed opcodes that work with exceptions to reflect them now being represented as one item on the stack instead of three (see [gh-89874](https://github.com/python/cpython/issues/89874)). Removed COPY_DICT_WITHOUT_KEYS, GEN_START, POP_BLOCK, SETUP_FINALLY and YIELD_FROM. # 3.12 - Removed the [LOAD_METHOD](https://docs.python.org/3.12/library/dis.html#opcode-LOAD_METHOD) instruction. It has been merged into [LOAD_ATTR](https://docs.python.org/3.12/library/dis.html#opcode-LOAD_ATTR). [LOAD_ATTR](https://docs.python.org/3.12/library/dis.html#opcode-LOAD_ATTR) will now behave like the old [LOAD_METHOD](https://docs.python.org/3.12/library/dis.html#opcode-LOAD_METHOD) instruction if the low bit of its oparg is set. (Contributed by Ken Jin in [gh-93429](https://github.com/python/cpython/issues/93429).) - Removed the JUMP_IF_FALSE_OR_POP and JUMP_IF_TRUE_OR_POP instructions. (Contributed by Irit Katriel in [gh-102859](https://github.com/python/cpython/issues/102859).)
blackjackal010 commented 7 months ago

hi, i would like to work on this