alacritty / vte

Parser for virtual terminal emulators
https://docs.rs/vte/
Apache License 2.0
242 stars 56 forks source link

fix ub unpack routine #102

Closed ethanpailes closed 11 months ago

ethanpailes commented 11 months ago

This patch fixes the undefined behavior in the unpack routine resulting from the fact that enum representation is undefined. Previously, the code assumed that the enum values were represented in the sensible way, which technically it had no way to be sure of. With the repr tags, it is ok to cast directly to the enum values.

I'm just opening this as a PR directly rather than filing an issue first since the fix seems pretty simple. There are some other options for fixing it such as using match statements.

Reference on enum repr rules: https://rust-lang.github.io/unsafe-code-guidelines/layout/enums.html. It says "If no explicit #[repr] attribute is used, then the layout of a data-carrying enum is typically not specified. However, in certain select cases, there are guaranteed layout optimizations that may apply, as described below." Maybe this is one of those select cases, but it is probably best not to rely on that.