search

reactorlabs / rir

GNU General Public License v2.0
62 stars 18 forks source link

Serializing an EXTERNALSXP can lead to an endless loop #1303

Open rihafilip opened 3 months ago

rihafilip commented 3 months ago

When a WriteItem is called with an EXTERNALSXP that is not Code, Function or DispatchTable, it ends up in an endless loop.

WriteItem calls to an externalCodeWrite function pointer, which is set to serializeRir. Then it calls WriteItem with the result of rirDecompile.

Currently, the rirDecompile function only deserializes Code, Function and DispatchTable types. For all others it returns the SEXP unmodified.

Hence the second call to WriteItem is the same as the first one and it loops endlesly.

rihafilip commented 3 months ago

Reproducing this issue is a bit trickier, easiest way I found was with the recording tool: triggering a EnvStubMaterialized deopt, which has a LazyEnvironment as a trigger, creates a new event with this trigger saved and at the end of execution forces a LazyEnvironment to be serialized, which leads to an endless loop described above.

Smallest example I've been able to find:

RIR_RECORD_SERIALIZE=1 RIR_RECORD=/tmp/rec.rds RIR_RECORD_FILTER=Deopt R -e "library(ggplot2)"

By putting an assert(false) on line 18 in interpreter/decompile.cpp, the code fails faster on the incorrect inputs.