Out of Memory crash when optimizing code containing shifts, ternary and bitwise operators on storage variables via IR

[Jack-Clark]

Description

Given the following contract:

contract Test {
    int8 b;
    int8 c;
    int8 o;
    int8 d;
    int8 e;
    int8 f;
    bool g;
    int8 h;
    int8 i;
    uint112 j;
    int8 k;
    int8 l;
    int8 m;
    int8 n;

    constructor() {
        c = k <<= 0 <= j ? j : 0;
        o <<= j;
        f = m >> j;
        g = (e >>= 0) >= (g ? int40(0) : int56(0));
        l = h &= d <<= j;
        n = (o = b) | (h >>= 0) << (0 <= i ? 0 : j);
    }
}

I get an out of memory error when running solc --bin --optimize --via-ir Test.sol. I can't reproduce the issue with either the --optimize or --via-ir flags set separately, or with both omitted. I have run this on two machines both with 32GB RAM and both OOM. Here is the dmesg output from one machine:

oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service,task=solc-0.8.18,pid=301840,uid=1000
Out of memory: Killed process 301840 (solc-0.8.18) total-vm:22943372kB, anon-rss:22925304kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:44944kB oom_score_adj:0

Environment

• Compiler version: 0.8.18+commit.87f61d96.Linux.g++
• Operating system: Ubuntu 20.04

[cameel]

Slightly simplified repro:

contract C {
    uint128 u128;
    int8 a;
    int8 b;
    int8 c;

    bool x = (b = b >> 0) == int8(x ? 0 : 0);

    int8 d = d << u128;
    int8 e = e << u128;
    int8 f = (a = a << u128);
    int8 g = (c &= (a = a << u128));
    int8 h = a | (c = c >> 0) << (x ? u128 : u128);
}

I couldn't get it any simpler than that. Haven't verified it actually runs into OOM but it does keep running for a long time. It keeps eating more and more memory, but much more slowly than the original repro.

It looks like some pathological case in Yul optimizer.

It is only reproducible on 0.8.18+. Below that it finishes fast. The only thing in the changelog that looks related is this:

• Optimizer: Added optimization rule and(shl(X, Y), shl(X, Z)) => shl(X, and(Y, Z)).