Livelock on deserialization of an enum, with nested FST-usage in its clinit

[s-rwe]

Encountered a possible livelock with FST, which occurs on a single thread without any concurrency in play.

It is triggered by a combination of nested FST-use and class-initialization, and basically results in a deadlock of a thread running on 100% CPU, busy-waiting for itself indefinitely.

Fairly minimal snippet where this happens:

import org.nustaq.serialization.simpleapi.DefaultCoder;

public class FST {
  private static final DefaultCoder coder = new DefaultCoder();

  // Dummy serialized version of E1.A
  static final byte[] serializedE1 = new byte[]
      {(byte) 0xFA, 0x01, 0x06,
          0x46, 0x53, 0x54, 0x24, 0x45, 0x31, // "FST$E1"
          0x00};

  // --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.math=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.text=ALL-UNNAMED --add-opens=java.sql/java.sql=ALL-UNNAMED
  public static void main(String[] args) {
    System.out.println("Starting");
    E1 e1 = (E1) coder.toObject(serializedE1);
    System.out.println("Done, got: " + e1);
  }

  enum E1 {
    A, B, C;

    final byte[] serializedMe;

    E1() {
      serializedMe = coder.toByteArray(this);
    }
  }
}

The (sole) main thread will hang in following busy-lock-acquire of FSTClazzInfoRegistry.getCLInfo():

  while(!rwLock.compareAndSet(false,true));

What happens here to trigger the issue:

1. Deserialization of an enum constant with FST, for which the enum-class has not yet been initialized (I can't reproduce the same issue with a regular class instead of an enum)
2. The initialization of that enum-class triggers another, nested FST-related operation (serialization or deserialization) somewhere, on the same coder instance (doesn't matter on what sort of class this nested operation is done, AFAICS)

The self-serialization of E1 in its constructor in the example is quite artificial and just done to keep the snippet small; in real life this could be anything done by the <clinit> of the enum and its constants - like some static final field which triggers some other FST-call somewhere further down the line, in some other class.

The problem does not manifest if the nested use of FST is using a separate coder instance.

FST version this happens with: 3.0.4-jdk17