Strange error after topic deletion, needs investigation

[ivanyu]

Sorry for the unclear issue, this needs looking deeper, filing to not forget.

The setup:

1. 3 node cluster + tiered storage (S3).
2. A topic was successfully created, filled, read from the remote.
3. The topic was deleted and the leader node started logging this continuously:

   [2023-06-05 04:57:23,491] ERROR Error encountered while writing committed offsets to a local file (org.apache.kafka.server.log.remote.metadata.storage.PrimaryConsumerTask:222)
   java.nio.file.NoSuchFileException: /kafka/log/topic1-0/remote_log_snapshot.tmp
       at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
       at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
       at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
       at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:181)
       at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
       at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357)
       at org.apache.kafka.server.log.remote.metadata.storage.RemoteLogMetadataSnapshotFile.write(RemoteLogMetadataSnapshotFile.java:92)
       at org.apache.kafka.server.log.remote.metadata.storage.FileBasedRemoteLogMetadataCache.flushToFile(FileBasedRemoteLogMetadataCache.java:107)
       at org.apache.kafka.server.log.remote.metadata.storage.RemotePartitionMetadataStore.syncLogMetadataSnapshot(RemotePartitionMetadataStore.java:124)
       at org.apache.kafka.server.log.remote.metadata.storage.PrimaryConsumerTask.syncCommittedDataAndOffsets(PrimaryConsumerTask.java:215)
       at org.apache.kafka.server.log.remote.metadata.storage.PrimaryConsumerTask.consumeFromPrimaryConsumer(PrimaryConsumerTask.java:193)
       at org.apache.kafka.server.log.remote.metadata.storage.PrimaryConsumerTask.run(PrimaryConsumerTask.java:172)
       at java.base/java.lang.Thread.run(Thread.java:833)

   Without looking deeper into the code I can't tell if it's just cosmetics that needs to be suppressed or it's an indicator of some real issue.