search

linkedin / kafka-remote-storage-azure

BSD 2-Clause "Simplified" License
13 stars 2 forks source link

Optimize Topic-Partition to Azure Blobs Mapping #3

Open sutambe opened 2 years ago

sutambe commented 2 years ago

The current implementation of the mapping of a topic-partition to an Azure Blob Container is as follows. The container name is an optional prefix and the canonical representation of the topic UUID. The index files and the local log segment file itself are copied as individual blobs. In summary the mapping is as follows.

|----------- container name ------------|----------------blob name---------------------|
|OptionalContainerPrefix-TopicIdJavaUUID|PartitionNum.LogSegmentCanonicalUUID.IndexType|

There are some disadvantages of this mapping.

  1. As described in #2, the container is a dangling resource after topic deletion.
  2. Reading a single remote segment file and all the index files are 6 round-trip calls. The files include .segment, .LEADER_EPOCH, .OFFSET .PRODUCER_SNAPSHOT, .TIMESTAMP, and .TRANSACTION. There may or may not be a performance impact of 6 round-trips.
  3. There's a possibility of partial failures where a subset of the files are copied correctly but not the rest. Garbage collection of the orphan remote log segments is important.

Due to the above limitations (a potentially more), a better mapping of log segment to Azure Blobs is desirable.

malthe commented 1 year ago

Would Azure Table Storage be able to meet some of the indexing needs? It's considerably cheaper (transaction cost) and you can run batch requests, avoiding multiple round-trips.