Aiven-Open / tiered-storage-for-apache-kafka

RemoteStorageManager for Apache Kafka® Tiered Storage
Apache License 2.0
95 stars 20 forks source link

rsm.config.fetch.chunk.cache.path not in force #505

Closed funky-eyes closed 8 months ago

funky-eyes commented 8 months ago

What can we help you with?

Where would you expect to find this information?

remote.log.metadata.topic.replication.factor=1
remote.log.storage.system.enable=true
remote.log.metadata.topic.retention.ms=-1
rsm.config.fetch.chunk.cache.class=io.aiven.kafka.tieredstorage.fetch.cache.DiskChunkCache
rsm.config.fetch.chunk.cache.path=/data01/kafka-tiered-storage-cache
rsm.config.storage.backend.class=io.aiven.kafka.tieredstorage.storage.s3.S3Storage
# Pick some cache size, 16 GiB here:
rsm.config.fetch.chunk.cache.size=17179869184
# # Prefetching size, 16 MiB here:
rsm.config.fetch.chunk.cache.prefetch.max.size=16777216
rsm.config.storage.s3.bucket.name=kafka
rsm.config.storage.s3.region=kafka
rsm.config.storage.s3.path.style.access.enabled=true
rsm.config.storage.s3.endpoint.url=http://minio-tdbackup-offline.txxxxx.txxxxx:9000
rsm.config.storage.aws.secret.access.key=xxxxx
rsm.config.storage.aws.access.key.id=kafka
rsm.config.chunk.size=4194304
remote.log.storage.manager.class.path=/home/admin/core-0.0.1-SNAPSHOT/*:/home/admin/s3-0.0.1-SNAPSHOT/*
remote.log.storage.manager.class.name=io.aiven.kafka.tieredstorage.RemoteStorageManager
remote.log.metadata.manager.class.name=org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
remote.log.metadata.manager.listener.name=PLAINTEXT

image This folder produces nothing.

ivanyu commented 8 months ago

Hi @funky-eyes. Did you try to fetch some data?

funky-eyes commented 8 months ago

Hi @funky-eyes. Did you try to fetch some data?

Yes, I've found that as soon as real-time data is consumed, it seems that there is no longer any data in this cache