search

PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
17.52k stars 1.64k forks source link

Should RedisStorageContainer include a default expiration time? #15530

Open Hank-Yan opened 1 month ago

Hank-Yan commented 1 month ago

Describe the current behavior

I understand that all result_storage storage media files require third-party management. Currently, with Prefect’s behavior, when the cache expires, the files stored in result_storage are not automatically cleared, which is quite confusing.

This is especially concerning when Redis is used as storage since Redis memory usage grows rapidly. Should we add a default expiration time or a global expiration time as a temporary solution?

Describe the proposed behavior

Ideally, when the cache expires, the files should be automatically deleted, which would be the best outcome.

For the case of using Redis for result_storage, I would suggest first adding a global expiration time or a default to expiration time serve as a transitional solution.

Thank you for your attention.

Example Use

No response

Additional context

No response

zzstoatzz commented 1 month ago

hi @Hank-Yan - can you clarify which version of prefect you are using? ideally, please include the output of prefect version

Hank-Yan commented 1 month ago

Hi @zzstoatzz my prefect info is below

  1. My prefect info : Version: 3.0.3 API version: 0.8.4 Python version: 3.11.7 Git commit: d9c51bc3 Built: Fri, Sep 20, 2024 8:52 AM OS/Arch: linux/x86_64 Profile: local Server type: server Pydantic version: 2.9.0b2 Integrations: prefect-redis: 0.2.0

  2. prefect config view result:
    PREFECT_PROFILE='local' PREFECT_API_KEY='****' (from profile) PREFECT_API_URL='http://ip:port/api' (from profile) PREFECT_DEFAULT_RESULT_STORAGE_BLOCK='redisstoragecontainer/redis-storage-block' (from profile) PREFECT_RESULTS_DEFAULT_SERIALIZER='json' (from profile)

  3. redis-storage-block source code is: from prefect.blocks import redis redis.RedisStorageContainer.from_host( host="127.0.0.1", port=6379, password="default").save("redis-storage-block", overwrite=True)