search

kamu-data / kamu-cli

Next-generation decentralized data lakehouse and a multi-party stream processing network
https://kamu.dev
Other
300 stars 13 forks source link

Ingest caching should account for templated variables #847

Open sergiimk opened 1 week ago

sergiimk commented 1 week ago

Describe the bug

Currently changing an environment variable does not invalidate the ingest cache

Steps To Reproduce

  1. Use a dataset that templates in some env vars
  2. Ingest data with one env var value
  3. Delete dataset
  4. Create dataset again
  5. Ingest data with a different env var value

Current Behavior

The ingest will reuse the cached data

Expected Behavior

The ingest will fetch data anew as the env var changes may have influenced the result

kamu system info

0.203.0

Logs

No response

Anything else?

No response