search

tweag / funflow

Functional workflows
MIT License
362 stars 33 forks source link

Can we enhance the caching functionality to work seamlessly with Haskell code changes? #218

Open bglgwyng opened 4 months ago

bglgwyng commented 4 months ago

Currently, caching does not respond to changes in Haskell code. As a result, we need to dockerize the Haskell code, and caching should be applied to the Docker task. I believe there isn't a way to achieve this without dockerizing the Haskell code, since we can't hash functions.

To address this, we should consider modularizing the Haskell code and building each Docker image independently. This approach will allow us to maximize Funflow's caching capabilities. Do we have any better options?

Additionally, even though we lack a robust solution for this issue at the moment, it would be beneficial to have the ability to disable caching globally. This would prevent caching from causing problems during development. Currently, the runFlowWithConfig does not provide an option to turn off caching. The only available setting allows us to specify the caching store directory, which requires creating a specific directory to effectively disable the cache. Is there a more efficient way to achieve this?

I am relatively new to Funflow, and my question may contain misunderstandings. Please correct me if I am mistaken.

GuillaumeDesforges commented 4 months ago

Hi @bglgwyng!

As I understand it, funflow has a cache function based only on the input data, while you'd like it to use the input data and the code. This makes sense (at least to me). If you'd like to help move that feature forward, we would be open to a PR. :slightly_smiling_face:

To further discuss, you can join us on Discord! https://discord.gg/w3pCHJ997s