search

jacopotagliabue / you-dont-need-a-bigger-boat

An end-to-end implementation of intent prediction with Metaflow and other cool tools
MIT License
842 stars 65 forks source link

Comparison between this and something like Kedro or ZenML? #11

Closed IanQS closed 2 years ago

IanQS commented 2 years ago

Hey there! Thanks for this amazing work! I was wondering if you /any users here had done a comparison between these projects. Seems like this repo just describes a general set of tools and how they link up together whereas the others take an opinionated stance and do the linking themselves?

I'm in the process of evaluating tools / frameworks.

I'm currently a single person looking to set up the groundwork for things to come. So far my plans have been to stay local for as long as possible before moving to some distributed computing framework (Dask gave me a lot of trouble in the past). I'm also looking to avoid using tools such as AWS or GCP for as long as possible so ideally the discussion would revolve around local machines. I'd love to hear thoughts and opinions.

jacopotagliabue commented 2 years ago

Hi @IanQS, thanks for our interest in our work! The general philosophy is collected in this series: the repo does not really "describe tools", it actually gives you working end-to-end code that goes from millions of data points to an endpoint - without any need for devOps people, and with pretty good future proofing, as most components would scale by themselves.

We prefer the "pick and choose approach" over the "one platform for all", which tends to work less well for people with sophisticated needs: using metaflow has been a sweet spot in that sense, as a framework easy to adopt, but also infinitely extensible. If you look for a (still end-to-end, fully ready to go) simpler stack, we just released this: https://github.com/jacopotagliabue/post-modern-stack - it is an easier starting point, compared to this repository.

On the issue local vs cloud, again metaflow is your friend: you start local and scale to AWS when you can / are ready, with minimal changes. We wrote extensively about distributed computing so won't repeat it here ;-)

If you'd like to chat more, do reach out - I'm usually around in the MLOps Community slack: mlops-community.slack.com - if you look for their next hands-on workshop, it should be me explaining the above repo step by step, in case you want to see a live demo!