Open antonymilne opened 4 hours ago
P.S. this isn't really a feature request but it didn't fit in any other category so there you are... I guess the feature request might be please make it easier to not use parallel runner, omegaconf or pre-commit-hooks but this is kind of an edge case I guess and it's possible already with some workarounds, so I don't think you need to do anything about it for my purposes anyway. Maybe it's another small data point for @astrojuanlu's considerations on how modular kedro should be though. The biggest problem here I think is the omegaconf dependency but if and when they release 2.4.0 that will be resolved for this particular setup anyway.
Mainly I just put this here as a report of what's currently possible since there was nowhere better to put it.
Edit: oh wait, I see that discussions are open now. Maybe I should have put it there. I leave it up to you to decide whether you want to move it anyway!
Greetings friends 😀 and FYI @maxschulz-COL @maartenbreddels
Just wanted to report that it is sort of possible to run Kedro in the browser using PyCafe! ☕ 🚀 There's a couple of workarounds needed but for the spaceflights project at least it's not too hard. Take a look here for my example project: https://py.cafe/antonymilne/kedro-vizro-dashboards
I did
kedro new
withkedro==0.19.9
and used the spaceflights example code (the project is in the folder descriptively namedblah
). The project is run inapp.py
. At the moment PyCafe requires some kind of app so there's also a simple Vizro app just so that the kedro code can run. Of course in reality this could be an actual interesting dashboard reporting results of the pipeline.The catches/workarounds that I noticed with this simple example. No doubt there will be further difficulties for more realistic projects, and I didn't even try to install all the requirements in
blah/requirements.txt
, just took what looked like the bare minimum to get the pipeline to run.antlr4-python3-runtime
thatomegaconf
currently requires. This is fixed inomegaconf
2.4.0 but that's only in dev release. I see that you're aware of this difficulty already in https://github.com/facebookresearch/hydra/issues/2699 and https://github.com/omry/omegaconf/issues/1158. So this is why I setomegaconf==2.4.0.dev3
in the requirements.txt, which appears to work wellpre-commit-hooks
hasruamel-yaml-clib
as a transitive dependency, which doesn't have a wheel file available. I was actually quite surprised to seepre-commit-hooks
as a kedro requirement, and I see this was a bit of a controversial addition at the time (https://github.com/kedro-org/kedro/pull/3436). It's easy to fix if you don't actually want to run pre-commit though, just withruamel-yaml-clib # mock
in requirements.txt.pandas.CSVDataset
orpandas.ExcelDataset
.No module named '_multiprocessing'
I mocked out the parallel runner.