That is a good idea. In my recent big update, I switched spark with delta-rs. This means that no heavy Spark cluster is required for the pipeline part anymore. That helps, for sure. Docker might be another step. I will think more, but it should start with a couple of commands for now. I also made a video that helps with it.
That is a good idea. In my recent big update, I switched spark with delta-rs. This means that no heavy Spark cluster is required for the pipeline part anymore. That helps, for sure. Docker might be another step. I will think more, but it should start with a couple of commands for now. I also made a video that helps with it.