Choose your execution engine (in-memory/in-database)

[simonaubertbd]

Hello,

A dreamt feature would be the ability to choose the execution engine. Some dataprep/etl tools have it (like Alteryx or Dataiku). That means two possibilities :

• in-memory when the tool retrieve the data on its machine and treats along the differents steps and write. -in-database where data stay in the database and the tool only send sql queries to proceed the different steps. It allows very big volume in a very small time, and does not depend of the resource of the etl machine.

This is hard to develop of course but I think this has definitely this place on the roadmap.

Best regards,

Simon

[tgourdel]

Thanks, definitely on my radar! I'm particularly looking at https://github.com/ibis-project/ibis to enable this.