The engine needs a mechanism for determining it should use dask datasets or pandas datasets. It may be a good idea to create some sort of config variable so this can be modified by the user as well.
AC:
When working with large datasets (threshold to be determined in this issue) the engine should use dask. Otherwise it should use pandas.
The engine needs a mechanism for determining it should use dask datasets or pandas datasets. It may be a good idea to create some sort of config variable so this can be modified by the user as well.
AC:
When working with large datasets (threshold to be determined in this issue) the engine should use dask. Otherwise it should use pandas.