Projects including Fugue itself needs to test with different backends. The current way to test is to manually create Spark, Dask, Ray and Duckdb sessions, test and stop them. In order to get good unit testing performance, it's actually tricky to setup those backends. Another issues is, if there is session level pytest fixture, then some sessions will be started and stopped in each single test, adding significant overhead.
So, we will create fixtures for Fugue and related projects. As the first release, we will have these hard coded fixtures (with the best possible performance for unit tests)
[x] Spark support, fugue_spark_session
[x] Dask support: fugue_dask_client
[x] Ray support: fugue_ray_session
[x] Duckdb support: fugue_duckdb_connection
[x] Make them configurable in the common pytest config
[x] Make them available automatically as long as fugue is installed
[x] Create enumerated engine (parameterized), so it will automatically test all available backend
To use them, we just need to pip install fugue with the specific extra, then here is an example:
import fugue.api as fa
def test_my_app(fugue_dask_client):
with fa.engine_context(fugue_dask_client):
"""test"""
Currently, for spark backend, please keep using pytest-spark and spark_session provided by the package.
Projects including Fugue itself needs to test with different backends. The current way to test is to manually create Spark, Dask, Ray and Duckdb sessions, test and stop them. In order to get good unit testing performance, it's actually tricky to setup those backends. Another issues is, if there is session level pytest fixture, then some sessions will be started and stopped in each single test, adding significant overhead.
So, we will create fixtures for Fugue and related projects. As the first release, we will have these hard coded fixtures (with the best possible performance for unit tests)
fugue_spark_session
fugue_dask_client
fugue_ray_session
fugue_duckdb_connection
To use them, we just need to pip install fugue with the specific extra, then here is an example:
Currently, for spark backend, please keep using pytest-spark and
spark_session
provided by the package.