search

malexer / pytest-spark

pytest plugin to run the tests with support of pyspark
MIT License
85 stars 30 forks source link

Reuse across invocations #16

Open dustinfarris opened 4 years ago

dustinfarris commented 4 years ago

When using pytest-watch, is there a way for the spark session/context to be reused across test runs?

malexer commented 4 years ago

As far as I can see from quick look into pytest-watch - there is no easy way to do it out of the box because pytest-watch starts new process (pytest) when it detects changes in your source files.

Can you elaborate a bit what problem are you trying to solve by reusing session/context across test runs?

dustinfarris commented 4 years ago

Just trying to save time. Rebooting the session/context on every iteration is ~5 seconds of waiting each time that could be eliminated if there was a way to keep it from being garbage collected after each run.

Sent from my iPhone

On Dec 24, 2019, at 04:15, Alex (Oleksii) Markov notifications@github.com wrote:

 As far as I can see from quick look into pytest-watch - there is no easy way to do it out of the box because pytest-watch starts new process (pytest) when it detects changes in your source files.

Can you elaborate a bit what problem are you trying to solve by reusing session/context across test runs?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

malexer commented 4 years ago

ok, fair enough I will think about it, but for now I don't see any easy solution :(