nbdev_test fails when notebook imports polars package due to `set_start_method` conflict

[daniel-vainsencher]

Failure and repro

Trivial notebook/nbdev repo with import polars as pl fails (repro CI run) with error message:

ImportError: Polars only works with python multiprocessing method set to: `mp.set_start_method("spawn")` or `mp.set_start_method("forkserver")`before importing polars. See: https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

and the stack confirms (lines 324-5) the method is set to fork.

This reproduces locally (Linux) under nbdev_test, but not when run in a jupyter-notebook, thus I am assuming the difference is nbdev_test or a dep.

Context

Polars explanation of their assertion seems to suggest fork is not a uniformly appropriate choice.

I have not found nbdev_test UI to easily override the method (which would be a reasonable workaround).

My investigation so far

Candidate deciders of the method/callers of set_start_method include nbdev_test (which seems to implicitly allow 'fork'), or fastcore (but CI is not on darwin) and shell (but again, not on darwin).

I did not find justification for the different rules being applied to set the start method, and am not familiar enough with multiprocessing to propose a wise patch. But, as a relative newcomer to nbdev who is loving both it and polars, I thank everyone working on both and hope we can fix the conflict.

[jmoralez]

My 2 cents: In my case this only happens when installing polars from conda. I was able to get rid of it by installing from PyPI.