joblib.Parallel has pre_dispatch default to 2 * n_jobs, which seems unintuitive for a Civis Platform user expecting only n_jobs parallel jobs. The Sphinx docs do mention this behavior and subtly recommend setting n_jobs explicitly:
The default value of pre_dispatch is "2n_jobs", which will run a maximum of 2 n_jobs jobs at once in the Civis Platform. Set pre_dispatch="n_jobs" in your :class:~joblib.Parallel call to run at most n_jobs jobs.
Perhaps we should more explicitly warn about this in the documentation, with one of the Sphinx call-out note boxes. In this case, the .. warning:: paragraph directive would seem appropriate.
joblib.Parallelhaspre_dispatchdefault to2 * n_jobs, which seems unintuitive for a Civis Platform user expecting onlyn_jobsparallel jobs. The Sphinx docs do mention this behavior and subtly recommend settingn_jobsexplicitly:Perhaps we should more explicitly warn about this in the documentation, with one of the Sphinx call-out note boxes. In this case, the
.. warning::paragraph directive would seem appropriate.