When running a df.apply operation which uses a Series.to_dict() call on a Dask engine, I get a traceback telling me: AssertionError: daemonic processes are not allowed to have children. This was without Ray/modin[ray] installed. Other apply operations succeed.
Installing modin[ray] also causes this operation to succeed. Weirdly (to me), the first operation in the included reproducible example seems to run on the Dask engine before Modin automatically creates a Ray instance to handle the second operation.
Expected Behavior
The Dask engine should support all typical Pandas operations.
Error Logs
```python-traceback
python -m rembe.benchmarks.minimal_test
Fine worked fine
UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 44471 instead
2024-07-17 22:23:59,043 - distributed.nanny - ERROR - Failed to start process
Traceback (most recent call last):
File "/z/home/mxak/miniforge3/envs/datamesh_no_dl/lib/python3.12/site-packages/distributed/nanny.py", line 452, in instantiate
result = await self.process.start()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/z/home/mxak/miniforge3/envs/datamesh_no_dl/lib/python3.12/site-packages/distributed/nanny.py", line 752, in start
await self.process.start()
File "/z/home/mxak/miniforge3/envs/datamesh_no_dl/lib/python3.12/site-packages/distributed/process.py", line 55, in _call_and_set_future
res = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/z/home/mxak/miniforge3/envs/datamesh_no_dl/lib/python3.12/site-packages/distributed/process.py", line 215, in _start
process.start()
File "/z/home/mxak/miniforge3/envs/datamesh_no_dl/lib/python3.12/multiprocessing/process.py", line 118, in start
assert not _current_process._config.get('daemon'), \
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: daemonic processes are not allowed to have children
2024-07-17 22:23:59,045 - distributed.nanny - ERROR - Failed to start process
Traceback (most recent call last):
File "/z/home/mxak/miniforge3/envs/datamesh_no_dl/lib/python3.12/site-packages/distributed/nanny.py", line 452, in instantiate
result = await self.process.start()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/z/home/mxak/miniforge3/envs/datamesh_no_dl/lib/python3.12/site-packages/distributed/nanny.py", line 752, in start
await self.process.start()
File "/z/home/mxak/miniforge3/envs/datamesh_no_dl/lib/python3.12/site-packages/distributed/process.py", line 55, in _call_and_set_future
res = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/z/home/mxak/miniforge3/envs/datamesh_no_dl/lib/python3.12/site-packages/distributed/process.py", line 215, in _start
process.start()
File "/z/home/mxak/miniforge3/envs/datamesh_no_dl/lib/python3.12/multiprocessing/process.py", line 118, in start
assert not _current_process._config.get('daemon'), \
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: daemonic processes are not allowed to have children
```
Modin version checks
[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest released version of Modin.
[X] I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow this guide.)
Reproducible Example
Issue Description
When running a
df.apply
operation which uses aSeries.to_dict()
call on a Dask engine, I get a traceback telling me:AssertionError: daemonic processes are not allowed to have children
. This was without Ray/modin[ray] installed. Otherapply
operations succeed.Installing
modin[ray]
also causes this operation to succeed. Weirdly (to me), the first operation in the included reproducible example seems to run on the Dask engine before Modin automatically creates a Ray instance to handle the second operation.Expected Behavior
The Dask engine should support all typical Pandas operations.
Error Logs
Installed Versions