altendky
commented
2 years ago Mmm... multiprocessing is being used with the default start method of fork with which "All resources of the parent are inherited by the child process". Explicitly specifying spawn instead with multiprocessing.set_start_method("spawn") works. spawn is described in part with "unnecessary file descriptors and handles from the parent process will not be inherited".
new commands that work
``` cat > x.py << EOF import multiprocessing import socket import memory_profiler address = ("127.0.0.1", 33125) def main(): multiprocessing.set_start_method("spawn") sock = socket.socket() sock.bind(address) profiled(sock) sock3 = socket.socket() sock3.bind(address) memory_profiler.memory_usage((profiled, [sock3])) print("really made it") def profiled(sock): sock.close() sock2 = socket.socket() sock2.bind(address) sock2.close() # auaughhghgh multiprocessing! if __name__ == "__main__": main() EOF cat x.py python3.9 -m venv venv venv/bin/python -m pip install --upgrade pip setuptools wheel venv/bin/pip install memory-profiler==0.60.0 psutil==5.8.0 venv/bin/python x.py venv/bin/python --version --version venv/bin/pip freeze uname -a lsb_release -a ```new working terminal session
```console $ cat > x.py << EOF > import multiprocessing > import socket > > import memory_profiler > > > address = ("127.0.0.1", 33125) > > > def main(): > multiprocessing.set_start_method("spawn") > > sock = socket.socket() > sock.bind(address) > > profiled(sock) > > sock3 = socket.socket() > sock3.bind(address) > > memory_profiler.memory_usage((profiled, [sock3])) > print("really made it") > > > def profiled(sock): > sock.close() > > sock2 = socket.socket() > sock2.bind(address) > sock2.close() > > > # auaughhghgh multiprocessing! > if __name__ == "__main__": > main() > EOF ``` ```console $ cat x.py import multiprocessing import socket import memory_profiler address = ("127.0.0.1", 33125) def main(): multiprocessing.set_start_method("spawn") sock = socket.socket() sock.bind(address) profiled(sock) sock3 = socket.socket() sock3.bind(address) memory_profiler.memory_usage((profiled, [sock3])) print("really made it") def profiled(sock): sock.close() sock2 = socket.socket() sock2.bind(address) sock2.close() # auaughhghgh multiprocessing! if __name__ == "__main__": main() ``` ```console $ python3.9 -m venv venv ``` ```console $ venv/bin/python -m pip install --upgrade pip setuptools wheel Requirement already satisfied: pip in ./venv/lib/python3.9/site-packages (21.1.1) Collecting pip Using cached pip-21.3.1-py3-none-any.whl (1.7 MB) Requirement already satisfied: setuptools in ./venv/lib/python3.9/site-packages (56.0.0) Collecting setuptools Using cached setuptools-60.1.0-py3-none-any.whl (952 kB) Collecting wheel Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB) Installing collected packages: wheel, setuptools, pip Attempting uninstall: setuptools Found existing installation: setuptools 56.0.0 Uninstalling setuptools-56.0.0: Successfully uninstalled setuptools-56.0.0 Attempting uninstall: pip Found existing installation: pip 21.1.1 Uninstalling pip-21.1.1: Successfully uninstalled pip-21.1.1 Successfully installed pip-21.3.1 setuptools-60.1.0 wheel-0.37.1 ``` ```console $ venv/bin/pip install memory-profiler==0.60.0 psutil==5.8.0 Collecting memory-profiler==0.60.0 Using cached memory_profiler-0.60.0-py3-none-any.whl Collecting psutil==5.8.0 Using cached psutil-5.8.0-cp39-cp39-manylinux2010_x86_64.whl (293 kB) Installing collected packages: psutil, memory-profiler Successfully installed memory-profiler-0.60.0 psutil-5.8.0 ``` ```console $ venv/bin/python x.py really made it ``` ```console $ venv/bin/python --version --version Python 3.9.5 (default, Jun 3 2021, 15:18:23) [GCC 9.3.0] ``` ```console $ venv/bin/pip freeze memory-profiler==0.60.0 psutil==5.8.0 ``` ```console $ uname -a Linux p1 5.4.0-91-generic #102-Ubuntu SMP Fri Nov 5 16:31:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux ``` ```console $ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal ```Ok, so with the cause identified, what next? I'm not exactly sure. Probably some documentation commenting on this with references to each of multiprocessing, spawn, fork, inherit-ed/ing, file descriptors, etc so it is findable by searching. I hesitate to mess with the multiprocessing global state and auto-configure this since code using memory-profiler may choose something else. If we do set the start method then it should only happen if not otherwise configured by the 'outer' code. I'll think it over and see what I can come up with.
Creating and binding a socket outside of a call to
memory_profiler.memory_usage()causes it to not close properly inside the call. After calling.close()the port is still claimed and a new socket is unable to be bound to the same port. Withoutmemory_profilerin the middle it works. I haven't gotten to digging into this yet but will share anything I find.Expand the collapsed sections at the bottom for a full set of commands to recreate and a full terminal session.
For linkage, I initially reported this at https://github.com/CFMTech/pytest-monitor/issues/53.
commands
``` cat > x.py << EOF import socket import memory_profiler address = ("127.0.0.1", 33125) def main(): sock = socket.socket() sock.bind(address) profiled(sock) sock3 = socket.socket() sock3.bind(address) memory_profiler.memory_usage((profiled, [sock3])) def profiled(sock): sock.close() sock2 = socket.socket() sock2.bind(address) sock2.close() main() EOF cat x.py python3.9 -m venv venv venv/bin/python -m pip install --upgrade pip setuptools wheel venv/bin/pip install memory-profiler==0.60.0 psutil==5.8.0 venv/bin/python x.py venv/bin/python --version --version venv/bin/pip freeze uname -a lsb_release -a ```full console session
```console $ cat > x.py << EOF > import socket > > import memory_profiler > > > address = ("127.0.0.1", 33125) > > > def main(): > sock = socket.socket() > sock.bind(address) > > profiled(sock) > > sock3 = socket.socket() > sock3.bind(address) > > memory_profiler.memory_usage((profiled, [sock3])) > > > def profiled(sock): > sock.close() > > sock2 = socket.socket() > sock2.bind(address) > sock2.close() > > > main() > EOF ``` ```console $ cat x.py import socket import memory_profiler address = ("127.0.0.1", 33125) def main(): sock = socket.socket() sock.bind(address) profiled(sock) sock3 = socket.socket() sock3.bind(address) memory_profiler.memory_usage((profiled, [sock3])) def profiled(sock): sock.close() sock2 = socket.socket() sock2.bind(address) sock2.close() main() ``` ```console $ python3.9 -m venv venv ``` ```console $ venv/bin/python -m pip install --upgrade pip setuptools wheel Requirement already satisfied: pip in ./venv/lib/python3.9/site-packages (21.1.1) Collecting pip Using cached pip-21.3.1-py3-none-any.whl (1.7 MB) Requirement already satisfied: setuptools in ./venv/lib/python3.9/site-packages (56.0.0) Collecting setuptools Using cached setuptools-60.1.0-py3-none-any.whl (952 kB) Collecting wheel Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB) Installing collected packages: wheel, setuptools, pip Attempting uninstall: setuptools Found existing installation: setuptools 56.0.0 Uninstalling setuptools-56.0.0: Successfully uninstalled setuptools-56.0.0 Attempting uninstall: pip Found existing installation: pip 21.1.1 Uninstalling pip-21.1.1: Successfully uninstalled pip-21.1.1 Successfully installed pip-21.3.1 setuptools-60.1.0 wheel-0.37.1 ``` ```console $ venv/bin/pip install memory-profiler==0.60.0 psutil==5.8.0 Collecting memory-profiler==0.60.0 Using cached memory_profiler-0.60.0-py3-none-any.whl Collecting psutil==5.8.0 Using cached psutil-5.8.0-cp39-cp39-manylinux2010_x86_64.whl (293 kB) Installing collected packages: psutil, memory-profiler Successfully installed memory-profiler-0.60.0 psutil-5.8.0 ``` ```console $ venv/bin/python x.py Traceback (most recent call last): File "/home/altendky/repos/chia-blockchain/tmp/x.py", line 29, in