Recursive Execution of Script When Using Proxy on macOS

[MrDebugger]

Describe the bug When running a script that utilizes the Proxy class from the proxy.py library on macOS, the script enters a loop of recursive executions, creating multiple processes repeatedly and leading to port binding errors. This behavior does not occur on Linux.

To Reproduce Steps to reproduce the behavior:

1. save test.py with the following code and Run:

   from proxy import Proxy, sleep_loop
   with Proxy() as pr:
   sleep_loop(pr)

2. Execute the script on macOS.
3. Observe the script entering a loop of recursive executions and repeated port binding errors.

Expected behavior The script should initialize the Proxy instance, start the necessary processes, and handle connections without recursively re-executing the script and causing port binding errors.

Version information

• OS: macOS 12.3
• Browser: N/A
• Device: MacBook Pro
• proxy.py Version: 2.4.4

Additional context This behavior does not occur on Linux but only on MacOs

Screenshots

[MrDebugger]

a little more debugging on my side

Code

import logging
import argparse
import os
from proxy import Proxy, sleep_loop

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

logger.info(f"RUNNING in process {os.getpid()}")
try:
    with Proxy(['--port', str(8899)]) as pr:
        sleep_loop(pr)
except Exception as e:
    logger.error(f"An error occurred in process {os.getpid()}: {e}")

Logs

administrator@64719 dsolver % /opt/homebrew/bin/python3 /Users/administrator/Downloads/bot-mitigation/dsolver/test2.py
INFO:__main__:RUNNING in process 69838
INFO:proxy.common.plugins:Loaded plugin proxy.http.proxy.HttpProxyPlugin
DEBUG:proxy.core.listener.tcp:Listening on 127.0.0.1:8899
DEBUG:proxy.core.acceptor.pool:Started acceptor#0 process 69840
DEBUG:proxy.core.acceptor.pool:Started acceptor#1 process 69841
DEBUG:proxy.core.acceptor.pool:Started acceptor#2 process 69842
DEBUG:proxy.core.acceptor.pool:Started acceptor#3 process 69843
DEBUG:proxy.core.acceptor.pool:Started acceptor#4 process 69844
DEBUG:proxy.core.acceptor.pool:Started acceptor#5 process 69845
DEBUG:proxy.core.acceptor.pool:Started acceptor#6 process 69846
DEBUG:proxy.core.acceptor.pool:Started acceptor#7 process 69847
DEBUG:proxy.core.acceptor.pool:Started 8 acceptors in threadless (local) mode
INFO:__mp_main__:RUNNING in process 69840
INFO:proxy.common.plugins:Loaded plugin proxy.http.proxy.HttpProxyPlugin
INFO:__mp_main__:RUNNING in process 69842
ERROR:__mp_main__:An error occurred in process 69840: [Errno 48] Address already in use
INFO:proxy.common.plugins:Loaded plugin proxy.http.proxy.HttpProxyPlugin
ERROR:__mp_main__:An error occurred in process 69842: [Errno 48] Address already in use
DEBUG:asyncio:Using selector: KqueueSelector
DEBUG:proxy.core.work.threadless:Working on 0 works
INFO:__mp_main__:RUNNING in process 69843
INFO:proxy.common.plugins:Loaded plugin proxy.http.proxy.HttpProxyPlugin
ERROR:__mp_main__:An error occurred in process 69843: [Errno 48] Address already in use
INFO:__mp_main__:RUNNING in process 69841
INFO:proxy.common.plugins:Loaded plugin proxy.http.proxy.HttpProxyPlugin
ERROR:__mp_main__:An error occurred in process 69841: [Errno 48] Address already in use
DEBUG:asyncio:Using selector: KqueueSelector
DEBUG:proxy.core.work.threadless:Working on 0 works
DEBUG:asyncio:Using selector: KqueueSelector
DEBUG:proxy.core.work.threadless:Working on 0 works
DEBUG:asyncio:Using selector: KqueueSelector
DEBUG:proxy.core.work.threadless:Working on 0 works
INFO:__mp_main__:RUNNING in process 69845
INFO:proxy.common.plugins:Loaded plugin proxy.http.proxy.HttpProxyPlugin
ERROR:__mp_main__:An error occurred in process 69845: [Errno 48] Address already in use
INFO:__mp_main__:RUNNING in process 69847
INFO:proxy.common.plugins:Loaded plugin proxy.http.proxy.HttpProxyPlugin
ERROR:__mp_main__:An error occurred in process 69847: [Errno 48] Address already in use
INFO:__mp_main__:RUNNING in process 69844
INFO:proxy.common.plugins:Loaded plugin proxy.http.proxy.HttpProxyPlugin
ERROR:__mp_main__:An error occurred in process 69844: [Errno 48] Address already in use
DEBUG:asyncio:Using selector: KqueueSelector
DEBUG:proxy.core.work.threadless:Working on 0 works
DEBUG:asyncio:Using selector: KqueueSelector
DEBUG:proxy.core.work.threadless:Working on 0 works
INFO:__mp_main__:RUNNING in process 69846
INFO:proxy.common.plugins:Loaded plugin proxy.http.proxy.HttpProxyPlugin
ERROR:__mp_main__:An error occurred in process 69846: [Errno 48] Address already in use
DEBUG:asyncio:Using selector: KqueueSelector
DEBUG:proxy.core.work.threadless:Working on 0 works
DEBUG:asyncio:Using selector: KqueueSelector
DEBUG:proxy.core.work.threadless:Working on 0 works

[abhinavsingh]

@MrDebugger Thank you for reporting this. Are you on an Intel Mac or Apple silicon? I quickly tried installing proxy.py in a new virtual env and it seem to work as expected:

╰─ proxy --log-level d                                                                                                                                                                                   ─╯
2024-07-04 09:31:05,834 - pid:70097 [I] plugins.load:89 - Loaded plugin proxy.http.proxy.HttpProxyPlugin
2024-07-04 09:31:05,835 - pid:70097 [D] tcp.listen:95 - Listening on 127.0.0.1:8899
2024-07-04 09:31:05,836 - pid:70097 [D] pool._start:153 - Started acceptor#0 process 70099
2024-07-04 09:31:05,837 - pid:70097 [D] pool._start:153 - Started acceptor#1 process 70100
2024-07-04 09:31:05,839 - pid:70097 [D] pool._start:153 - Started acceptor#2 process 70101
2024-07-04 09:31:05,840 - pid:70097 [D] pool._start:153 - Started acceptor#3 process 70102
2024-07-04 09:31:05,842 - pid:70097 [D] pool._start:153 - Started acceptor#4 process 70103
2024-07-04 09:31:05,844 - pid:70097 [D] pool._start:153 - Started acceptor#5 process 70104
2024-07-04 09:31:05,846 - pid:70097 [D] pool._start:153 - Started acceptor#6 process 70105
2024-07-04 09:31:05,851 - pid:70097 [D] pool._start:153 - Started acceptor#7 process 70106
2024-07-04 09:31:05,851 - pid:70097 [D] pool.setup:109 - Started 8 acceptors in threadless (local) mode
2024-07-04 09:31:05,951 - pid:70099 [D] selector_events.__init__:54 - Using selector: KqueueSelector
2024-07-04 09:31:05,951 - pid:70099 [D] threadless.run:422 - Working on 0 works
2024-07-04 09:31:05,952 - pid:70101 [D] selector_events.__init__:54 - Using selector: KqueueSelector
2024-07-04 09:31:05,952 - pid:70101 [D] threadless.run:422 - Working on 0 works
2024-07-04 09:31:05,952 - pid:70100 [D] selector_events.__init__:54 - Using selector: KqueueSelector
2024-07-04 09:31:05,952 - pid:70100 [D] threadless.run:422 - Working on 0 works
2024-07-04 09:31:05,964 - pid:70102 [D] selector_events.__init__:54 - Using selector: KqueueSelector
2024-07-04 09:31:05,964 - pid:70102 [D] threadless.run:422 - Working on 0 works
2024-07-04 09:31:05,964 - pid:70103 [D] selector_events.__init__:54 - Using selector: KqueueSelector
2024-07-04 09:31:05,964 - pid:70103 [D] threadless.run:422 - Working on 0 works
2024-07-04 09:31:05,964 - pid:70105 [D] selector_events.__init__:54 - Using selector: KqueueSelector
2024-07-04 09:31:05,964 - pid:70105 [D] threadless.run:422 - Working on 0 works
2024-07-04 09:31:05,964 - pid:70104 [D] selector_events.__init__:54 - Using selector: KqueueSelector
2024-07-04 09:31:05,964 - pid:70106 [D] selector_events.__init__:54 - Using selector: KqueueSelector
2024-07-04 09:31:05,964 - pid:70104 [D] threadless.run:422 - Working on 0 works
2024-07-04 09:31:05,965 - pid:70106 [D] threadless.run:422 - Working on 0 works

When you get this error, can you check if the assigned ports are already in use on your system?

ps -p <pid>

[abhinavsingh]

Also see if following command too runs into error:

proxy --log-level d --num-acceptors 1

If yes, plz check if this port is already assigned

netstat -an | grep <port>

[MrDebugger]

@abhinavsingh

I am pretty sure the port is not assigned to anything I have tried changing the port too in the script.

Here is the output to the requested command

2024-07-05 08:58:56,243 - pid:9289 [D] utils.set_open_file_limit:320 - Open file soft limit set to 1024
2024-07-05 08:58:56,243 - pid:9289 [I] plugins.load:89 - Loaded plugin proxy.http.proxy.HttpProxyPlugin
2024-07-05 08:58:56,243 - pid:9289 [D] tcp.listen:95 - Listening on 127.0.0.1:8899
2024-07-05 08:58:56,245 - pid:9289 [D] pool._start:153 - Started acceptor#0 process 9291
2024-07-05 08:58:56,245 - pid:9289 [D] pool.setup:109 - Started 1 acceptors in threadless (local) mode
2024-07-05 08:58:56,308 - pid:9291 [D] selector_events.__init__:54 - Using selector: KqueueSelector
2024-07-05 08:58:56,308 - pid:9291 [D] threadless.run:422 - Working on 0 works

The error only occurs when we run the proxy server in a python file or module

Here is my machine info

    Hardware Overview:

      Model Name: Mac mini
      Model Identifier: Macmini9,1
      Model Number: Z12P000KKLL/A
      Chip: Apple M1
      Total Number of Cores: 8 (4 performance and 4 efficiency)
      Memory: 16 GB
      System Firmware Version: 10151.1.1
      OS Loader Version: 10151.1.1

[abhinavsingh]

@MrDebugger IIUC, your experience is -- proxy.py works fine when run from command line but it runs into port issues when started from a script. This is unusual and not seen before.

What's your Python version?

[MrDebugger]

@abhinavsingh You are correct, the issue is only in script and in MacOs. My python version is 3.11.6

[MrDebugger]

@abhinavsingh Any progress on this issue?

[abhinavsingh]

@MrDebugger Thank you for reporting this. Luckily, I am able to reproduce this. From first look, it wasn't clear why exactly this is happening. I did following to debug a little:

def __enter__(self) -> 'Proxy':
        print("ENTERING SETUP " * 10)
        self.setup()

I added this print statement, ran your script and noticed:

1) Proxy context manager __enter__ method is getting called twice. Hence, it runs into [Errno 48] Address already in use errors since we are already listening on the port. 2) Same behaviour doesn't exhibit with proxy command itself, which technically is using the same Proxy class internally

Then, I tried this script:

from proxy import Proxy, sleep_loop

if __name__ == "__main__":
    with Proxy(["--port", "8899"]) as pr:
        sleep_loop(pr)

and IT WORKS as expected.

Without if __name__ == "__main__" construct, looks like, code gets executed twice 🤔.

[abhinavsingh]

I'll close this issue since the underlying issue is likely something else. Proxy seems to be working fine in itself. If you have any insights on the same, do share.