Server.stop() does not kill the java process in Windows

[ghost]

When the server is started it does so using subprocess.Popen('browsermob-proxy.bat')

The result process's pid on windows is the cmd.exe process that controls the bat file, but I don't think Windows is linking the java.exe which the bat file spawns.

Calling server.stop() kills the cmd.exe, but a java.exe is left hanging around at about 35MB.

This may be a problem with windows not killing child process correctly and it is apparently not possible to get the java pid in the bat file without some actual windows-style grep for java.exe.

[fpoolev]

This problem still exists on Windows 7 as described above.

[DarthOpto]

try proxy.close() prior to the server.stop() that helped me, although if the test fails mid way through it still doesn't close the process.

[dshuga]

Calling proxy.close() prior to server.stop() didn't resolve this issue for me on Windows 7.

[cbdelavenne]

Having the same issue on Mac OS X and Windows Server 2012.

This is my workaround:

import psutil

def terminate_browsermob_processes(self):
    self._browsermob_proxy.close()
    self._browsermob_server.stop()

    # Find BrowserMob-Proxy processes that may still be alive and kill them
    for process in psutil.process_iter():
        try:
            process_info = process.as_dict(attrs=['name', 'cmdline'])
            if process_info.get('name') in ('java', 'java.exe'):
                for cmd_info in process_info.get('cmdline'):
                    if cmd_info == '-Dapp.name=browsermob-proxy':
                        process.kill()
        except psutil.NoSuchProcess:
            pass

[ghost]

On Debian 8 (not Windows)

Python: Selenium v3.0.2, python3.4, browsermobproxy v0.5.0 JAVA: BrowserMob Proxy version 2.1.4

When run, it will create these two applicable lines in the ps aux: /bin/sh /usr/local/browsermobproxy/bin/browsermob-proxy --port=8080 and: java -Dapp.name=browsermob-proxy -Dbasedir=/usr/local/browsermobproxy -jar /usr/local/browsermobproxy/lib/browsermob-dist-2.1.4.jar --port=8080

But all of the various suggestions (above) to stop the BMP Server always leave this in the ps aux output:

java -Dapp.name=browsermob-proxy -Dbasedir=/usr/local/browsermobproxy -jar /usr/local/browsermobproxy/lib/browsermob-dist-2.1.4.jar --port=8080

I need to get rid of this proc: How? Suggestions? Something akin to @cbdelavenne 's solution is not possible, it would destroy the ongoing main process.

The shutdown code:

        if(self.route == 'proxify'):
            if(debug):
                logging.info(Fore.YELLOW +'Turning off the browsermob-proxy server')
            #self.proxy.close()
            self.proxy_server.stop()
        if(debug):
            logging.info(Fore.YELLOW +'Turning off the Selenium WebDriver')
        self.driver.quit()
        if(debug):
            logging.info(Fore.YELLOW +'Turning off the X-Window Frameless Visual Buffer (headless browsing)')
        self.display.stop()

Produces this debugging output:

INFO:root:Turning off the browsermob-proxy server
INFO:root:Turning off the Selenium WebDriver
DEBUG:selenium.webdriver.remote.remote_connection:DELETE http://127.0.0.1:43014/session/8e48269a5c3ef89499e896e7bdb59449 {"sessionId": "8e48269a5c3ef89499e896e7bdb59449"}
DEBUG:selenium.webdriver.remote.remote_connection:Finished Request
INFO:root:Turning off the X-Window Frameless Visual Buffer (headless browsing)
DEBUG:pyvirtualdisplay.abstractdisplay:unset DISPLAY
DEBUG:easyprocess:stopping process (pid=29711 cmd="['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1001']")
DEBUG:easyprocess:process is active -> sending SIGTERM
DEBUG:easyprocess:process has ended
DEBUG:easyprocess:return code=0
DEBUG:easyprocess:stdout=
DEBUG:easyprocess:stderr=

server.log:

Running BrowserMob Proxy using LittleProxy implementation. To revert to the legacy implementation, run the proxy with the command-line option '--use-littleproxy false'.
[INFO  2017-02-02T19:10:15,742 net.lightbody.bmp.proxy.Main] (main) Starting BrowserMob Proxy version 2.1.4 
[INFO  2017-02-02T19:10:15,784 org.eclipse.jetty.util.log] (main) jetty-7.x.y-SNAPSHOT 
[INFO  2017-02-02T19:10:15,852 org.eclipse.jetty.util.log] (main) started o.e.j.s.ServletContextHandler{/,null} 
[INFO  2017-02-02T19:10:16,140 org.eclipse.jetty.util.log] (main) Started SelectChannelConnector@0.0.0.0:8080 
[INFO  2017-02-02T19:10:17,643 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp359742806-16) Starting proxy at address: 0.0.0.0/0.0.0.0:8081 
[INFO  2017-02-02T19:10:17,685 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp359742806-16) Proxy listening with TCP transport 
[INFO  2017-02-02T19:10:17,831 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp359742806-16) Proxy started at address: /0:0:0:0:0:0:0:0:8081 
[INFO  2017-02-02T19:10:49,754 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp359742806-16) Shutting down proxy server gracefully 
[INFO  2017-02-02T19:10:49,755 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp359742806-16) Closing all channels (graceful) 
[INFO  2017-02-02T19:10:49,782 org.littleshoot.proxy.impl.ServerGroup] (qtp359742806-16) Shutting down server group event loops (graceful) 
[INFO  2017-02-02T19:10:51,991 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp359742806-16) Done shutting down proxy server 

bmp.log:

[INFO  2017-02-02T19:10:15,742 net.lightbody.bmp.proxy.Main] (main) Starting BrowserMob Proxy version 2.1.4 
[INFO  2017-02-02T19:10:15,784 org.eclipse.jetty.util.log] (main) jetty-7.x.y-SNAPSHOT 
[INFO  2017-02-02T19:10:15,852 org.eclipse.jetty.util.log] (main) started o.e.j.s.ServletContextHandler{/,null} 
[INFO  2017-02-02T19:10:16,140 org.eclipse.jetty.util.log] (main) Started SelectChannelConnector@0.0.0.0:8080 
[INFO  2017-02-02T19:10:17,643 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp359742806-16) Starting proxy at address: 0.0.0.0/0.0.0.0:8081 
[INFO  2017-02-02T19:10:17,685 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp359742806-16) Proxy listening with TCP transport 
[INFO  2017-02-02T19:10:17,831 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp359742806-16) Proxy started at address: /0:0:0:0:0:0:0:0:8081 
[INFO  2017-02-02T19:10:49,754 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp359742806-16) Shutting down proxy server gracefully 
[INFO  2017-02-02T19:10:49,755 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp359742806-16) Closing all channels (graceful) 
[INFO  2017-02-02T19:10:49,782 org.littleshoot.proxy.impl.ServerGroup] (qtp359742806-16) Shutting down server group event loops (graceful) 
[INFO  2017-02-02T19:10:51,991 org.littleshoot.proxy.impl.DefaultHttpProxyServer] (qtp359742806-16) Done shutting down proxy server

[andreyrusanov]

Any updates on it? I have the same problem(reproduced on CentOS7 and Ubuntu 16.04)

[andreyrusanov]

It seems I found the issue; It seems that Browsermob process should be started as a group of processes and then has to be terminated as a group. I will try to make a pull request.

[andreyrusanov]

PR is made(for Linux only since I don't have Windows machine to test a fix for Windows.

However, I have on assumption on possible fix(see PR description)

[michelebenolli]

Why is this marked as closed? The issue is currently present under Windows.

[darukavishnikov]

+1 still have the same issue under python

server.stop() ..\..\..\..\..\browsermobproxy\server.py:133: in stop if self.process.poll() is not None: E AttributeError: 'NoneType' object has no attribute 'poll'

Could it be windows permission case?

[dasshark]

This can be solved by modifying the stop() method and adding a windows handler like this:

import psutil, signal

parent_process = psutil.Process(self.process.pid)
child_process = parent_process.children(recursive=True)
for child in child_process:
  child.send_signal(signal.SIGTERM)
try:
  parent_process.send_signal(signal.SIGTERM)
except psutil.NoSuchProcess:
  pass

[alex-ber]

I've investigate this issue deeply. What happens, than when you start browsermobproxy server (I will reffer to it as _bmpdaemon, see https://medium.com/@arkadyt/setting-up-programmatic-aws-access-for-mfa-protected-federated-identities-3be22bcecf4b for details) it uses subprocess.Popen with command that contains bat-file. So, what happen on Windows Python process start's cmd that executes bat-file that starts Java process. When you're calling close() function on _bmpdaemon, it successfully kills cmd process. On Windows (and on Mac as per https://github.com/AutomatedTester/browsermob-proxy-py/issues/76) OS doesn't kills grand process automatically (I don't know what's happen on Linux, but I suspect it works their), that is close() function kills cmd process, but on Windows (and Mac) the Java process survives.

Inspired by @dasshark response above, I have witted the following helper function that I'm using calling instead of _bmpdaemon.stop().

import psutil, signal
from contextlib import suppress

def closeBmpDaemon(bmp_daemon):
    if bmp_daemon is not None and bmp_daemon.process is not None:
        childs_process = None
        try:
            cmd_process = psutil.Process(bmp_daemon.process.pid)
            childs_process = cmd_process.children(recursive=True)
            childs_process = [*childs_process, cmd_process]

            bmp_daemon.stop()
        finally:
            for child in childs_process:
                # we can't accidentally kill newly created process
                # we can kill only the process we have cached earlier
                # if process was already finished we will get NoSuchProcess
                # that we're just suppressing
                with suppress(psutil.NoSuchProcess):
                    child.send_signal(signal.SIGTERM)

EDIT I have added some comments and make some minor changes.

[alex-ber]

I have extracted this function (and more!) above into separate package. Here https://github.com/alex-ber/selenium-support/blob/main/alexber/seleniumsupport/_impl.py#L46 is source-code. Here https://pypi.org/project/selenium-support/ description of this package at PyPI Here https://alex-ber.medium.com/selenium-support-19330843c63a is detail description of this and another useful utility function for Selenium.

[KB00100100]

For Ubuntu 16.04, my personal solution:

1. add import signal
2. (line 112) small change

   self.process = subprocess.Popen(self.command,
                                       close_fds=True,
                                       preexec_fn=os.setsid,
                                       stdout=self.log_file,
                                       stderr=subprocess.STDOUT)

3. (line 141) replace self.process.kill() with os.killpg(self.process.pid,signal.SIGUSR1)

[alex-ber]

@KB00100100 hopefully my code is cross-platform. In about 2 month I'm going to test my code on Ubuntu. If my code will fail, I will try to use your code snippet. Thanks for sharing.