Shutdown agents after repl is disconnected

[paulbutcher]

Exiting lein repl with ^D leaves a zombie java process lying around. This can be replicated as follows:

$ lein --version
Leiningen 1.7.0 on Java 1.6.0_29 Java HotSpot(TM) 64-Bit Server VM
$ lein new zombie
Generating a project called zombie based on the 'default' template.
$ cd zombie/
$ ps | grep java
78329 ttys002    0:00.00 grep java
$ lein repl
Copying 1 file to /Users/paul/clj/zombie/lib
REPL started; server listening on localhost port 35587
user=> ^D
$ ps | grep java
78396 ttys002    0:04.06 /usr/bin/java -cp /Users/paul/clj/zombie/test:/Users/paul/clj/zombie/test-resources:/Users/paul/clj/zombie/src:/Users/paul/clj/zombie/classes:/Users/paul/clj/zombie/resources:/Users/paul/clj/zombie/lib/clojure-1.3.0.jar:/Users/paul/.lein/plugins/lein-eclipse-1.0.0.jar:/Users/paul/.lein/plugins/lein-newnew-0.2.4.jar:/Users/paul/.lein/plugins/lein-noir-1.2.1.jar:/Users/paul/.lein/plugins/org.cloudhoist-lein-pallet-new-0.1.1-SNAPSHOT.jar:/Users/paul/.lein/plugins/org.cloudhoist-pallet-lein-0.4.2-SNAPSHOT.jar -Dclojure.compile.path=/Users/paul/clj/zombie/classes -Dzombie.version=0.1.0-SNAPSHOT -Dclojure.debug=false clojure.main -e (do nil nil (do (clojure.core/ns leiningen.util.injected) (defn- compose-hooks [f1 f2] (fn [& args] (apply f2 f1 args))) (defn- join-hooks [original hooks] (reduce compose-hooks original hooks)) (defn- run-hooks [hook original args] (apply (join-hooks original (clojure.core/deref hook)) args)) (defn- prepare-for-hooks [v] (when-not (:robert.hooke/hook (meta (clojure.core/deref v))) (let [hook (atom ())] (alter-var-root v (fn [original] (with-meta (fn [& args] (run-hooks hook original args)) (assoc (meta original) :robert.hooke/hook hook :robert.hooke/original original))))))) (defn- add-unless-present [coll f] (if-not (some #{f} coll) (conj coll f) coll)) (defn add-hook "Add a hook function f to target-var. Hook functions are passed the\n  target function and all their arguments and must apply the target to\n  the args if they wish to continue execution." [target-var f] (prepare-for-hooks target-var) (swap! (:robert.hooke/hook (meta (clojure.core/deref target-var))) add-unless-present f)) (clojure.core/ns user)) (set! *warn-on-reflection* nil) (try (do (try (clojure.core/require (quote clojure.java.shell)) (clojure.core/require (quote clojure.java.browse)) (catch java.lang.Exception ___2837__auto__)) (set! clojure.core/*warn-on-reflection* false) (clojure.core/let [server__2838__auto__ (java.net.ServerSocket. 35587 0 (java.net.InetAddress/getByName "localhost")) acc__2839__auto__ (clojure.core/fn [s__2840__auto__] (clojure.core/let [ins__2841__auto__ (.getInputStream s__2840__auto__) outs__2842__auto__ (.getOutputStream s__2840__auto__) out-writer__2843__auto__ (java.io.OutputStreamWriter. outs__2842__auto__)] (clojure.core/doto (java.lang.Thread. (fn* [] (clojure.core/binding [clojure.core/*in* (clojure.core/-> ins__2841__auto__ java.io.InputStreamReader. clojure.lang.LineNumberingPushbackReader.) clojure.core/*out* out-writer__2843__auto__ clojure.core/*err* (java.io.PrintWriter. out-writer__2843__auto__) clojure.core/*warn-on-reflection* nil] (clojure.main/repl :init (fn* [] (clojure.core/let [is__2828__auto__ nil in__2829__auto__ (quote nil) mn__2830__auto__ (quote nil)] nil (clojure.core/when (clojure.core/and is__2828__auto__ (.exists (java.io.File. (clojure.core/str is__2828__auto__)))) (clojure.core/println (clojure.core/str "Warning: :repl-init-script is " "deprecated; use :repl-init.")) (clojure.core/load-file is__2828__auto__)) (clojure.core/when in__2829__auto__ (clojure.core/require in__2829__auto__)) (clojure.core/when mn__2830__auto__ (clojure.core/require mn__2830__auto__)) (clojure.core/in-ns (clojure.core/or in__2829__auto__ mn__2830__auto__ (quote user))))) :caught (clojure.core/fn [t__2831__auto__] (clojure.core/when-not (clojure.core/instance? java.net.SocketException t__2831__auto__) (clojure.main/repl-caught t__2831__auto__))) :read (clojure.core/fn [request-prompt__2832__auto__ request-exit__2833__auto__] (clojure.core/or ({:stream-end request-exit__2833__auto__, :line-start request-prompt__2832__auto__} (try (clojure.main/skip-whitespace clojure.core/*in*) (catch java.lang.Exception ___2834__auto__ :stream-end))) (clojure.core/let [input__2835__auto__ (clojure.core/read)] (clojure.main/skip-if-eol clojure.core/*in*) (if (clojure.core/= :leiningen.repl/exit input__2835__auto__) (do (.close clojure.core/*in*) request-exit__2833__auto__) input__2835__auto__)))))))) .start)))] (clojure.core/doto (java.lang.Thread. (fn* [] (clojure.core/when-not (.isClosed server__2838__auto__) (try (acc__2839__auto__ (.accept server__2838__auto__)) (catch java.net.SocketException e__2844__auto__ (.printStackTrace e__2844__auto__))) (recur)))) .start) (if false (clojure.main/repl) (do (clojure.core/when-not false (clojure.core/println "REPL started; server listening on" "localhost" "port" 35587)) (clojure.core/deref (clojure.core/promise)))))) (finally (clojure.core/when (clojure.core/and false (clojure.core/not= "1.5" (java.lang.System/getProperty "java.specification.version")) true) (clojure.core/shutdown-agents)))))
78400 ttys002    0:00.00 grep java

Exiting with ^C instead of ^D does not result in a zombie.

I've observed this on both OSX and Ubuntu.

[dakrone]

I'm seeing this intermittently also, with lein2.

[trptcolin]

Yuck, I was hoping maybe this was a lein1-only thing.

Maybe we could add shutdown hooks when launching subprocesses? http://stackoverflow.com/a/272728/495302

I'm kind of guessing wildly, though.

[technomancy]

Do these zombies stay forever, or only 60 seconds (after waiting for the agent thread pool to time out and spin down)?

I don't think shutdown hooks run until after the agent pool has shut down?

With lein1 we had to call shutdown-agents after the primary client had disconnected.

[paulbutcher]

Forever in my case at least (I noticed because my laptop was grindingly slow because of the 50 Java processes chewing up resources :(

[technomancy]

A workaround in 1.x is to add :shutdown-agents true to project.clj, but I'd like a better fix.

[technomancy]

Actually I can't reproduce this problem either on Leiningen 1 or 2; inside a project or outside. Can you provide more details about your system?

[paulbutcher]

My Mac is a pretty vanilla Lion:

pauls-macbook-pro:~ paul$ uname -a
Darwin pauls-macbook-pro.home 11.3.0 Darwin Kernel Version 11.3.0: Thu Jan 12 18:47:41 PST 2012; root:xnu-1699.24.23~1/RELEASE_X86_64 x86_64
pauls-macbook-pro:~ paul$ java -version
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-11D50b)
Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)

My Ubuntu VM is running Natty with the Sun VM:

paul@ubuntu:~$ uname -a
Linux ubuntu 2.6.35-32-generic #66-Ubuntu SMP Mon Feb 13 21:04:12 UTC 2012 i686 GNU/Linux
paul@ubuntu:~$ java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Client VM (build 20.1-b02, mixed mode, sharing)

I've checked with a few of my colleagues who are also running Ubuntu, and they don't see this behaviour. The difference seems to be that they're using OpenJDK instead of the Sun version.

[paulbutcher]

The :shutdown-agents true workaround doesn't work for me, I'm afraid :-(

[technomancy]

Turns out I can repro this on OpenJDK7, so it's not specific to Oracle's JDK.

[Raynes]

I can reproduce this too. It's really a dream killer.

[akandratovich]

I can reproduce it inside project folder with lein2:

andrew@andrew-u100 ~/dev/clojure/zombie $ java -version
java version "1.7.0_03"
Java(TM) SE Runtime Environment (build 1.7.0_03-b04)
Java HotSpot(TM) Client VM (build 22.1-b02, mixed mode)
andrew@andrew-u100 ~/dev/clojure/zombie $ ls -ahl
total 32K
drwxrwxr-x 5 andrew andrew 4.0K 2012-03-24 00:15 .
drwxrwxr-x 4 andrew andrew 4.0K 2012-03-24 00:14 ..
-rw-rw-r-- 1 andrew andrew   98 2012-03-24 00:14 .gitignore
-rw-rw-r-- 1 andrew andrew  265 2012-03-24 00:14 project.clj
-rw-rw-r-- 1 andrew andrew  218 2012-03-24 00:14 README.md
drwxrwxr-x 3 andrew andrew 4.0K 2012-03-24 00:14 src
drwxrwxr-x 3 andrew andrew 4.0K 2012-03-24 00:15 target
drwxrwxr-x 3 andrew andrew 4.0K 2012-03-24 00:14 test
andrew@andrew-u100 ~/dev/clojure/zombie $ ps aux | grep java
andrew    7732  0.0  0.0   4452   772 pts/0    S+   00:25   0:00 grep --colour=auto java
andrew@andrew-u100 ~/dev/clojure/zombie $ lein2 repl
#<Agent$Closeable$b14108b6@12d0297: {:ss #<ServerSocket ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=56326]>, :transport #<transport$bencode clojure.tools.nrepl.transport$bencode@46d7c4>, :greeting nil, :handler #<session$session$fn__409 clojure.tools.nrepl.middleware.session$session$fn__409@1b7b072>}>
Welcome to REPL-y!
Clojure 1.3.0
    Exit: Control+D or (exit) or (quit)
Commands: (help)
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
          (sourcery function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
Examples from clojuredocs.org:
          (clojuredocs name-here)
          (clojuredocs "ns-here" "name-here")
nil
user=> Bye for now!

andrew@andrew-u100 ~/dev/clojure/zombie $ ps aux | grep java
andrew    7755 80.8  5.2 387244 53804 pts/0    Sl   00:25   0:24 java -cp /home/andrew/dev/clojure/zombie/test:/home/andrew/dev/clojure/zombie/src:/home/andrew/dev/clojure/zombie/dev-resources:/home/andrew/dev/clojure/zombie/resources:/home/andrew/dev/clojure/zombie/target/classes:/home/andrew/.m2/repository/clojure-complete/clojure-complete/0.2.1/clojure-complete-0.2.1.jar:/home/andrew/.m2/repository/org/apache/httpcomponents/httpcore/4.1.2/httpcore-4.1.2.jar:/home/andrew/.m2/repository/org/apache/httpcomponents/httpclient/4.1.2/httpclient-4.1.2.jar:/home/andrew/.m2/repository/org/clojure/clojure/1.3.0/clojure-1.3.0.jar:/home/andrew/.m2/repository/cheshire/cheshire/2.0.2/cheshire-2.0.2.jar:/home/andrew/.m2/repository/commons-io/commons-io/1.4/commons-io-1.4.jar:/home/andrew/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/home/andrew/.m2/repository/clj-http/clj-http/0.2.1/clj-http-0.2.1.jar:/home/andrew/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.5/jackson-core-asl-1.8.5.jar:/home/andrew/.m2/repository/org/clojure/tools.nrepl/0.2.0-beta1/tools.nrepl-0.2.0-beta1.jar:/home/andrew/.m2/repository/org/thnetos/cd-client/0.3.3/cd-client-0.3.3.jar:/home/andrew/.m2/repository/commons-codec/commons-codec/1.5/commons-codec-1.5.jar:/home/andrew/.m2/repository/org/codehaus/jackson/jackson-smile/1.8.5/jackson-smile-1.8.5.jar -Dclojure.compile.path=/home/andrew/dev/clojure/zombie/target/classes -Dzombie.version=0.1.0-SNAPSHOT -Dclojure.debug=false clojure.main -e (do (do (require (quote clojure.tools.nrepl.server)) (require (quote complete.core))) nil (do (clojure.core/ns leiningen.core.injected) (defn- compose-hooks [f1 f2] (fn [& args] (apply f2 f1 args))) (defn- join-hooks [original hooks] (reduce compose-hooks original hooks)) (defn- run-hooks [hook original args] (apply (join-hooks original (clojure.core/deref hook)) args)) (defn- prepare-for-hooks [v] (when-not (:robert.hooke/hook (meta (clojure.core/deref v))) (let [hook (atom ())] (alter-var-root v (fn [original] (with-meta (fn [& args] (run-hooks hook original args)) (assoc (meta original) :robert.hooke/hook hook :robert.hooke/original original))))))) (defn- add-unless-present [coll f] (if-not (some #{f} coll) (conj coll f) coll)) (defn add-hook "Add a hook function f to target-var. Hook functions are passed the\n  target function and all their arguments and must apply the target to\n  the args if they wish to continue execution." [target-var f] (prepare-for-hooks target-var) (swap! (:robert.hooke/hook (meta (clojure.core/deref target-var))) add-unless-present f)) (clojure.core/ns user)) (set! *warn-on-reflection* nil) (do (clojure.tools.nrepl.server/start-server :port 0 :ack-port 48775)))
andrew    7801  0.0  0.0   4456   776 pts/0    S+   00:26   0:00 grep --colour=auto java
andrew@andrew-u100 ~/dev/clojure/zombie $ 

Also, in my case there is Agent.toString() in repl as you can see.

[trptcolin]

In IRC @technomancy, @Raynes, and I had talked about the approach in this latest patch (12e071a), but scoping it to the repl task. In order to do that, I think we'd have to either change the return value semantics of eval-in* or add a separate fn for lein repl to hook into the shutdown-hook-adding thing. Or maybe there's an easier way I'm not seeing at the moment.

[trptcolin]

Also, I thought a bit more about whether or not subprocesses we launch via eval-in-project should ever be allowed to live beyond the life of the Leiningen process spawning them, and I can't think of any reason. Thoughts on that are applicable here too!

[technomancy]

Oh, I thought the discussion in #leiningen about scoping it to the repl task was about shutting down all subprocesses when a task finishes. If it's about not letting them survive past the entire Leiningen process then I have no problem with it at all; full steam ahead, &c.

[trptcolin]

Ah, cool deal. So I've squashed this and put it in master. I'm assuming @Raynes's tweet last night means it worked ;)

All: let me know if this fixes things for you guys in lein2 master

This should be straightforward to backport to lein1 if we want.

[paulbutcher]

Can I vote for a fix in Lein 1 please? :-)

I've not yet made the switch to 2, and I don't believe that I'm alone...

[Raynes]

No way man, you silly lein 1 users aren't first class citizens anymore. Go away. ;)

[technomancy]

Yeah, I'm planning on releasing a 1.7.1 version with this fixed some time next week. In the mean time I think it doesn't repro on the 1.x branch using Java 1.6 for what it's worth.

[j1n3l0]

I know I'm probably not supposed to run lein2 repl in a lein1 project but when I do I get the following output:

$ lein2 version
Leiningen 2.0.0-SNAPSHOT on Java 1.6.0_23 OpenJDK Client VM

$ ps aux | grep java
ionyiah   8974  0.0  0.0   4204   760 pts/1    S+   16:34   0:00 grep --color=auto java

$ lein2 repl
#<Agent$Closeable$b14108b6@1d4d493: {:ss #<ServerSocket ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=37049]>, :transport #<transport$bencode clojure.tools.nrepl.transport$bencode@1536eec>, :greeting nil, :handler #<session$session$fn__405 clojure.tools.nrepl.middleware.session$session$fn__405@64160e>}>
Welcome to REPL-y!
Clojure 1.2.1
    Exit: Control+D or (exit) or (quit)
Commands: (help)
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
          (sourcery function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
Examples from clojuredocs.org:
          (clojuredocs name-here)
          (clojuredocs "ns-here" "name-here")
nil
user=> Bye for now!

$ ps aux | grep java
ionyiah   9040  0.0  0.0   4204   756 pts/1    S+   16:34   0:00 grep --color=auto java

$ 

Don't know if it should be the same issue as this but it looked quite similar except for the java process hanging around. Checked out sha d7a835efa9d34d8d8cadbd0a4dc7588e22538bd5.

[trptcolin]

Did you Ctrl-D to quit lein2 repl? If so, is there an issue, or are you just confirming that the fix worked? The process you're seeing in the output is the grep process itself, and ps aux | grep java | grep -v grep will show you non-grep processes.

[j1n3l0]

Yes, I had to use Ctrl-D to quit as neither (quit) nor (exit) would work. Those only work when I start lein2 repl in a directory without a project.clj file. Also I get this line in when the REPL starts:

#<Agent$Closeable$b14108b6@19ccba: {:ss #<ServerSocket ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=53726]>, :transport #<transport$bencode clojure.tools.nrepl.transport$bencode@1a3aa2c>, :greeting nil, :handler #<session$session$fn__405 clojure.tools.nrepl.middleware.session$session$fn__405@a09e41>}>

In summary:

• works in a clean directory
• does not work in a lein 1/2 project directory

[trptcolin]

OK, can you open a new issue for (quit) and (exit) not working, and/or be a little more explicit about what you mean by "works" and "does not work"? The only problem I'm seeing in your comments is the quit/exit thing you mention, and I'm assuming there's another problem you're referring to?

That line of output you see is normal, though it could probably be made more clear what it means.

Thanks!

[j1n3l0]

Ok, I created #479 for the (exit) or (quit) issue. However, if that line of output is expected, should I not see it every time I start a REPL? As I mentioned, I don't get it when I start an independent REPL session.

Thanks.

[trptcolin]

Point taken - it would be good to make that more consistent. Opened https://github.com/technomancy/leiningen/issues/480