java.lang.OutOfMemoryError: unable to create new native thread

[stania1]

Hi,

I took v0.2.0 for a spin in our GoCD server (which has 82 pipelines) in total - it ran for a while and then crashes with the following.

Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:714)
    at com.mashape.unirest.http.options.Options.refresh(Options.java:80)
    at com.mashape.unirest.http.Unirest.setTimeouts(Unirest.java:94)
    at in.ashwanthkumar.gocd.client.MinimalisticGoClient.invokeGET(MinimalisticGoClient.java:180)
    at in.ashwanthkumar.gocd.client.MinimalisticGoClient.getJSON(MinimalisticGoClient.java:159)
    at in.ashwanthkumar.gocd.client.MinimalisticGoClient.upstreamDependencies(MinimalisticGoClient.java:68)
    at in.ashwanthkumar.gocd.artifacts.Janitor$4$1.apply(Janitor.java:177)
    at in.ashwanthkumar.gocd.artifacts.Janitor$4$1.apply(Janitor.java:174)
    at in.ashwanthkumar.utils.collections.Lists.map(Lists.java:32)
    at in.ashwanthkumar.gocd.artifacts.Janitor$4.apply(Janitor.java:174)
    at in.ashwanthkumar.gocd.artifacts.Janitor$4.apply(Janitor.java:169)
    at in.ashwanthkumar.utils.collections.Lists.map(Lists.java:32)
    at in.ashwanthkumar.gocd.artifacts.Janitor.computeWhiteList(Janitor.java:169)
    at in.ashwanthkumar.gocd.artifacts.Janitor.run(Janitor.java:83)
    at in.ashwanthkumar.gocd.artifacts.Janitor.main(Janitor.java:63)

The process seems to have created 800+ threads before it crashed. I could look into the OS limits and configuration around threads, but is this expected behavior?

I haven't dug into the code to get more context - any pointers would be appreciated.

Thanks.

[ashwanthkumar]

:scream: 800 threads? I'll take a look at it. Meanwhile it would also help if you can check on some of these things for me.

• Do you by any chance pass any xmx values to the jvm while running the janitor? If so how much?
• can you also share the heap dump file in private if possible. That might help debug further. Also include the configuration used.

[ashwanthkumar]

One more thing - is it possible to get a jstack output of the process after it ran for a while?

[ashwanthkumar]

@stania1 I did some digging up, looks like you're right. There's a chance we might be creating too many threads internally. It's done by the HTTPClient library that I'm using (Unirest). I've added a 0.2.1-RC1 release with this fix. Can you please try with that version and let me know if you still have the problem?

[stania1]

0.2.1-RC1 release fixed it for me - the number of threads were constant at 19 and the process completed successfully. Thanks @ashwanthkumar for looking into this so quickly :)

[javase]

unirestInstance.shutDown();