IOException too many open files

[GoogleCodeExporter]

What steps will reproduce the problem?
Call Track.getTags a lot of times

What is the expected output? What do you see instead?
At some moment, Track.getTags throws CallException with underlying IOException: 
too many open files.

What version of the product are you using? On what operating system?
Last version from SVN

Please provide any additional information below.
Seems that some connections are not closed

Original issue reported on code.google.com by sbos....@gmail.com on 4 Jul 2011 at 1:57

[GoogleCodeExporter]

Can you please provide the full stack trace at least down to getTags ?
Also please specify which cache implementation you're using.

Original comment by jannikov...@gmail.com on 5 Jul 2011 at 10:16

[GoogleCodeExporter]

Here's the call stack.

de.umass.lastfm.CallException: java.net.SocketException: Too many open files
    at de.umass.lastfm.Caller.call(Caller.java:234)
    at de.umass.lastfm.Caller.call(Caller.java:151)
    at de.umass.lastfm.Track.getTopTags(Track.java:236)
    at ru.ispras.ssgps.lastfm.LastFmCrawler.main(LastFmCrawler.java:216)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: java.net.SocketException: Too many open files
    at java.net.Socket.createImpl(Socket.java:397)
    at java.net.Socket.connect(Socket.java:527)
    at java.net.Socket.connect(Socket.java:478)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
    at sun.net.www.http.HttpClient.New(HttpClient.java:306)
    at sun.net.www.http.HttpClient.New(HttpClient.java:323)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911)
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1014)
    at de.umass.lastfm.Caller.call(Caller.java:195)
    ... 12 more

eeeh, I don't know about the cache, I didn't set anything related to cache, so 
I think it's set to default

Original comment by sbos....@gmail.com on 5 Jul 2011 at 10:56

[GoogleCodeExporter]

> I didn't set anything related to cache, so I think it's set to default
Indeed, I though this may have been be a problem with the cache but apparently 
you're running into limitations of your operating system. 
Try to increase the limit of concurrent tcp connections or the limit of 
available ephemeral ports on your machine.
I'm leaving this issue open for now, even though it does not appear to be a 
problem in the library. But in case you find a solution please share it here 
for others to find, should someone else run into the same problem.

PS: I could not reproduce this error with a couple of thousand calls on my 
machine (Win7), but please notice that Last.fm have restrictions on the number 
of calls you are allowed to make: http://www.last.fm/api/tos
PPS: your line numbers (e.g. Caller.java:234) don't seem to make sense, are you 
sure you're using the latest version or did you change the code inside this 
file locally?

Original comment by jannikov...@gmail.com on 5 Jul 2011 at 1:01

[GoogleCodeExporter]

Oh, yes, I'd changed Caller.java to ensure that all streams/connections are 
closed. But my changes didn't help. Maybe you as the author could check one 
more time for such things.

I can't understand why I reach the limit of open files if I run all requests 
sequentially in one thread? And API limitations should not be related to this 
error.

Currenly I have:

sbos@malta:~/Downloads$ ulimit -Hn
10000
sbos@malta:~/Downloads$ ulimit -Sn
10000

And I do a lot of requests, far more than 2000. 

Original comment by sbos....@gmail.com on 5 Jul 2011 at 1:29

Attachments:

• Caller.java