search

omegahat / RCurl

Other
29 stars 14 forks source link

getURLAsynchronous file descriptor exhaustion #8

Open asieira opened 10 years ago

asieira commented 10 years ago

I am running a large number (32k) number of GET requests using getURLAsynchronous, 120 URLs at a time. This is what the call looks like within the loop:

ret = getURLAsynchronous(cfg$access.point, httpheader=c("X-Api-Key"=cfg$key), perform=16, verbose=FALSE, header=FALSE, binary=F, ssl.verifypeer=FALSE)
gc(FALSE)

After this batch is executed, showConnections shows me no entries other than stdin, stdout and stderr.

Still, when I try to use mclapply to process all the combined responses in parallel I get the following error:

Error in mcfork() (from util.R#160) : unable to create a pipe
Calls: <redacted> ... unlist -> lapply -> mclapply -> lapply -> FUN -> mcfork

If I run the same mclapply without calling getURLAsynchronous first, with test data of the same size, no such error happens.

I tried searching the RCurl documentation for any steps I could include in my code to close connections and/or release resources, and was unable to find any. So I'm assuming that would happen when the handles are garbage collected, hence the explicit call to gc() after each step.

Am I missing something here?

asieira commented 10 years ago

BTW, I saw references on the documentation to a function called curlGlobalCleanup but that simply is not found by R when I try to execute it.

Also, important to mention I am running R 3.0.3 64 bits on Mac OS X 10.9.2. sessionInfo tells me I am running RCurl_1.95-4.1.

asieira commented 10 years ago

The handle and multi handle objects seem to be missing finalizers (as per http://cran.r-project.org/doc/manuals/R-exts.html#External-pointers-and-weak-references) that call the libcurl cleanup functions to release resources.

asieira commented 10 years ago

Found a workaround. If I use forbid.reuse=T in the call to getURLAsynchronous the problem does not happen.

This however confirms that proper cleanup of the handles (which would cause the connections to be closed) would solve it in the general case and lead to optimal performance.