Low socket limit

[stakach]

Windows defines a default of 64 asynchronous sockets per application as defined by FD_SETSIZE See: http://support.microsoft.com/kb/111855

This limits windows as a platform for running large scale applications such as eventmachine.

Could we please set FD_SETSIZE to a considerably large limit in the makefile, say 99999, to allow for larger scale applications. Currently it is quite useless.

[luislavena]

Can you be more specific? Which version of Ruby are you talking about? For 1.8.x we used to increase FD_SIZE to 256. Ruby 1.9 does not suffer from that.

https://github.com/oneclick/rubyinstaller/blob/master/config/ruby_installer.rb#L18-31

If you found that Ruby 1.9.2 or 1.9.3 suffered from that, perhaps your report is better suited to Ruby itself:

http://bugs.ruby-lang.org

Please confirm.

[stakach]

Ruby 1.9.3 suffers from it, only for asynchronous IO and in any libraries that use functions Ruby core 1.9 now avoids such as select(2).

So the FD_SIZE should still be set and much higher than 256 for any serious async application.

[stakach]

Basically libraries such as eventmachine require it to be set otherwise they are limited to 64 sockets which is pretty meaningless and considering I've been doing production deployments using EM and thin on windows it is pretty scary.

[rdp]

for 1.8.x it's set to 256 isn't it?

[jonforums]

@stakach:

1) any specific examples of "larger scale applications" on Windows other than your thin/EM deploys?

2) your "larger scale applications" don't require 64bit and aren't limited by our current 32bit build?

A bit off-topic, but I'm curious on if/how the following relate to what you're doing, and what you think of Greg's patch from 9 months ago?

http://bugs.ruby-lang.org/issues/4906

http://www.ruby-forum.com/topic/869239

[stakach]

Yes, however 256 is way too low.

I've been hacking at Eventmachine for the last 2 days as I have application that is using ~150 sockets now and is intended to scale well beyond 1000 before the end of the year.

Right now it is 3am here and I'm configuring a Linux VM to NAT with the windows host so I don't hit the 64 limit. I would prefer not to be doing this again in a couple of months after I've reached the 256 limit...

There are 16383 ephemeral ports on windows (ie the probable max an application will ever use) can we use that?

[stakach]

@jonforums http://bugs.ruby-lang.org/issues/4906 === https://github.com/eventmachine/eventmachine/commit/5f1b8d770721c10a7481d82eca18b563de87da3c

The 64 is the number of sockets, not 64bits.. I wish it was bits.

[stakach]

Of course I'm not entirely sure of ruby internals, if there is a windows file descriptor wrapper for every socket (not just a ruby facade) then the limiting number of sockets would be Windows limit of 2048 file descriptors: http://msdn.microsoft.com/en-us/library/6e3b887c(v=vs.71).aspx

At least that limit is somewhat configurable at run time.

[luislavena]

@stakach I can't find the issue reported by @rdp on Ruby-Core about FD_SETSIZE

I see the value of the request, however, please see this on MSDN:

http://msdn.microsoft.com/en-us/library/windows/desktop/ms739169(v=vs.85).aspx

The maximum number of sockets that a Windows Sockets application can use is not affected by the manifest constant FD_SETSIZE.

But, further reading:

This value defined in the Winsock2.h header file is used in constructing the FD_SET structures used with select function. The default value in Winsock2.h is 64. If an application is designed to be capable of working with more than 64 sockets using the select and WSAPoll functions

We can up that numbre to 32767, which seems to be the maximum supported by iMaxSockets

If you agree, I'll change 1.9 recipe so next build will handle that.

[jonforums]

@stakach hehe..yeh, I know the 64 number is the number of sockets.

I asked about 64bits because some say they need 64bit (more than 4GB, etc) for "larger scale apps" as 32bit won't do. Seems like 32bit is fine for what you're doing, but I was interested in more details.

[stakach]

@jonforums the larger scale applications we build are building automation systems: https://github.com/stakach/em_control We're currently deploying our first big rollout (disastrous as we'd never scaled higher than 64 on windows) and we've got many more coming up. In fact this first 150 devices is only a small part of the rollout.

We also make a digital signage system that is entirely HTML5 (management + playback). Local deployments of that usually run on windows as we have to do powerpoint conversions and offloading to windows is more infrastructure + cost people don't want to see. Some of those deployments will be nearing 64 displays very soon.

[stakach]

@luislavena 32767 sounds great!

[jonforums]

@stakach how very cool...cloning your repo...OT, could https://github.com/adoxa/ansicon be helpful for your colorization needs?

[stakach]

@jonforums you looking at my telnet hack? (I stopped using that) Anyway it's not quite ready for public release, I want to package it up somewhat like rails and have a project generator.

Otherwise everyone will have to clone the repo :)

[jonforums]

you looking at my telnet hack? (I stopped using that)

yes

...I want to package it up somewhat like rails and have a project generator.

just scanned your repo, but would a single distributable .exe containing your Ruby stuff be a deployment option for you? FWIW, Luis has adopted a clone of exerb project https://github.com/luislavena/exerb-mingw and he and another guy have made some updates for 1.9.3. I haven't had time to play with the mods yet, but exerb/1.8.7 has worked well for https://github.com/vertiginous/pik

[stakach]

It could be and probably something we'd consider when we get this running in the cloud, for peoples local deployments. I'll look into it further, both look like interesting projects!

[jonforums]

Good luck...no more from me as it's now OT to your issue and Luis has accepted the mod for the next release :)

Please feel free to drop an ANN if/when it makes sense to http://groups.google.com/group/rubyinstaller since you're using RubyInstaller. And I'd also drop an ANN on http://www.ruby-forum.com/forum/ruby since many Rubyist's don't realize all the cool things being done with Ruby on Windows!

[stakach]

Thanks mate, will do!

[rdp]

isn't the max number limited by msvcrt.dll? if so what's the max actually usable maybe we should use that number (of course, that number might vary based on whether you're on a server or standard windows box...)

[stakach]

That's just the c run time. Technically socket limits are purely memory based, except for this particular case where we want to use this particular function for async io.

Windows imposes limits on IIS connections and network shares however they are all higher level.

[rdp]

http://www.squid-cache.org/mail-archive/squid-users/200601/0665.html is where I "guess" I got my info.

[stakach]

That limit does exist. I mention it in my 6th comment. It relates to open files not sockets.

[rdp]

Interesting. (oh BTW ruby core in 1.9 uses select(2) if you call IO.select doesn't it?) Anyway, with Ruby sockets per se (like require 'socket'; TCPSocket.new) I think it creates "a windows file descriptor wrapper for every socket". I don't know if EM does or not. It'd be interesting to see what the theoretical limit "actually is" with EM or the like, given a high FD_SETSIZE. The only drawback to a large FD_SETSIZE is that the whole set is put on the stack, but since 1.9 has individual threads anyway now, it probably wouldn't matter if it were large I guess....

[stakach]

Yeah it would be good to know, if I start getting near the 2048 boundary I'd be playing it cautious. Also if it is file descriptors you have to manually increase the limit from 512 see https://github.com/stakach/eventmachine/commit/7d55c9fb59badba8b152b232e77ce005279c5536

[luislavena]

@stakach I've committed a fix, and will be available in the next patchlevel release of Ruby 1.9.3.

In the meantime, if you want to test, you can build locally by cloning RubyInstaller repository and doing rake ruby19 to compile.

Really cool the em_control project. Will be interesting when you're done if you can do a formal announcement in RubyInstaller mailing list:

http://groups.google.com/group/rubyinstaller/

Cheers.

[stakach]

libffi didn't compile properly. Giving it another go. Awesome how everything is configured.

[stakach]

It was a path with a space. All good now

[stakach]

Ok the FD_SETSIZE size we set clashed with event machine (which defines a smaller number) causing a seg fault. Reducing FD_SETSIZE to equal eventmachines 1024 worked.

So the next release of eventmachine and ruby installer must always match!

Previously eventmachine was higher than ruby and it created so much weirdness! I'm creating an issue to let the eventmachine guys know.

[stakach]

Also, thanks everyone for your help! Couldn't have got this working without you!

[luislavena]

@stakach please link the issue of EventMachine with this using oneclick/rubyinstaller#104 so we are aware.

Cheers.

[stakach]

Ok, the eventmachine one is: eventmachine/eventmachine#303

[stakach]

Hi @luislavena looks like Windows will never support more than 2048 (technically 2045 thanks to STD streams) and the limits on file handles.

So we might as well drop the FD_SETSIZE before your next release to 2048 to save memory - the change won't effect eventmachine

[luislavena]

Where is the root of this assumption? Can you provide a link about this?

Windows is not limited to 2k sockets, so would love to know where this coming from.

Sorry for top posting. Sent from mobile. On Aug 12, 2012 8:43 PM, "Stephen von Takach" notifications@github.com wrote:

Hi @luislavena https://github.com/luislavena looks like Windows will never support more than 2048 (technically 2045 thanks to STD streams) and the limits on file handles.

So we might as well drop the FD_SETSIZE before your next release to 2048 to save memory - the change won't effect eventmachine

— Reply to this email directly or view it on GitHubhttps://github.com/oneclick/rubyinstaller/issues/104#issuecomment-7681074.

[stakach]

It is a Microsoft limit on the standard c library: http://msdn.microsoft.com/en-us/library/6e3b887c%28vs.71%29.aspx and a windows file descriptor is allocated for every socket: See the comment by @rdp

It's easy to test:

Server

require 'rubygems'
require 'eventmachine'

module EchoServer

    @@connected_clients = 0

def post_init
    @@connected_clients += 1
    p "there are now #{@@connected_clients} connected"
end
  def receive_data data
    send_data ">>>you sent: #{data}"
    close_connection if data =~ /quit/i
  end
end

EventMachine::run {
  EventMachine::start_server "127.0.0.1", 8081, EchoServer
}

Client

require 'rubygems'
require 'eventmachine'

class Echo < EventMachine::Connection
  def post_init
    @timer = EventMachine::PeriodicTimer.new(5) do
  send_data 'Hello'
end
 send_data 'Hello'
  end

  def receive_data(data)
    p 'recieved:' + data
  end

def unbind
    p ' connection totally closed'
    @timer.cancel
  end
end

EventMachine.run {
    (1..3000).each do |h|
        EventMachine.connect '127.0.0.1', 8081, Echo
    end
}

[stakach]

@luislavena ping

[luislavena]

Sorry, I'm with limited Internet access right now.

Will check tomorrow morning on my computer.

Sorry for top posting. Sent from mobile. On Aug 12, 2012 9:48 PM, "Stephen von Takach" notifications@github.com wrote:

@luislavena https://github.com/luislavena ping

— Reply to this email directly or view it on GitHubhttps://github.com/oneclick/rubyinstaller/issues/104#issuecomment-7681559.

[luislavena]

@stakach I haven't tested this example yet, but I don't think Ruby is using open methods on the sockets mapped to fd's, or it does?

[stakach]

I have a feeling it is the file descriptor limit being hit. Other wise someone would have had to have set WSAStartup's max sockets at 2045

[Azolo]

Running the example I get this error on client.rb at around 1000(+/-50) connections.

unable to create new socket: Too many open files (EventMachine::ConnectionError)

Here is the BasicSocket initialization

It looks like rb_update_max_fd is being called, but I have no idea what that actually does so it's just a hunch.

[shirosaki]

Ruby associates windows socket to a C run-time file descriptor by _open_osfhandle().

https://github.com/ruby/ruby/blob/trunk/win32/win32.c#L3486 http://msdn.microsoft.com/en-us/library/bdts1c9x(v=vs.71).aspx

So that socket size seems limited to file descriptor max (2048). If ruby did not use _open_osfhandle() for socket, socket size might not limited to 2048.

[luislavena]

Thank you @shirosaki

So we conclude that having FD_SETSIZE of 16K is pointless.

Sometimes I wish Ruby did IO/socket implementation differently.

Anyhow, going to change the limit, again.

Thanks @stakach

[cubiic]

Hi Everyone, and sorry for the headache here, but I'm facing almost the same problem is my production server, I would really appreciate any help regarding this issue, I'm running my system with Ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux] , at OS Ubuntu 12.04.1 LTS and em-http-request (1.0.3).

I have a ruby processes that consume msgs from a RabbitMQ queue using amqp gem, something like this :

require "bundler/setup"
require "amqp"
require 'eventmachine'
require 'em-http'

AMQP.start(:host => $AMQP_URL) do |connection|
  @channel ||= AMQP::Channel.new(connection)
  @queue   ||= @channel.queue("results")

  puts " [*] Waiting for messages. "

  @queue.subscribe do |body|    
    http = EventMachine::HttpRequest.new(URL).post :body => body          
    http.callback { # do something   }        
    http.errback  { $LOG.error "[errorback] -> #{http.error}" }
  end
end

Now the URL is slow, and the queue has so much messages ( > 30K ), I got this error in the log: [errorback] -> unable to create new socket: Too many open files

Any help would be highly appreciated since I've been trying all my time figuring out how to solve it but with no results at all.

Thanks in advance

[shirosaki]

@cubiic have you tried to increase the open file descriptor maximum?

such as: $ ulimit -n 100000 http://serverfault.com/questions/235356/open-file-descriptor-limits-conf-setting-isnt-read-by-ulimit-even-when-pam-limi

[cubiic]

@shirosaki I really appreciate your welling to help, but guide me more here since I'm not an expert with this, please check out what I've been trying to do :

https://gist.github.com/cubiic/f7c8b1ca3f56ac9fd73b

As you can see I've been trying to do that unsuccessfully,

Thanks in advance

[jonforums]

@cubiic while interesting, this linux issue has nothing to do with rubyinstaller or ruby on windows.

Worse, the documentation for any fix will be buried in this thread and very difficult to for a linux user to find. That's a shame, and a waste of your and Shirosaki-san's efforts. The EM list, or the ruby-talk list (or even stackoverflow) are much better places to work/document the problem.

[cubiic]

Sorry man , but I though the problem is relevant based on the return error i'm getting, if you don't feel so I can just close it, but I found "unable to create new socket: Too many open files" error relevant by googling it

[jonforums]

It's related, but irrelevant on this project's issue list.

But don't miss my main point: I don't want the doco for a fix getting "buried" here. Switch the conversation (with an fyi link back here if you like) to a more linux visible place, like ruby-talk so we linux users can better help/benefit.

[shirosaki]

@cubiic $ ulimit -S -n 100000 would work for non-root user. http://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user

If more guide is needed, please switch the place as Jon stated.

[cubiic]

Thanks a lot for the help here,