Feature request: Performance - scalability info & load tests

diegopau commented 10 years ago

I've been watching and trying different implementations of Pusher.

Slanger (https://github.com/stevegraham/slanger) seems to scale really well as you can launch as many nodes as you want and they will use Redis to share channel state. But Slanger is not in such active development as Poxa is, and some features are missing. I am very glad to have found Poxa.

It would be awesome to have some info on how well Poxa could scale, maybe some performance info about tests against a single Heroku dyno and updating the README with this.

Another thing that I find very useful (and I couldn't find anything like it) would be a simple tool that would load test Poxa, with just a few params like number of channels to generate, number of connections (users) per channel and the ratio in which this connetions are stablished.

The data I am more curious about is how many simultaneous users (open websockets connections) can a single heroku dyno or an Amazon t3.micro instance handle.

Thanks for sharing this implementation!

darrencauthon commented 10 years ago

I'd love to see some hard numbers on this implementation, as well as others. I've used this in apps and hit no limits, but these aps were limited in scope and didn't stretch the app.

I could be of some help, I think. I've gone as far as writing shell scripts to set up erlang, elixir, and Poxa, and I've used them as soon as last night. I assume testing it could be as simple as firing multiple threads as client apps to a Poxa (or any other) instance... but what are the metrics to gather?

To make this actionable, for me, I'd need specific metrics to try to capture. Any ideas?

edgurgel commented 10 years ago

I'm happy that you found Poxa :). If you wanna give it a try you can use the heroku button to deploy an instance. If you find any issue please warn us as it's a new thing that I just added.

So, Poxa is built on top of cowboy and it will have a similar performance. You can find a well known benchmark on top of it (http://www.techempower.com/benchmarks/) and many others just searching for "erlang cowboy benchmark (websocket)". Most of them are good numbers.

I'm right now making my pusher_client support presence channels and working on a "complete" suite of integration tests for Poxa. After that it should be fairly easy to run tons of connections using pusher_client and check whatever we want to. As @darrencauthon asked, what kind of metrics would be good to capture?

One thing that I didn't try (yet) is to support multiple nodes. To do this we would need to use gproc with its gproc_dist. It shouldn't be hard. The only problem that we would have is: we would need to be aware of net splits and things that just distributed systems have AND have a good test suite to keep the features we already have. This is something that I plan to work with soon (maybe 1 month?).

Also if anyone is interested on working on some feature (we have a TODO on README.md) feel free to open an issue and start discussing and working on it.

And also remember that you can always migrate to pusher.com (it's a nice service) if you are too big for Poxa. I think it's good to have an "escape plan" if you need :)

Any thoughts?

edgurgel commented 10 years ago

Also @darrencauthon, if you want to share the shell scripts you are using feel free to add to the wiki! :)

diegopau commented 10 years ago

Hi again, and sorry for the delay!

We are a small team in Spain and we are trying to save as much costs as we can! (most of our money is going away just on the taxes of creating a company in this country...). The big issue we have with Pusher is not about how big chattyhive can be inside Pusher, is how much it is costing us per user!

We might have build the app in a different way, but the way it is done right now we are using a Pusher connection per user online in our app. So with the 50$ plan of Pusher we can have just 500 online users which is not a lot. Some of this users might even not be really chatting, but just with the tab open, and still it count as a connection.

Anyway! the issue is not so much the traffic itself but the number of simultaneous open websocket connections that the Pusher implementation can handle. And I have a feeling that with a hosted solution even one Heroku dyno will handle more. In this site: http://veldstra.org/2013/10/25/heroku-websocket-performance-test.html they talk about one single dyno maintaining 6000 open websockets and a ratio of 160 new websockets per second. I guess we can't expect this numbers in a real app but they're looking good anyway.

I am not a developer myself, I am working on the design, planning the tasks to do, and taking some technical decisions; but my programming is rusted (not sure if this expresion sounds good in engish). The only back-end developer is learning Django as he develops. I will try myself to do some load tests and post the results.

I can't help with the programming but we will probably use poxa in our app, so i can help with the testing and reporting. I will also promote it so more people will know about it!

About the metrics, what i would like more to see is:

Param to choose how many channels to create
Param to choose how many users to connect per channel (but different users will join different channels)
Param to choose how many connections we want to create per second (how many users will join a channel per second)
Param to choose how many users we want to maintain connected at the same time. So if we choose 100, when the user 101 try to join a channel the user 1 is leaving its channel. And if we choose: user per channel * number of channels = all connections will be maintained.
Param to choose the size of the event the users will trigger to their channel each second. If blank then the users won't send any messages to the channel.

In the output it would be good to see:

How many connections were succesfully stablished, and how many failed.
Time consumed in the whole test
Mean MB transferred by one user, total MB transferred by all user, mean MB received by one user, total MB received by all users.
Latency: min, mean and max.

For more params to measure in websockets, its good to take a look into Thor: https://github.com/observing/thor#example

I don't know if its the kind of metrics you asked for! i hope you find it useful.

edgurgel commented 10 years ago

I mentioned using Pusher if your app becomes too big for Poxa, considering that your app is big enough and the income is sufficient to pay for it and have good support too. I'm not saying to just use Pusher now.

The websocket performance benchmark should be done on cowboy alone (the server that Poxa is built on top of). The only thing that Poxa does is the Pusher protocol on top of cowboy websockets.

I would prefer to create tons of connections, subscribe to channels and trigger events. Then collect how fast it was executed and how many subscribers got correctly the events. I think that's a reasonable way of collecting some performance data.

This is already possible using pusher_client. I can setup a small script to run these and after that we can tweak it for a more complete information. I'll post here an update as soon as I'm done with presence channels on pusher_client.

Also if in the end you decide to use Poxa, keep posting issues and report any problem you discover! :)

diegopau commented 10 years ago

Looking forward to this script Eduardo. If i understood well I will be able to choose how many connections are going to be created. Still my major interest is to know how many simultaneous websocket connections can be working at the same time.

We have already talked about it!. As soon as the whole Pusher API is implemented (I see two items with partial support and we are using everything except the Client Events) we will make Poxa our first implementation for pushing events and leave Pusher as a backup if we run into issues with Poxa. But we won't launch until October - November so we have some time for testing and reporting!!

I will help as much as I can.

darrencauthon commented 10 years ago

@edgurgel Here are the scripts I have so far: https://gist.github.com/darrencauthon/85688316e2f555bce70c

I'll add them to the Wiki (as well as the Poxa script) as soon as I worked out one issue I'm having with a Digital Ocean box. With the base install of Ubuntu 12 I have some missing dependency when I run mix deps.get... it just blows up. (sorry, didn't document the error out of frustration) On AWS or on DO with the the Ruby/Rails default image, everything works fine.

diegopau commented 10 years ago

@darrencauthon which AMI would you recommend to set this up on EC2? The scripts look like would be OK for a Ubuntu distribution, what about the Amazon Linux AMI based on CentOS?

Another good thing to know would be if Poxa is more CPU or more RAM demanding. This way it would be easier to choose the right EC2 instance!

darrencauthon commented 10 years ago

@diegopau I can't recommend one AMI over another, as I've only used the Ubuntu ones. :smile: I wrote those scripts for Ubuntu alone.

CPU and RAM stats would be good stats to gather, too. My educated guess is that this app would be more memory-intensive than CPU, but that's just a guess.

edgurgel commented 10 years ago

Yeah, I'm not sure, but I would suppose that memory would be the bigger resource needed. One nice thing is that Poxa does not have any database or service as dependency, so you can just bring a bigger machine without the need to migrate anything.

diegopau commented 10 years ago

Can I start a wiki about how to deploy it to Amazon EC2 with the Amazon Linux AMI?

The only issue that i had was that I installed the last version of Elixir (1.0.0-rc1) and I had to edit the mix.exs file to compile Poxa.

darrencauthon commented 10 years ago

Sure, I'd like to help with it, too.

My advice is to install the specifics needed by poxa. Don't jump ahead or behind... Check out the latest release tag (0.2.0?) and install elixir 0.14.0 and Erlang 17.0. If you deviate from dependencies, you may get caught in the wave of elixir updates and other dependencies. — Sent from Mailbox

On Sat, Sep 6, 2014 at 1:33 PM, Diego Ocampo Pérez notifications@github.com wrote:

Can I start a wiki about how to deploy it to Amazon EC2 with the Amazon Linux AMI?

The only issue that i had was that I installed the last version of Elixir (1.0.0-rc1) and I had to edit the mix.exs file to compile Poxa.

Reply to this email directly or view it on GitHub: https://github.com/edgurgel/poxa/issues/12#issuecomment-54723824

edgurgel commented 10 years ago

Soon we will hit 1.0.0 on Elixir and we will have less problems like this. As @darrencauthon said, it's better to stick with tagged releases.

Also, as soon as exrm fix this issue https://github.com/bitwalker/exrm/issues/55, poxa will have releases without the Erlang runtime and you will be able to run it just having Erlang 17.* and not even needing to install Elixir. So basically install Erlang 17, unzip the release and use it as it's without any other dependency.

diegopau commented 10 years ago

I've just wrote the wiki entry, feel free to modify it if something is wrong!

Its a wiki about setting up the last commit from the Poxa repo, not about how to use the releases.

edgurgel commented 10 years ago

One important thing that I recently added to README is that for production environment we should use consolidated protocols (it's an optimisation for protocols code) so you should

mix compile.protocols

And run the like this:

elixir -pa _build/prod/consolidated -S mix run --no-halt

That's exactly how I run it on Heroku.

Also notice that Poxa releases will always run using consolidated protocols.

I'll add this explanation to README.md

bitwalker commented 10 years ago

@edgurgel bitwalker/exrm#55 has been closed, sorry for the blocker!

edgurgel commented 10 years ago

@bitwalker great news! :)

edgurgel commented 10 years ago

@diegopau, I created a VERY simple script( https://gist.github.com/edgurgel/eb5da7f9d63aa7131b06) that:

connect N process
Subscribe to a channel named "channel"
Submit 10*N events

It prints out how much time it took to subscribe and how much time to receive the events acknowledgment. This acknowledgment may be a bottleneck if N is too high. It would be better to have a distributed benchmark with multiple machines requesting, but this would be a little bit more complicated.

Notice that if you use a large number on N, you will probably reach system limits of TCP connections. You may need to tweak ulimits.

You need to run it on poxa directory using:

MIX_ENV=test mix run connect.exs 400

If you do not give an argument, the default is N = 100.

It will run on localhost Poxa, you can change server location, app_key, secret etc on the script.

diegopau commented 10 years ago

@edgurgel, I am trying now to setup the 0.3.0 release on EC2.

The only problem I am having is with differences on parameteres to pass when launching poxa now that I am using the release and not the last repo commit.

Is still necessary the --no-halt param? what about -pa _build/prod/consolidated? where should I place the my_config.exs file so I can use it with the --config my_config.exs param?

Also: I understand that the benchmark script is to be used only with the repo code not with a release, am I right?

edgurgel commented 10 years ago

There's no need for --no-halt. If you untar the release and run:

$ ./bin/poxa
Usage: poxa {start|start_boot <file>|foreground|stop|restart|reboot|ping|rpc <m> <f> [<a>]|console|console_clean|console_boot <file>|attach|remote_console|upgrade}

You will see some possible commands. start, stop, restart etc.

So for example you can:

./bin/poxa start

then ./bin/poxa ping, if it replies with pong, the app is running

You can also bin/poxa remote_console and open a console on the running application for example.

The consolidated params is not needed as the release already comes with it by default.
I'll update README with instructions related to the config file. Thanks for the question.
The benchmark script can run with any Poxa server. You will just need to configure the params on the script. I added 'localhost', 'app_key', etc just to make it easier to use with a local Poxa. The script needs to be run on the repo code place just because of dependencies of the client libraries. You can even try with http://poxa.herokuapp.com, port 80, app_key, secret, app_id

edgurgel commented 10 years ago

Also another important thing, using releases you just need Erlang 17+ installed. There's no need for Elixir as it's already compiled to Erlang byte code.

edgurgel commented 10 years ago

I just releases Poxa 0.3.1 with an easy way to configure the release. I also released Poxa using Erlang 17.0 and 17.1. You can choose depending on your machine.

https://github.com/edgurgel/poxa/releases/tag/v0.3.1

README.md is updated. Any feedback is welcome.

edgurgel commented 10 years ago

As an example, I'm running Poxa 0.3.1-Erlang 17.1 release on this ec2 machine: http://ec2-54-68-208-73.us-west-2.compute.amazonaws.com/

diegopau commented 10 years ago

@edgurgel first of all thanks a lot for the script and the hints on how to run the releases. I updated the wikis with the info you gave. I tried to run the last release in an EC2 with only Erlang installed and its working perfectly.

About the script, I must be doing something wrong...

I am running the last realease (0.3.1) if i do a poxa ping I get a pong so everything seems fine! Its running on port 8080.

What i did to run the script is:

Inside my ec2 instance I cloned the repository, I wget the connect.exs script.
(Poxa 0.3.1 is running as daemon)
From the repo folder I placed the connect.exs, edited with my app_id, app_key and secret.
I run mix deps.get
I run the script with MIX_ENV=test mix run connect.exs 400

I get the following output:

[ec2-user@ip-172-31-43-157 poxa]$ MIX_ENV=test mix run connect.exs 400

=INFO REPORT==== 16-Sep-2014::20:23:09 === application: logger exited: stopped type: temporary * (Mix) Could not start application poxa: exited in: Poxa.start(:normal, []) * (EXIT) an exception was raised: \ (MatchError) no match of right hand side value: {:error, {{:shutdown, {:failed_to_start_child, :ranch_acceptors_sup, {{:badmatch, {:error, :eaddrinuse}}, [{:ranch_acceptors_sup, :init, 1, [file: 'src/ranch_acceptors_sup.erl', line: 30]}, {:supervisor, :init, 1, [file: 'supervisor.erl', line: 243]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 306]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 239]}]}}}, {:child, :undefined, {:ranch_listener_sup, :http}, {:ranch_listener_sup, :start_link, [:http, 100, :ranch_tcp, [port: 8080], :cowboyprotocol, [env: [dispatch: [{:, [], [{["ping"], [], Poxa.PingHandler, []}, {["console"], [], Poxa.Console.WSHandler, []}, {[], [], :cowboy_static, {:priv_file, :poxa, 'index.html'}}, {["static", :...], [], :cowboy_static, {:priv_dir, :poxa, 'static'}}, {["apps", :app_id, "events"], [], Poxa.EventHandler, []}, {["apps", :app_id, "channels"], [], Poxa.ChannelsHandler, []}, {["apps", :app_id, "channels", :channel_name], [], Poxa.ChannelsHandler, []}, {["apps", :app_id, "channels", :channel_name, "users"], [], Poxa.UsersHandler, []}, {["app", :app_key], [], Poxa.WebsocketHandler, []}]}]]]]}, :permanent, :infinity, :supervisor, [:ranch_listener_sup]}}} (poxa) lib/poxa.ex:23: Poxa.start/2 (kernel) application_master.erl:272: :application_master.start_it_old/4

bitwalker commented 10 years ago

The error message you are receiving appears to be because the port it's attempting to bind to is already in use, but poxa isn't handling that error condition, so it fails with a badmatch error. Perhaps you have another instance running, or an app already running on that port?

edgurgel commented 10 years ago

@bitwalker yeah, I just need to make the script ensure what I need to be started and change the command to:

MIX_ENV=test mix run --no-start connect.exs 200

poxa is being started but I don't need it to run at all

edgurgel commented 10 years ago

Ok @diegopau, I updated the gist. Now it won't try to run Poxa too.

To run just do:

MIX_ENV=test mix run --no-start connect.exs 100

diegopau commented 10 years ago

Still not working for me! (probably my fault this time):

$ MIX_ENV=test mix run --no-start connect.exs 100 connect.exs:33: warning: variable reason is unused connect.exs:33: warning: variable state is unused Running with n = 100 \ (EXIT from #PID<0.48.0>) {:badmatch, {:error, :closed}}

Poxa is running on localhost:8080.

And if poxa is running with the console param I can see this log when I launch de script:

20:18:22.990 [error] Unable to load crypto library. Failed with error:
":load_failed, Failed to load NIF library /home/ec2-user/poxa-0.3.1/lib/crypto-3.3/priv/lib/crypto: 'libcrypto.so.1.0.0: cannot open shared object file: No such file or directory'"
OpenSSL might not be installed on this system.

20:18:22.997 [error] The on_load function for module crypto returned {:error,
 {:load_failed,
  'Failed to load NIF library /home/ec2-user/poxa-0.3.1/lib/crypto-3.3/priv/lib/crypto: \'libcrypto.so.1.0.0: cannot open shared object file: No such file or directory\''}}

20:18:22.997 [error] Error in process <0.277.0> on node 'poxa@127.0.0.1' with exit value: {undef,[{crypto,hash,[sha,<<60 bytes>>],[]},{cowboy_websocket,websocket_handshake,3,[{file,"src/cowboy_websocket.erl"},{line,165}]},{cowboy_protocol,execute,4,[{file,"src/cowboy_protocol.erl"},{line,435}]}]}

20:18:22.997 [error] Error in process <0.271.0> on node 'poxa@127.0.0.1' with exit value: {undef,[{crypto,hash,[sha,<<60 bytes>>],[]},{cowboy_websocket,websocket_handshake,3,[{file,"src/cowboy_websocket.erl"},{line,165}]},{cowboy_protocol,execute,4,[{file,"src/cowboy_protocol.erl"},{line,435}]}]}

20:18:22.997 [error] Error in process <0.270.0> on node 'poxa@127.0.0.1' with exit value: {undef,[{crypto,hash,[sha,<<60 bytes>>],[]},{cowboy_websocket,websocket_handshake,3,[{file,"src/cowboy_websocket.erl"},{line,165}]},{cowboy_protocol,execute,4,[{file,"src/cowboy_protocol.erl"},{line,435}]}]}

20:18:22.998 [error] Error in process <0.269.0> on node 'poxa@127.0.0.1' with exit value: {undef,[{crypto,hash,[sha,<<60 bytes>>],[]},{cowboy_websocket,websocket_handshake,3,[{file,"src/cowboy_websocket.erl"},{line,165}]},{cowboy_protocol,execute,4,[{file,"src/cowboy_protocol.erl"},{line,435}]}]}

(and it goes on like that, i guess 100 times because N=100)

diegopau commented 10 years ago

I am also getting a crash on poxa when running it with the parameter "foreground":

$ /home/ec2-user/poxa-0.3.1/bin/poxa
Usage: poxa {start|start_boot <file>|foreground|stop|restart|reboot|ping|rpc <m> <f> [<a>]|console|console_clean|console_boot <file>|attach|remote_console|upgrade}

$ /home/ec2-user/poxa-0.3.1/bin/poxa foreground
Exec: /usr/local/lib/erlang/erts-6.0/bin/erlexec -noshell -noinput +Bd -boot /home/ec2-user/poxa-0.3.1/releases/0.3.1/poxa -mode embedded -config /home/ec2-user/poxa-0.3.1/releases/0.3.1/sys.config -args_file /home/ec2-user/poxa-0.3.1/releases/0.3.1/vm.args -- foreground
Root: /home/ec2-user/poxa-0.3.1
{error_logger,{{2014,9,17},{20,14,23}},"Unable to load crypto library. Failed with error:~n\"~p, ~s\"~nOpenSSL might not be installed on this system.~n",[load_failed,"Failed to load NIF library /home/ec2-user/poxa-0.3.1/lib/crypto-3.3/priv/lib/crypto: 'libcrypto.so.1.0.0: cannot open shared object file: No such file or directory'"]}
{error_logger,{{2014,9,17},{20,14,23}},crash_report,[[{initial_call,{supervisor,kernel,['Argument__1']}},{pid,<0.565.0>},{registered_name,[]},{error_info,{exit,{on_load_function_failed,crypto},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,330}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[kernel_sup,<0.543.0>]},{messages,[]},{links,[<0.544.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,128}],[]]}
{error_logger,{{2014,9,17},{20,14,23}},supervisor_report,[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{on_load_function_failed,crypto}},{offender,[{pid,undefined},{name,kernel_safe_sup},{mfargs,{supervisor,start_link,[{local,kernel_safe_sup},kernel,safe]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]}
{error_logger,{{2014,9,17},{20,14,24}},crash_report,[[{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.542.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,crypto}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,133}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}},{ancestors,[<0.541.0>]},{messages,[{'EXIT',<0.543.0>,normal}]},{links,[<0.541.0>,<0.540.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,376},{stack_size,27},{reductions,117}],[]]}
{error_logger,{{2014,9,17},{20,14,24}},std_info,[{application,kernel},{exited,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,crypto}}},{kernel,start,[normal,[]]}}},{type,permanent}]}
{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,crypto}}},{kernel,start,[normal,[]]}}}"}

Crash dump was written to: erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,crypto}}},{kernel,start,[normal,[]]}}})

edgurgel commented 10 years ago

I'm afraid the issue is related to the way OpenSSL is generated on different distros. Can you try to generate the release on the machine you are deploying? It should be trivial: https://github.com/edgurgel/poxa#release

The generated release will have a .tar.gz on rel/poxa/

I'll try to find a way to avoid this kind of issue on releases.

bitwalker commented 10 years ago

Could you elaborate a bit? Perhaps I can fix this upstream in exrm if it's general enough of a problem.

edgurgel commented 10 years ago

This error message:

{error_logger,{{2014,9,17},{20,14,23}},"Unable to load crypto library. Failed with error:~n\"~p, 
~s\"~nOpenSSL might not be installed on this system.~n",[load_failed,"Failed to load NIF library 
/home/ec2-user/poxa-0.3.1/lib/crypto-3.3/priv/lib/crypto: 'libcrypto.so.1.0.0: cannot open shared object 
file: No such file or directory'"]}

I had this issue when compiling from different Linux Distros. I'm not sure if it's exrm, relx fault. I would bet that's an issue on Erlang compiling process.

bitwalker commented 10 years ago

Hmm, that would make sense, except the error is a bit puzzling. It's not saying it's failing to load due to being the wrong architecture, but that it's failing because it simply can't find the Erlang libcrypto.so file, which I would expect it to find. Then again, based on the path it's looking in, I would expect the release to have been built with {include_erts, true} for that to be true, if not, then it appears to be looking in the wrong location.

edgurgel commented 10 years ago

I'm using this config file with {include_erts, false}: https://github.com/edgurgel/poxa/blob/master/rel/relx.config

edgurgel commented 10 years ago

@bitwalker, I'm adding crypto as dependent application. Do you think this could be the issue? https://github.com/edgurgel/poxa/blob/master/mix.exs#L14

bitwalker commented 10 years ago

@diegopau Does it only fail when running the foreground command, or does it fail with the same error for anything which results in the app being booted (i.e. foreground, console*, start*)?

@edgurgel No, if it was missing that would be a problem, so I think that's fine. Are you able to replicate this at all? Might help to have a test case for tracking down the bug/verifying a fix.

edgurgel commented 10 years ago

@bitwalker you can probably replicate if you have a linux distro that's not Ubuntu and grab the release here: https://github.com/edgurgel/poxa/releases/tag/v0.3.1

Then start and open on http://localhost:8080

put any app_key and secret and hit "Connect"

This should trigger the error.

bitwalker commented 10 years ago

Think CentOS would work? I'll give it a shot. I haven't dug in to poxa really, what's the best way to test this out?

edgurgel commented 10 years ago

I edited the last comment to easily trigger the error!

edgurgel commented 10 years ago

Yeah, I do think CentOS will work.

bitwalker commented 10 years ago

@edgurgel Looks like you are correct, it's due to either a missing, or incorrect version of OpenSSL. The libcrypto.so file is an OpenSSL shared library, and if it can't be loaded, the error in @diegopau's message is given.

diegopau commented 10 years ago

I am on a CentOS EC2 instance (the Amazon Linux AMI):

sudo yum install openssl
Loaded plugins: priorities, update-motd, upgrade-helper
amzn-main/latest                                         | 2.1 kB     00:00
amzn-updates/latest                                      | 2.3 kB     00:00
Package 1:openssl-1.0.1i-1.78.amzn1.x86_64 already installed and latest version

"Then start and open on http://localhost:8080 put any app_key and secret and hit "Connect" This should trigger the error."

Yes I just did that and it triggers the problem. Of course I have to start poxa with param console (instead of running as daemon) to see any log in the terminal!

@edgurgel I am trying to generate the release from the repo but I am not able to do it with: MIX_ENV=prod mix do deps.get, release should i replace release for a release number or something'?

´´´ $ ls app.json config elixir_buildpack.config mix.exs Procfile test _build connect.exs lib mix.lock README.md CHANGELOG.md deps LICENSE priv rel [ec2-user@ip-172-31-43-157 poxa]$ MIX_ENV=prod mix do deps.get, release All dependencies up to date \ (Mix) The task release could not be found ´´´

edgurgel commented 10 years ago

@diegopau, yeah you are right. I fixed on README. This behaviour changed on some older mix version.

The correct command is

MIX_ENV=prod mix do deps.get, compile, release

diegopau commented 10 years ago

$ MIX_ENV=prod mix do deps.get, compile, release                                     
All dependencies up to date
==> signaturex
Compiled lib/signaturex/crypto_helper.ex
Compiled lib/signaturex.ex
Generated signaturex.app
==> gproc (compile)
==> ranch (compile)
==> jsx (compile)
==> jsex
Compiled lib/jsex.ex
Generated jsex.app
==> conform
warning: the dependency conform requires Elixir ">= 0.15.1 and ~> 1.0.0" but you                                      are running on v0.15.1
Compiled src/conf_parse.erl
Compiled lib/conform/config.ex
Compiled lib/conform/parse.ex
Compiled lib/conform.ex
Compiled lib/conform/schema.ex
Compiled lib/conform/translate.ex
Compiled lib/conform/utils/utils.ex
Compiled lib/mix/tasks/conform.configure.ex
Compiled lib/mix/tasks/conform.effective.ex
Compiled lib/mix/tasks/conform.new.ex
Compiled lib/conform/utils/code.ex
Compiled lib/mix/tasks/conform.release.ex
Generated conform.app
==> exrm
warning: the dependency exrm requires Elixir ">= 0.15.1 and ~> 1.0.0" but you ar                                     e running on v0.15.1
Compiled lib/exrm/config.ex
Compiled lib/exrm/plugin.ex
Compiled lib/exrm/utils.ex
Compiled lib/mix/tasks/release.plugins.ex
Compiled lib/mix/tasks/release.clean.ex
Compiled lib/exrm/plugins/conform.ex
Compiled lib/mix/tasks/release.ex
Compiled lib/exrm/plugins/consolidation.ex
Compiled lib/exrm/appups.ex
Generated exrm.app
==> cowlib (compile)
==> cowboy (compile)
==> uuid
erl -make
==> poxa
Compiled lib/poxa/auth_signature.ex
Compiled lib/poxa.ex
Compiled lib/poxa/authentication.ex
Compiled lib/poxa/authorization_helper.ex
Compiled lib/poxa/channel.ex
Compiled lib/poxa/channels_handler.ex
Compiled lib/poxa/console/console.ex
Compiled lib/poxa/console/console_ws_handler.ex
Compiled lib/poxa/crypto_helper.ex
Compiled lib/poxa/ping_handler.ex
Compiled lib/poxa/event_handler.ex
Compiled lib/poxa/presence_channel.ex
Compiled lib/poxa/presence_subscription.ex
Compiled lib/poxa/pusher_event.ex
Compiled lib/poxa/supervisor.ex
Compiled lib/poxa/users_handler.ex
Compiled lib/poxa/subscription.ex
Compiled lib/poxa/websocket_handler.ex
Generated poxa.app
==> Generating relx configuration...
==> Merging custom relx configuration from rel/relx.config...
==> Generating sys.config...
==> Generating boot script...
==> Performing protocol consolidation...
==> Conform: Loading schema...
==> Conform: Schema succesfully loaded!
==> Conform: Generating escript..
==> Failed to build conform escript!
** (exit) 1
    lib/mix/tasks/conform.release.ex:74: Mix.Tasks.Conform.Release.build_escript                                     /0
    (mix) lib/mix/project.ex:184: Mix.Project.in_project/4
    (elixir) lib/file.ex:1045: File.cd!/2
    lib/mix/tasks/conform.release.ex:60: Mix.Tasks.Conform.Release.with_conform/                                     2
    lib/exrm/plugins/conform.ex:54: ReleaseManager.Plugin.Conform.before_release                                     /1
    lib/mix/tasks/release.ex:227: anonymous fn/2 in Mix.Tasks.Release.execute_be                                     fore_hooks/1
    (elixir) lib/enum.ex:1261: Enum."-reduce/3-lists^foldl/2-0-"/3
    lib/mix/tasks/release.ex:68: Mix.Tasks.Release.do_run/1

Should I install elixir 1.0.0 or is it just a recommendation?

edgurgel commented 10 years ago

Are you sure that elixir 1.0.0 is being used?

Can you check elixir --version

diegopau commented 10 years ago

Ok, I upgraded to elixir 1.0.0, generated the 0.3.2 release, run it with and tried the script, and it worked!

´´´ Running with n = 100 100 process connected and subscribed. It took 169.961 milliseconds 1000 Events received. It took 178.758 milliseconds ´´´

So the issue is on where the compiled releases from https://github.com/edgurgel/poxa/releases expect to find the openssl library no?

Have to go now, but I will soon experiment a little more with different params on the script :+1:

edgurgel commented 10 years ago

Yeah. I think that's the issue. I'll setup some VMs and see how hard it's to point to the right place where OpenSSL shared library is.

Good luck on the experiment :)

diegopau commented 10 years ago

EDITED: I reviewed it and it might still be some trouble with file descriptors, let me look at it a little more.

I did some more testing with the script.

I increased the ulimits to 95000 for ec2-user so I won't have any trouble with file descriptors.

No errors for N=400:

[ec2-user@ip-172-31-43-157 poxa]$ MIX_ENV=test mix run --no-start connect.exs 400
connect.exs:33: warning: variable reason is unused
connect.exs:33: warning: variable state is unused
Running with n = 400
400 process connected and subscribed. It took 591.205 milliseconds
4000 Events received. It took 461.986 milliseconds

If N=1000 however the script hangs, and I got this error messsages in the terminal which is running the "poxa console" command:

09:29:09.129 [error] Error in process <0.3758.0> on node 'poxa@127.0.0.1' with exit value: {[{reason,{badmatch,{error,emfile}}},{mfa,{'Elixir.Poxa.WebsocketHandler',websocket_info,3}},{stacktrace,[{uuid,get_node,0,[{file,"src/uuid.erl"},{line,302}]},{uuid,uuid1,2,[{file,"src/uuid.erl"},{line,88}]},{'Elixir.Poxa.WebsocketHandler'...

.....

09:29:09.138 [error] Error in process <0.3760.0> on node 'poxa@127.0.0.1' with exit value: {[{reason,{badmatch,{error,emfile}}},{mfa,{'Elixir.Poxa.WebsocketHandler',websocket_info,3}},{stacktrace,[{uuid,get_node,0,[{file,"src/uuid.erl"},{line,302}]},{uuid,uuid1,2,[{file,"src/uuid.erl"},{line,88}]},{'Elixir.Poxa.WebsocketHandler'...

....

09:29:09.157 [error] Ranch listener :http had connection process started with :cowboy_protocol:start_link/4 at #PID<0.3722.0> exit with reason: {[reason: {:badmatch, {:error, :emfile}}, mfa: {Poxa.WebsocketHandler, :websocket_info, 3}, stacktrace: [{:uuid, :get_node, 0, [file: 'src/uuid.erl', line: 302]}, {:uuid, :uuid1, 2, [file: 'src/uuid.erl', line: 88]}, {Poxa.WebsocketHandler, :generate_uuid, 0, [file: 'lib/poxa/websocket_handler.ex', line: 128]}, {Poxa.WebsocketHandler, :websocket_info, 3, [file: 'lib/poxa/websocket_handler.ex', line: 106]}, {:cowboy_websocket, :handler_call, 7, [file: 'src/cowboy_websocket.erl', line: 588]}, {:cowboy_protocol, :execute, 4, [file: 'src/cowboy_protocol.erl', line: 435]}], msg: :start, req: [socket: #Port<0.9547>, transport: :ranch_tcp, connection: :keepalive, pid: #PID<0.3722.0>, method: "GET", version: :"HTTP/1.1", peer: {{127, 0, 0, 1}, 42707}, host: "localhost", host_info: :undefined, port: 80, path: "/app/f073ebb6f5d1b918e59e", path_info: :undefined, qs: "protocol=7", qs_vals: [{"protocol", "7"}], bindings: [app_key: "f073ebb6f5d1b918e59e"], headers: [{"host", "localhost"}, {"connection", "Upgrade"}, {"origin", "ws://localhost"}, {"sec-websocket-version", "13"}, {"sec-websocket-key", "2Qizes+xgir7NhQ4eELnlg=="}, {"upgrade", "websocket"}], p_headers: [{"upgrade", ["websocket"]}, {"connection", ["upgrade"]}], cookies: :undefined, meta: [websocket_version: 13, websocket_compress: false], body_state: :waiting, buffer: "", multipart: :undefined, resp_compress: false, resp_state: :done, resp_headers: [], resp_body: "", onresponse: :undefined], state: nil], [{:cowboy_protocol, :execute, 4, [file: 'src/cowboy_protocol.erl', line: 435]}]}

09:29:09.159 [info]  Subscribing to channel channel

09:29:09.161 [error] Ranch listener :http had connection process started with :cowboy_protocol:start_link/4 at #PID<0.3724.0> exit with reason: {[reason: {:badmatch, {:error, :emfile}}, mfa: {Poxa.WebsocketHandler, :websocket_info, 3}, stacktrace: [{:uuid, :get_node, 0, [file: 'src/uuid.erl', line: 302]}, {:uuid, :uuid1, 2, [file: 'src/uuid.erl', line: 88]}, {Poxa.WebsocketHandler, :generate_uuid, 0, [file: 'lib/poxa/websocket_handler.ex', line: 128]}, {Poxa.WebsocketHandler, :websocket_info, 3, [file: 'lib/poxa/websocket_handler.ex', line: 106]}, {:cowboy_websocket, :handler_call, 7, [file: 'src/cowboy_websocket.erl', line: 588]}, {:cowboy_protocol, :execute, 4, [file: 'src/cowboy_protocol.erl', line: 435]}], msg: :start, req: [socket: #Port<0.9549>, transport: :ranch_tcp, connection: :keepalive, pid: #PID<0.3724.0>, method: "GET", version: :"HTTP/1.1", peer: {{127, 0, 0, 1}, 39476}, host: "localhost", host_info: :undefined, port: 80, path: "/app/f073ebb6f5d1b918e59e", path_info: :undefined, qs: "protocol=7", qs_vals: [{"protocol", "7"}], bindings: [app_key: "f073ebb6f5d1b918e59e"], headers: [{"host", "localhost"}, {"connection", "Upgrade"}, {"origin", "ws://localhost"}, {"sec-websocket-version", "13"}, {"sec-websocket-key", "i+O4V24XUq3B10ZeKztBNw=="}, {"upgrade", "websocket"}], p_headers: [{"upgrade", ["websocket"]}, {"connection", ["upgrade"]}], cookies: :undefined, meta: [websocket_version: 13, websocket_compress: false], body_state: :waiting, buffer: "", multipart: :undefined, resp_compress: false, resp_state: :done, resp_headers: [], resp_body: "", onresponse: :undefined], state: nil], [{:cowboy_protocol, :execute, 4, [file: 'src/cowboy_protocol.erl', line: 435]}]}

09:29:09.163 [error] Ranch listener :http had connection process started with :cowboy_protocol:start_link/4 at #PID<0.3758.0> exit with reason: {[reason: {:badmatch, {:error, :emfile}}, mfa: {Poxa.WebsocketHandler, :websocket_info, 3}, stacktrace: [{:uuid, :get_node, 0, [file: 'src/uuid.erl', line: 302]}, {:uuid, :uuid1, 2, [file: 'src/uuid.erl', line: 88]}, {Poxa.WebsocketHandler, :generate_uuid, 0, [file: 'lib/poxa/websocket_handler.ex', line: 128]}, {Poxa.WebsocketHandler, :websocket_info, 3, [file: 'lib/poxa/websocket_handler.ex', line: 106]}, {:cowboy_websocket, :handler_call, 7, [file: 'src/cowboy_websocket.erl', line: 588]}, {:cowboy_protocol, :execute, 4, [file: 'src/cowboy_protocol.erl', line: 435]}], msg: :start, req: [socket: #Port<0.9635>, transport: :ranch_tcp, connection: :keepalive, pid: #PID<0.3758.0>, method: "GET", version: :"HTTP/1.1", peer: {{127, 0, 0, 1}, 37934}, host: "localhost", host_info: :undefined, port: 80, path: "/app/f073ebb6f5d1b918e59e", path_info: :undefined, qs: "protocol=7", qs_vals: [{"protocol", "7"}], bindings: [app_key: "f073ebb6f5d1b918e59e"], headers: [{"host", "localhost"}, {"connection", "Upgrade"}, {"origin", "ws://localhost"}, {"sec-websocket-version", "13"}, {"sec-websocket-key", "FCped9sSxjxu8SjlQl9Rhw=="}, {"upgrade", "websocket"}], p_headers: [{"upgrade", ["websocket"]}, {"connection", ["upgrade"]}], cookies: :undefined, meta: [websocket_version: 13, websocket_compress: false], body_state: :waiting, buffer: "", multipart: :undefined, resp_compress: false, resp_state: :done, resp_headers: [], resp_body: "", onresponse: :undefined], state: nil], [{:cowboy_protocol, :execute, 4, [file: 'src/cowboy_protocol.erl', line: 435]}]}

09:29:09.165 [error] Ranch listener :http had connection process started with :cowboy_protocol:start_link/4 at #PID<0.3760.0> exit with reason: {[reason: {:badmatch, {:error, :emfile}}, mfa: {Poxa.WebsocketHandler, :websocket_info, 3}, stacktrace: [{:uuid, :get_node, 0, [file: 'src/uuid.erl', line: 302]}, {:uuid, :uuid1, 2, [file: 'src/uuid.erl', line: 88]}, {Poxa.WebsocketHandler, :generate_uuid, 0, [file: 'lib/poxa/websocket_handler.ex', line: 128]}, {Poxa.WebsocketHandler, :websocket_info, 3, [file: 'lib/poxa/websocket_handler.ex', line: 106]}, {:cowboy_websocket, :handler_call, 7, [file: 'src/cowboy_websocket.erl', line: 588]}, {:cowboy_protocol, :execute, 4, [file: 'src/cowboy_protocol.erl', line: 435]}], msg: :start, req: [socket: #Port<0.9637>, transport: :ranch_tcp, connection: :keepalive, pid: #PID<0.3760.0>, method: "GET", version: :"HTTP/1.1", peer: {{127, 0, 0, 1}, 59448}, host: "localhost", host_info: :undefined, port: 80, path: "/app/f073ebb6f5d1b918e59e", path_info: :undefined, qs: "protocol=7", qs_vals: [{"protocol", "7"}], bindings: [app_key: "f073ebb6f5d1b918e59e"], headers: [{"host", "localhost"}, {"connection", "Upgrade"}, {"origin", "ws://localhost"}, {"sec-websocket-version", "13"}, {"sec-websocket-key", "3qKotb3axsh4f83bmMT2tQ=="}, {"upgrade", "websocket"}], p_headers: [{"upgrade", ["websocket"]}, {"connection", ["upgrade"]}], cookies: :undefined, meta: [websocket_version: 13, websocket_compress: false], body_state: :waiting, buffer: "", multipart: :undefined, resp_compress: false, resp_state: :done, resp_headers: [], resp_body: "", onresponse: :undefined], state: nil], [{:cowboy_protocol, :execute, 4, [file: 'src/cowboy_protocol.erl', line: 435]}]}

09:29:09.174 [info]  Registering #PID<0.2911.0> to channel channel

.....

diegopau commented 10 years ago

@edgurgel I wasn't changing the file descriptors well. Now everything works perfectly.

Results are awesome in my opinion. With N up to 10000 everything runs fine.

]$ MIX_ENV=test mix run --no-start connect.exs 10000
connect.exs:33: warning: variable reason is unused
connect.exs:33: warning: variable state is unused
Running with n = 10000
10000 process connected and subscribed. It took 64237.699 milliseconds
100000 Events received. It took 91967.301 milliseconds

With some more than that I might be hiting some limits on the machine, cause it goes or very slow or it just hangs.

I also notice that there is a progresion, like 2_N is not taking 2_time_needed. Or even with the same N it needs more time everytime I try. Could it be that some of the resources are not being freed when I abourt the Poxa process? (I will do more tests next week, so please don't take this tests too seriously, I have to confirm all this!!):

$ MIX_ENV=test mix run --no-start connect.exs 5000
connect.exs:33: warning: variable reason is unused
connect.exs:33: warning: variable state is unused
Running with n = 5000
5000 process connected and subscribed. It took 14034.427 milliseconds
50000 Events received. It took 15215.677 milliseconds

$ MIX_ENV=test mix run --no-start connect.exs 10000
connect.exs:33: warning: variable reason is unused
connect.exs:33: warning: variable state is unused
Running with n = 10000
10000 process connected and subscribed. It took 64237.699 milliseconds
100000 Events received. It took 91967.301 milliseconds

$ MIX_ENV=test mix run --no-start connect.exs 12000
connect.exs:33: warning: variable reason is unused
connect.exs:33: warning: variable state is unused
Running with n = 12000
12000 process connected and subscribed. It took 102346.937 milliseconds

BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded
       (v)ersion (k)ill (D)b-tables (d)istribution
a

$ MIX_ENV=test mix run --no-start connect.exs 12000
connect.exs:33: warning: variable reason is unused
connect.exs:33: warning: variable state is unused
Running with n = 12000
12000 process connected and subscribed. It took 116290.948 milliseconds

Results are really good in any case for a EC2 micro instance :)

edgurgel commented 10 years ago

Good to know that the results are good enough for such a small instance :)

I would suppose that the problem is the way we collect the results. It waits an "ack" from each event. I can probably make it smarter. I think it's an issue with the benchmark (which is kinda naive) and not with Poxa itself. Also if you are generating the request from different machines you can get better results for higher N values. The ideal benchmark would be to run the benchmark on multiple machines to hit Poxa, but it would need a smarter approach.

@diegopau, do you think we can close this issue? I'll put the "benchmark" script on the wiki and we can open more issues as needed. What do you think?

edgurgel / poxa

Feature request: Performance - scalability info & load tests #12

The only issue that i had was that I installed the last version of Elixir (1.0.0-rc1) and I had to edit the mix.exs file to compile Poxa.