search

losvedir / transit-lang-cmp

Programming language comparison by reimplementing the same transit data app
MIT License
426 stars 31 forks source link

Use plug #13

Closed josevalim closed 1 year ago

josevalim commented 2 years ago

I decided to use Plug for a more apple to apple comparison with other languages. Even though Phoenix overhead is low, it stills supports static assets, performs content negotiation, and more. But perhaps more importantly, it makes the application size between languages more comparable. :)

Something else that was also impacting the results is that you set the logger level to :warning in the console backend and not on config :logger, level: :warning. This means the logger machinery is invoked and the results are discarded just before writing to console. The logger machinery should be fast, but it adds a bit. In any case, this version does not add Plug.Logger, so it is no longer a concern.

Finally, I also added a configuration so Cowboy keeps connections alive for longer. IIRC it defaults to 100 requests, which is a sane default for actual apps, but probably too low for benchmarks.

If you want to squeeze a bit more, I would also suggest moving to persistent_term. If the data never or rarely changes (such as hourly), it will be cleaner and more performant than ETS.

PS: I could not download the transit data (I get a 403) but I am hoping the code still works. I did benchmark with an empty JSON response though and this version is roughly twice faster.

losvedir commented 2 years ago

Wow @josevalim! I'm 🤩 .

Thanks for the great ideas here. I just pushed up a commit fixing the logger level issue. I'll do a separate one for the cowboy keep-alive. I'm interested in how these parts individually contribute. Finally, I'll probably do another change for Plug, though I also want to look into a bit what the other languages are doing. .NET for example uses ASP.net, Scala is using Cask, and rust is using Axum. Only Go is using framework-less standard library, so I'm thinking maybe another approach would be to make sure Go is doing the things that a real webapp should be doing.

But I got tons of PRs and issues on this repo (thanks everyone!), and I just started a new job, so it's kind of slow going. I will get to it eventually, though.

losvedir commented 1 year ago

Okay, took a while but I finally had a chance to re-benchmark all my apps (since I updated to macOS Ventura), and then rebase this branch on the latest main, and re-benchmark Elixir again.

The performance was actually roughly comparable to the Phoenix version (with a slight edge towards Plug for the highest response rate one), which I think is a strong credit to Phoenix! But I figured I should still merge this since, as you say, it makes the code more apples to apples with the other languages.

I'll definitely look at persistent_term. I didn't think to use it since I never had to at work, but it actually makes a lot of sense for this use case (and I'll have to mention it to the folks who manage the actual repo that this one is somewhat based on).

I could not download the transit data (I get a 403) but I am hoping the code still works.

Sigh, yeah the MBTA blocks traffic from a lot of countries for some misguided reason. 😢

I did benchmark with an empty JSON response though and this version is roughly twice faster.

Yeah, I didn't realize ahead of time, but the benchmark ends up being heavily influenced simply by the JSON encoding step. I got a 2x bump switching from Jason to Jsonrs, which surprised me since I've always used Jason (well, after the days of Poison anyway) without thinking twice.