Botan vs OpenSSL?

[dariusc93]

I would like to know how will this be compared to OpenSSL? Like does botan depend on openssl, and if not how secured is it vs openssl? How well does it perform and how is the compatibility between the two?

[etcimon]

Botan has a lot of features that OpenSSL doesn't. It doesn't depend on OpenSSL, and the two have completely different API. I think Botan and OpenSSL are popular enough that you can find the answers through a google search. Keep in mind that this version is from Aug. 2014

For benchmarks, keep in mind that native instructions are usually faster, Botan has lots of native instruction cryptos so it mind end up faster on certain processors e.g. for AES-NI and SSSE3

http://en.wikipedia.org/wiki/Comparison_of_TLS_implementations http://stackoverflow.com/questions/4613739/botan-vs-openssl-vs-crypto-for-cryptography

[etcimon]

how secured is it vs openssl?

I personally believe Botan is incredibly more secure than OpenSSL.

OpenSSL's heartbleed was national news and may have been exploited for years, the costs for identity theft would still be piling up today. I believe Botan is less widespread and harder to crack due to the more secure allocator strategies, so the opportunities are far greater for black hat security analysts to target OpenSSL, thus there's also security advantages for Botan from keeping a low profile.

[dariusc93]

Question, how stable is botan right now and also does it have support for RSA, ECC, and DSA?

[etcimon]

Question, how stable is botan right now and also does it have support for RSA, ECC, and DSA?

It's tested with the same tests as the C++ library, which is over 10 years old and used widely by the industry top specialists. Yes, it does have RSA, ECC, DSA, and is stable for all of the ciphersuites listed here

Also, there is a vibe.d implementation of Botan TLS connections that I am currently developing. This is coming along with HTTP/2 support. It has built-in support for PSK and SRP as well.

[dariusc93]

I was referring to your implementation of Botan in D, but if it is stable, I will be looking into using it very shortly

[etcimon]

I was referring to your implementation of Botan in D, but if it is stable, I will be looking into using it very shortly

Yes, you can look at the test results here: https://travis-ci.org/etcimon/botan#L124

There are hundreds of thousands of tests, thousands of validity checks for each algorithm, running on each implementation (SSE2, x86, x64, etc).

[etcimon]

I've added a small example for hashing:

https://github.com/etcimon/botan/blob/38a73337dd0323405fae63a91326ed6b0f23aded/examples/sha1/source/sha1.d

It compiles into a 965kb executable with DMD release.

[dariusc93]

Should the directory be sha2? ;)

[etcimon]

XD

[dariusc93]

Also I get "source/sha1.d(14): Error: no property 'unique' for type 'botan.hash.hash.HashFunction'"

[etcimon]

Also I get "source/sha1.d(14): Error: no property 'unique' for type 'botan.hash.hash.HashFunction'"

I added this 10 minutes ago in memutils, you can copy it over manually:

auto unique(T)(T obj) {
    return Unique!T(obj);
}

[dariusc93]

Done. So far it looks pretty good to use. I also got this message when i ran it "E: Could not mlock 65536 bytes" but it didnt error out or anything.

[etcimon]

Done. So far it looks pretty good to use. I also got this message when i ran it "E: Could not mlock 65536 bytes" but it didnt error out or anything.

That's memutils, trying to lock memory. It's used in the SafeVector implementation, prevents your hashes/keys from touching swap/HDD in case you hibernated or needed memory. It's a safety precaution but not that important if you control the physical device. Not sure why debian fails to do it, red hat handles it fine. It falls back to regular Zeroize in any case.

If you need more hashes, you will have to include the "versions" in the dub.json, and find the standard name for the lookup. That'll be in botan.engine.core_engine.d btw

[dariusc93]

I would totally stick with RHEL/Centos if I dont get errors when compiling on debian/ubuntu and trying to execute on there (With vibe that is). On my server, I have enough memory to where I dont need swap at all (16GB to 256GB), and some servers as small as 1GB wont need swap for its general purpose.

[etcimon]

I have enough memory to where I dont need swap at all (1

In that case you can remove the "SecurePool" in the memutils versions and it'll silence it

I should find a decent way to disable it..

[etcimon]

Also, in memutils there's a few utilities related to debugging the memory (it checks for leaks, corruption and reusage)

Don't benchmark anything unless you have a "DisableDebugger" version set =)

[dariusc93]

Should I set it in memutils or in my app?

[etcimon]

You can add DisableDebugger to your app's dub.json and dub will transmit the version to memutils when compiling it

[dariusc93]

Also do you think that memutil will somehow solve the issue i been having with memory leaks?

[etcimon]

Also do you think that memutil will somehow solve the issue i been having with memory leaks?

I haven't had the chance to reproduce it. Nothing seems to leak from every way I've tested, so maybe you could setup a server (vm?) with the configurations and send me access, I'd debug it from there

[dariusc93]

That would work. How do you wish for me to send you the info?

[etcimon]

email: etcimon at gmail

[dariusc93]

Ok thanks, ill be sending an email shortly with the info. Do you want me to already have the application running?

[dariusc93]

Email sent fyi. http_server is currently running

[dariusc93]

I notice you were active on the server. Found out anything?

[etcimon]

I found the fix and created a pull request here. You could try your test again if you like

[dariusc93]

Thanks, I did test it and it does clear up which is great.

[dariusc93]

Hmm I think I spoke to soon. I guess there was a zombie instance I was looking at but currently it is sitting at 1.5G. Let me retry

[etcimon]

I saw you test it, seems not to be going above 398kb memory usage

[dariusc93]

That was because I tested it with 1000 concurrent clients instead of 10000

[etcimon]

That was because I tested it with 1000 concurrent clients instead of 10000

The leak was with even 10 concurrent clients, perhaps 10k is a little much =)

[dariusc93]

Lol maybe it is a bit to much but I wanted to see if the memory would free up. Even a sign of it going down would be good enough.

[etcimon]

Lol maybe it is a bit to much but I wanted to see if the memory would free up. Even a sign of it going down would be good enough.

The request pools are not going through the GC, and currently the freelists in vibe.utils.memory don't have support for prune, so the normal behavior is for the freelist memory to not be freed until someone implements it =)

[etcimon]

If you're wondering about the GC freeing the memory correctly, I think you should try it some other way (like allocating some strings during a request and leaving it on the GC)

[etcimon]

By freeing correctly, I mean giving it back to the OS. The memory is now being re-used correctly in every way in the application

[dariusc93]

Ah, I assume that it was passing through to GC, but with the memory spiking up even with such a small example as http-server that would be problem-some in most cases. One primary reason was in case of an attack unless I can lower the pool size down, restrict it, or restrict the number of concurrent connections to the server all at once

[etcimon]

One primary reason was in case of an attack unless I can lower the pool size down, restrict it, or restrict the number of concurrent connections to the server all at once

You could put a load balancer with a connection pool/queue in front of it, or you can also implement a condition in the AutoFreeList.free where the memory gets freed rather than recycled if the freelist contains over a certain amount of items.

This is an issue in database servers as well, it's recommended to use a connection pool in front of them to optimize performance.

As for DDOS attacks, you should use a proper hosting service that can offer protection, I don't know of many web servers that cope well against them because the solution is usually more specialized.

[etcimon]

With 10k connections, your average buffer size will spike the memory up to 640,000kb in vibe.d because of the 64kb read/write buffer.

The operating system also has a TCP buffer of about 64kb per connection, so it's not surprising to see 1.3gb or more from buffers alone.

A good load balancer will probably allocate a lower buffer while the connection is waiting for a slot

[dariusc93]

On my production server, I do have nginx in front. I was going to stick with haproxy but i read that they are rewriting their code in lua due to lack of contributions to the code that is in C. My servers can handle pretty well towards most attacks and the provider I am with (not with the VPS i spawned up) do offer basic protection up to a point, but a small attack as such as 400Mbps could possibly cause an issue with vibe, though like you said i could use a connection pool.

[etcimon]

but a small attack as such as 400Mbps could possibly cause an issue with vibe, though like you said i could use a connection pool.

That amount of bandwidth is not supposed to reach your server. I use versaweb for protection against this

[dariusc93]

Yea its not suppose too, but I have yet to perform such test against my production servers yet to see how well the provider (SoftLayer) handles everything, but more as a precaution since I do know applications can crash from OOM. Since I do not want to purchase another server just for DDoS Protection, I might go with a provider that offer IPs that connects to my servers either via GRE or VPN since most of my servers wont be using a domain but an IP. If VPN, that would be great since I can leave my backend servers on a private network and connect to them via VPN. I do plan on using Prolexic/Akamai, Arbor, or anything as such (and yes I know it will be costly but I can afford it).

[dariusc93]

This is just down the line though.

[etcimon]

Yes, on my end I'll probably need the better memory management as well so it's something you can expect

[dariusc93]

Okay :) I've also been running some test with botan (the one in C++), and yea it does perform better than OpenSSL in some cases. I really do hope this does get implemented into vibe.d so i could hopefully deploy my app without having to install anything additional.

[dariusc93]

I will have to say, im surprised scrypt havent been implemented into botan lol

[etcimon]

Yes, the main advantage of botan is the algorithm factory and abstractions that automatically add support into tls, x509 and pkcs formats. So you're free to copy over any algorithm and test suite you need

[dariusc93]

@etcimon I was trying to add this to my project to replace dauth, and I keep getting no package file was found, expected one of the following: [immutable(FilenameAndFormat)("dub.json", json), immutable(FilenameAndFormat)("package.json", json)]

This happens when I add it outside of botan (some of my testing was done within botan, like how you have the hash example). Any clue why?

[etcimon]

This error occurs when a dependency path is wrong in the dub.json file

[dariusc93]

Ignore.. oddly enough it wants to work now. 

[posita]

I was going to stick with haproxy but i read that they are rewriting their code in lua due to lack of contributions to the code that is in C.

@dariusc93, you probably figured this out already, but that was an April Fool's post from (I believe) Willy Tarreau.