How to Decrypt larger chunks of Data (>16K)?

[Hitman831]

I have to decrypt around 140K bytes of data for my use. The problem is, it cannot decrypt such a large chunk at once. And when I try to break it into 16K smaller chunks, decrypt is passing chunk by chunk but, meanwhile I'm getting another read on the socket and the existing decrypt fails. Any solution?

[david-maw]

Without seeing your code it's difficult to guess what's happening but you are correct in thinking that to use TLS/SSL you need to break up large chunks of data into smaller blocks before transmission (anything less than 65 kB should do). Assuming you are using this implementation each block of data will have been decrypted before it's delivered to your code so all you should need to do is reassemble your chunk from the individual blocks. Of course you'll have to use some conventions to describe your message structure (TCP just delivers streams of bytes and need not respect message boundaries).

[Hitman831]

Thank you for the help. Appreciate it. Also, I'm facing a similar issue where I have to encrypt larger data. But, it's failing. Any idea on how can Schannel handle it? Or is it a TLS/SSL limitation?

[david-maw]

I'm afraid it's been a while since I dug into this in any depth but I seem to remember SCHANNEL had a limitation of 16 kB or so, which sort of makes sense, a single SSL/TLS message is limited to about that (technically 2**14 + 2048 I believe). In theory longer messages should be broken up and sent in fragments, which is basically what TCP does anyway but I'd speculate that SCHANNEL does not do this, perhaps because (for example) doing so makes error reporting challenging. Incidentally that suggests my "less than 65kB" earlier should probably have been "less than 16 kB", but if you really want to know for sure, I suggest testing.

Anyway, the safest solution is probably to split your long message into shorter ones and reassemble them after decryption, that way you never ask SCHANNEL to handle any individual message longer than 16 kB all at one. If you want to experiment with SCHANNEL you could try using my client & server examples then messing with message lengths to see what happens. I always find it easiest to start with working code then break it...

[Hitman831]

Hi David, Thanks for a quick response. I've been trying to loop the entire data into iteration of (<16K) chunks. But while doing this, it gets interrupted by a socket read in between. Is it possible? If so, what do you suggest would be a better option to go ahead.

[david-maw]

Are you reading into the same buffer you are decrypting from? I presume you must be using some sort of asynchronous read because otherwise it couldn’t interrupt other processing (decryption in this case). In general each async read needs its own buffer otherwise you run the risk they’ll overlap (as you have seen) and once a read completes you need to copy the data it delivered to some safe location (essentially a queue) then re-initiate the read using the same buffer it did before because that buffer is now available because you already copied the data from it. You can get clever an use some sort of buffer queue, but memory is so cheap and modern CPUs copy data so fast that it’s hardly worth bothering.