crystal-lang / crystal

The Crystal Programming Language
https://crystal-lang.org
Apache License 2.0
19.37k stars 1.62k forks source link

Add API for `Base64.encode` / `Base64.decode` with an `IO` as the source #14603

Open jgaskins opened 4 months ago

jgaskins commented 4 months ago

Related to #2964 Originally mentioned on the forum

In order to base64-encode data with Base64.encode, that data must exist in RAM. Technically, it just needs to be something that responds to to_slice, but at least in the stdlib all of those things are actually stored in RAM:

➜  crystal git:(b64-encode-io-to-io) ✗ ag 'def to_slice( :|$)' src
src/io/memory.cr
418:  def to_slice : Bytes

src/llvm/memory_buffer.cr
14:  def to_slice

src/slice.cr
848:  def to_slice : self

src/string.cr
5449:  def to_slice : Bytes

src/bit_array.cr
626:  def to_slice : Bytes

src/static_array.cr
369:  def to_slice : Slice(T)

My use case is to base64-encode a file of arbitrary size. Some APIs accept JSON but allow you to send/receive file data inside that payload. I've had to do this a few times — for example, some email service providers allow you to send attachments embedded in JSON to their HTTP API, one integration I had to build for mortgage lenders to integrate with their loan origination system base64-encoded documents inside XML payloads, things like that.

In addition to certain email providers' specific APIs, you have to do the same in SMTP itself. SMTP is a text-based protocol, so attachments are typically encoded into a text-friendly format. The shard I typically use to send emails has to load the entire file into RAM.

So since base64-encoding nontrivial amounts of data is not an uncommon requirement, I figured this might be a good addition to the Base64 stdlib.

BlobCodes commented 4 months ago

Implemented in #13574 (which I will work on soon.. probably)

jgaskins commented 4 months ago

Implemented in #13574 (which I will work on soon.. probably)

That PR looks like it only implements this overload for decode. Am I overlooking encode?

BlobCodes commented 4 months ago

Ah, seems like I misremembered what I did in that PR.. Yes, it only implements IO-to-IO decode