Partial encoding with padding

[mitsuhiko]

Today when calling encode_append it will always write the padding into the chunk. It's not clear to me if there is a way without buffering ahead of time to encode without padding and add the padding once at the end. Motivating use case:

fn basic_auth(username: &str, password: Option<&str>) -> String {
    let mut buf = "Basic ".to_string();
    BASE64.encode_append(username.as_bytes(), &mut buf);
    BASE64.encode_append(b":", &mut buf);
    if let Some(password) = password {
        BASE64.encode_append(password.as_bytes(), &mut buf);
    }
    buf
}

Today this creates an unexpected HTTP basic auth header as padding is added three times. While probably technically true, I feel uncomfortable with emitting this type of base64.

[ia0]

Thanks for bringing this up! Your assessment is correct:

1. It is correct to emit padding multiple times.
2. To avoid emitting padding multiple times, buffering is needed.
3. The library doesn't provide any buffering capabilities.

I'm currently in a considerable refactor of the internals of the library to address #69 without performance cost. So this comes at the right time to be taken into account. I would need to think more about it to find a satisfying solution, but I can provide you a workaround right now to unblock you until there's a release solving your problem in a nice way:

fn basic_auth(username: &str, password: Option<&str>) -> String {
    let mut buffer = Base64Buf::default();  // This line is new and replaces BASE64.
    let mut output = "Basic ".to_string();
    buffer.encode_append(username.as_bytes(), &mut output);
    buffer.encode_append(b":", &mut output);
    if let Some(password) = password {
        buffer.encode_append(password.as_bytes(), &mut output);
    }
    buffer.finalize(&mut output);  // This line is new.
    output
}

#[derive(Default)]
struct Base64Buf {
    buf: [u8; 4],
}

impl Base64Buf {
    fn encode_append(&mut self, mut input: &[u8], output: &mut String) {
        if self.buf[3] != 0 {
            let len = self.buf[3] as usize;
            let add = std::cmp::min(3 - len, input.len());
            self.buf[len..][..add].copy_from_slice(&input[..add]);
            self.buf[3] += add as u8;
            input = &input[add..];
            if self.buf[3] == 3 {
                BASE64.encode_append(&self.buf[..3], output);
                self.buf[3] = 0;
            }
        }
        let len = input.len() / 3 * 3;
        BASE64.encode_append(&input[..len], output);
        input = &input[len..];
        let len = input.len();
        self.buf[..len].copy_from_slice(input);
        self.buf[3] = len as u8;
    }

    fn finalize(self, output: &mut String) {
        let len = self.buf[3] as usize;
        BASE64.encode_append(&self.buf[..len], output);
    }
}

If you don't want to use this workaround until I release something that solves both #69 and this issue, then I could consider providing a generic version of this buffered struct. But it will also take some time and will probably be removed/modified when going to v3.

[ia0]

Hi @mitsuhiko , I gave up on finding a nice generic solution solving #69 and this issue at the same time (I'll try again next year), and will instead solve both separately. I've merged #89 which should fix your issue. You may subscribe to #90 to be notified when it's published on crates.io. In the meantime, you may use the git version and provide feedback if it doesn't solve your issue. Here is how it would look like now:

// Sadly, the following is needed until 3.0.0 is released.
// You may share a single such static in your program.
static BASE64: data_encoding::Encoding = data_encoding::BASE64;

fn basic_auth(username: &str, password: Option<&str>) -> String {
    let mut buf = "Basic ".to_string();
    let mut encoder = BASE64.new_encoder(&mut buf);
    encoder.append(username.as_bytes());
    encoder.append(b":");
    if let Some(password) = password {
        encoder.append(password.as_bytes());
    }
    encoder.finalize();
    buf
}

[mitsuhiko]

Thank you @ia0. This is great and solves my problem nicely.