serde-rs / json

Strongly typed JSON library for Rust
Apache License 2.0
4.7k stars 536 forks source link

Make all serialization functions available for no-std #1122

Open devrandom opened 3 months ago

devrandom commented 3 months ago

We ran into OOMs in a memory constrained no-std environment, when serializing. We noticed that to_writer was not available for no-std, even though there is nothing preventing it from working.

This PR exposes the relevant serialization functions. It also exposes the crate's no-std io shim, since types from there appears in the ser API.

related to #1040

devrandom commented 3 months ago

If I understand correctly, you are concerned that if someone refers to the no_std version of structs / fns in serde_json::io, then they could break if the implementation changes to std via feature unification.

While you are correct in general, I don't think that's possible in this case, because the alloc version of serde_json::io is a strict subset of std::io, so something that compiled with the alloc version cannot fail to compile once the implementation changes to std. There are no extra fns in the former that are not in the latter, etc. . I tried to write a failing example, but could not come up with one.

devrandom commented 3 months ago

OK, I did find one incompatibility, in the construction of Error - and I think it's fixed now. Added the binary that I used to test.

dtolnay commented 3 months ago

Some other examples of things that compile only if std is off, and fail non-additively as soon as std is enabled anywhere:

fn main() {
    let msg = Box::new("...");
    let _ = serde_json::io::Error::new(serde_json::io::ErrorKind::Other, &msg);
}
fn main() {
    match serde_json::io::ErrorKind::Other {
        serde_json::io::ErrorKind::Other => {}
    };
}
fn main() {
    let error = serde_json::io::Error::new(serde_json::io::ErrorKind::Other, "...");
    let _ = std::panic::catch_unwind(|| error);
}
use std::fmt::{self, Debug, Display};

enum MyError {
    JsonIo(serde_json::io::Error),
}

impl Display for MyError {
    fn fmt(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
        match self {
            MyError::JsonIo(error) => error.fmt(formatter),
        }
    }
}
trait WriteExt: serde_json::io::Write {/* ... */}

impl serde_json::io::Write for &mut dyn WriteExt {
    fn write(&mut self, bytes: &[u8]) -> serde_json::io::Result<usize> {/* ... */}
    fn flush(&mut self) -> serde_json::io::Result<()> {/* ... */}
}

Not sure what can be done about this one:

struct Buffer {/* ... */}

impl serde_json::io::Write for Buffer {
    fn write(&mut self, bytes: &[u8]) -> serde_json::io::Result<usize> {/* ... */}
    fn flush(&mut self) -> serde_json::io::Result<()> {/* ... */}
}

impl core2::io::Write for Buffer {
    fn write(&mut self, bytes: &[u8]) -> core2::io::Result<usize> {/* ... */}
    fn flush(&mut self) -> core2::io::Result<()> {/* ... */}
}
devrandom commented 2 months ago

OK, good points.

I pushed a commit that changes the approach - the no-std io module is now always exported.

However, this is still non-additive, because Serializer uses different Write depending on the feature.

I'm not sure there's a complete solution without duplicating Serializer for the two cases.