Untagged enums with empty variants (de)serialize in unintuitive ways

[Sushisource]

Hello - I've got some structs like:

#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)]
pub struct SubscriptionUpdate {
    pub result: SubscriptionResult,
    pub subscription: u64,
}

#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)]
#[serde(untagged)]
pub enum SubscriptionResult {
    #[serde(rename = "ready")]
    Ready,
    #[serde(rename = "invalid")]
    Invalid,
    Whatever {
        something: String,
    },
}

I would expect

let t = SubscriptionUpdate {
    subscription: 0,
    result: SubscriptionResult::Invalid,
};

To serialize as { "result": "invalid", "subscription": 0} but it's { "result": "null", "subscription": 0}

That strikes me as really surprising, and I haven't yet figured how to get it to behave the way I'd like. Any advice? Does this qualify as a bug?

Love serde, btw, great work!

[dtolnay]

It's actually "result": null, not "null". Playground

Untagged means that the serialized data does not identify which variant it represents via some name / tag. "result": "invalid" does identify a specific variant so it wouldn't make sense to consider that untagged.

The recommended way would be something like this. You could check whether serde_aux or serde_with or some other helper crate already provides an implementation of this.

use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize, Debug)]
struct SubscriptionUpdate {
    result: SubscriptionResult,
    subscription: u64,
}

macro_rules! named_unit_variant {
    ($variant:ident) => {
        pub mod $variant {
            pub fn serialize<S>(serializer: S) -> Result<S::Ok, S::Error>
            where
                S: serde::Serializer,
            {
                serializer.serialize_str(stringify!($variant))
            }

            pub fn deserialize<'de, D>(deserializer: D) -> Result<(), D::Error>
            where
                D: serde::Deserializer<'de>,
            {
                struct V;
                impl<'de> serde::de::Visitor<'de> for V {
                    type Value = ();
                    fn expecting(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
                        f.write_str(concat!("\"", stringify!($variant), "\""))
                    }
                    fn visit_str<E: serde::de::Error>(self, value: &str) -> Result<Self::Value, E> {
                        if value == stringify!($variant) {
                            Ok(())
                        } else {
                            Err(E::invalid_value(serde::de::Unexpected::Str(value), &self))
                        }
                    }
                }
                deserializer.deserialize_str(V)
            }
        }
    };
}

mod strings {
    named_unit_variant!(ready);
    named_unit_variant!(invalid);
}

#[derive(Serialize, Deserialize, Debug)]
#[serde(untagged)]
enum SubscriptionResult {
    #[serde(with = "strings::ready")]
    Ready,
    #[serde(with = "strings::invalid")]
    Invalid,
    Whatever {
        something: String,
    },
}

fn main() {
    let t = SubscriptionUpdate {
        subscription: 0,
        result: SubscriptionResult::Invalid,
    };

    let j = serde_json::to_string(&t).unwrap();
    println!("{}\n", j);
    println!("{:#?}", serde_json::from_str::<SubscriptionUpdate>(&j).unwrap());
}

[Sushisource]

@dtolnay Thanks a lot for that sample code! Works great. I do understand the logic behind why it shows up as null, but it still might be nice to have this mentioned in the docs / some built-in way to overcome it.

[dtolnay]

Sounds good. Where in the docs would you expect to find this information?

[Sushisource]

I would expect to find it in here https://serde.rs/enum-representations.html I think, perhaps called out in the Untagged section, by mentioning empty variants specifically.

[sztomi]

In some cases, you can solve this problem by applying untagged to just one variant:

#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
pub enum ResponseMessage {
  Cancelled,
  NotStopped,
  #[serde(untagged)]
  Error(String),
}

This will rename the tagged variants as you would expect and allow serializing anything else as a string.

[MohsenNz]

Thank you @sztomi . I was looking for this. it's better than previous solution :)

[kevin-leptons]

In some cases, you can solve this problem by applying untagged to just one variant:

#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
pub enum ResponseMessage {
  Cancelled,
  NotStopped,
  #[serde(untagged)]
  Error(String),
}

This will rename the tagged variants as you would expect and allow serializing anything else as a string.

LOL. This one must be included in the official document. I can't believe that deserializing enum is a trick.