Transition to `quote` + `syn` for code generation.

[gibbz00]

Apil 13

Throwing this out here ASAP as a draft so that others can become aware of it. Will most definitely clash with PRs such as https://github.com/tokio-rs/prost/pull/1019.

Built upon #1020. (rebased)

Will create a proper write-up once things become presentable.

Update (14 April)

Post quote + syn POC made separation of concerns a lot clearer, which lead to a bunch of module splitting. Might be worth trying to send for these as separate PRs, which this draft then builds upon. Ex. placing unescape_c_escape_string and its unit tests in a separate module.

Update (21 April)

Think this is ready to be reviewed now.

Goals:

1. Lay a groundwork for enabling codegen plugin/modularity.
2. Make it easier to reason and debug about the generated code output

Assessment:

1. Still plenty left to be done. Refrained from making larger API changes before input from others, in addition to the experience gained from trying to add JSON mapping.

2. Similar to outcome of 1, but here an example of what has changed:

Example:

Before

self.buf.push_str(&format!(
    "impl {}::Name for {} {{\n",
    self.config.prost_path.as_deref().unwrap_or("::prost"),
    to_upper_camel(message_name)
));
self.depth += 1;

self.buf.push_str(&format!(
    "const NAME: &'static str = \"{}\";\n",
    message_name,
));
self.buf.push_str(&format!(
    "const PACKAGE: &'static str = \"{}\";\n",
    self.package,
));

let prost_path = self.config.prost_path.as_deref().unwrap_or("::prost");
let string_path = format!("{prost_path}::alloc::string::String");

let full_name = format!(
    "{}{}{}{}{message_name}",
    self.package.trim_matches('.'),
    if self.package.is_empty() { "" } else { "." },
    self.type_path.join("."),
    if self.type_path.is_empty() { "" } else { "." },
);
let domain_name = self
    .config
    .type_name_domains
    .get_first(fq_message_name)
    .map_or("", |name| name.as_str());

self.buf.push_str(&format!(
    r#"fn full_name() -> {string_path} {{ "{full_name}".into() }}"#,
));

self.buf.push_str(&format!(
    r#"fn type_url() -> {string_path} {{ "{domain_name}/{full_name}".into() }}"#,
));

self.depth -= 1;
self.buf.push_str("}\n");

After

let name_path = self.prost_type_path("Name");
let message_name_syn = message_name.parse_syn::<syn::Type>();
let package_name = &self.package;
let string_path = self.prost_type_path("alloc::string::String");
let fully_qualified_name =
    FullyQualifiedName::new(&self.package, &self.type_path, message_name);
let domain_name = self
    .config
    .type_name_domains
    .get_first(fq_message_name.as_ref())
    .map_or("", |name| name.as_str());

let fq_name_str = fully_qualified_name.as_ref().trim_start_matches('.');
let type_url = format!("{}/{}", domain_name, fq_name_str);

quote! {
    impl #name_path for #message_name_syn {
        const NAME: &'static str = #message_name;
        const PACKAGE: &'static str = #package_name;

        fn full_name() -> #string_path { #fq_name_str.into() }
        fn type_url() -> #string_path { #type_url.into() }
    }
}

Other examples

Some(quote! {
    #(#documentation)*
    #(#(#type_attributes)*)*
    #(#(#message_attributes)*)*
    #[allow(clippy::derive_partial_eq_without_eq)]
    #[derive(Clone, PartialEq, #prost_path)]
    #maybe_skip_debug
    pub struct #ident {
        #(#resolved_fields,)*
        #(#resolved_oneof_fields,)*
    }

    #nested

    #maybe_type_name
})

quote! {
    #(#documentation)*
    #maybe_deprecated
    #[prost(#field_type_attr, #maybe_label #maybe_boxed tag=#field_number_string, #maybe_default)]
    #field_attributes
    pub #field_identifier: #field_type
}

One takeaway it that this PR removes manual indentation handling, delegating it to the format feature.

Possible future "improvements" which lead to breaking changes.

• Consider changing the ServiceGenerator API to pass a token TokenStream/syn::Item rather than buf: &mut String. See tonic's usage: https://github.com/hyperium/tonic/blob/eeb3268f71ae5d1107c937392389db63d8f721fb/tonic-build/src/prost.rs#L247-L276
• Consider returning parse errors in config for operations such as config.field_attribute("in", "#[serde(rename = \"in\")]");

PR splitting

Waiting for #1029 and #1030 to get merged before rebasing this one on top of them. Unsure how much else there's to split up.

[caspermeijn]

Unsure how much else there's to split up.

I suggest the following to make reviewing easier:

• Split off "this PR removes manual indentation handling, delegating it to the format feature." That is a significant change that could be merged separately.
• You have made multiple PRs to move code from code_generator.rs to separate modules, however the diff this PR is still a big red file and big green files. So I can't compare the two versions. Maybe a rebase will solve that.

[gibbz00]

Hi, thank you for starting to look into this pr 😊

I'll see what I can do during the weekend.