Procedural macros interactions with constant evaluation

[9il]

From 1566-proc-macros.md:

Interactions with constant evaluation

Both procedural macros and constant evaluation are mechanisms for running Rust code at compile time. Currently, and under the proposed design, they are considered completely separate features. There might be some benefit in letting them interact.

It is terrific feature! It allows to do awesome language idioms such as D's static if/static foreach but in library code.

For example static if/ static foreach and #2000 are required to port ndslice to Rust from D. I am aware about ndarray cargo package, ndslice implementation has more optimisation powers.

Related issues https://github.com/rust-lang/rust/issues/38356

[eddyb]

AFAIK, D uses the C++ approach to generics (aka "templates"), which is to type-check after monomorphizing them. The macro system is different in that it produces source and must be sequenced before the type-system can exist, for reasons of global reasoning (e.g. impl coherence). Even if we'd introduce more interactions, they'd require very careful design to prevent unsoundness.

Now, for static if/foreach, what's the usecase #2000 or an extension thereof can't handle?

Also, going all the way up to the macro system for an optimization, instead of pushing it down? I'd understand this better if this were about getting an algorithm on tuples/arrays to type-check.

[9il]

This is base struct for Rust's ndarray

pub struct ArrayBase<S, D>
    where S: Data
{
    /// Rc data when used as view, Uniquely held data when being mutated
    data: S,
    /// A pointer into the buffer held by data, may point anywhere
    /// in its range.
    ptr: *mut S::Elem,
    /// The size of each axis
    dim: D,
    /// The element count stride per axis. To be parsed as `isize`.
    strides: D,
}

2000 allows to make dim a compile time constant.

Assume you have an updated N-dimensional ArrayBase with fixed dimensions at compile time args. Then you have a ArrayBase's method, say atIndex for simplicity, that accepts M Args, M <= N. Based on M value and Args's types we can decide what return Type atIndex should have and how to compute it.

In D it would look similar to (very simplified):

struct ArrayBase(size_t N, T)
{
    ...
    auto atIndex(Args...)(Args args)
    {
        enum M = Args.length;
        auto ptr = this._ptr;
        static foreach (i; 0 .. M)
        {
              // move ptr to the element/subarray
        }
        static if (M == N && allIndexes!Args) // allIndexes!Args is a template the evaluates to false/true
        {
               return *ptr;
        }
        else
       {
              enum NewN = ... // compute new N using CTFE and information about types
              return ArrayBase!(NewN, T)(*ptr, other_args);
       }
    }
}

2000 introduces a compilte time constants, but they can not be used to generate/choose code to compile. D's CTFE is something recursive like, it allows to do CTFE, mix code, and do CTFE with mixed code again. #2000 makes me wonder if Rust will be able to do the same

[Diggsey]

@9il with specialisation, you can change the implementation based on a constant number without requiring a macro.

[9il]

@Diggsey, could you please provide an example?

[Diggsey]

Well with both #2000 and specialization, you could do something like this:

trait Algorithm {
    fn calculate(&self) -> Result;
}

impl<const N> Algorithm for Array<N> {
    default fn calculate(&self) -> Result {
        <general impl>
    }
}

impl Algorithm for Array<1> {
    fn calculate(&self) -> Result {
        <simple base case>
    }
}

[Centril]

@9il are you satisfied with the provided solution? if so, please close this issue. :)

[9il]

Specialization solve's static if issue, but Rust still doesn't have static foreach alternative.

[eddyb]

I think looping over tuples should be supplanted by VG (variadic generics). It can't literally be a for loop, but with generic closures, we can get really close.

As for looping over integers and using them in const generics, you should be able to do it with a recursive impl, other than for the lack of a way to match on 0 vs N + 1 in the initial const generics.

You can probably use specialization to handle N in general, vs 0? Or add a where clause that wouldn't hold for N = 0, such as where [(); N - 1]: (yes, no bounds, but the type must still be well-formed)

And, again, sugar would use generic closures, e.g. for<const i: usize> |...| {...}.

[Centril]

It can't literally be a for loop, but with generic closures, we can get really close.

And, again, sugar would use generic closures, e.g. for |...| {...}.

I'm working on an RFC re. that. :)

[eddyb]

@Centril heh, I was hoping we'd revive @Amanieu's #1650, glad to have someone on it!

[Coder-256]

Is there any progress on this? I think this should be revisited due to the recent progress for const generics. For example, now there are some really silly things that rely on this:

foo!(u16::from_le_bytes([0x37, 0x13]));

foo can't actually read the resulting number without adding in a special case for from_le_bytes, nevertheless any user-provided const fn.

Side question: this is a bit of a weird question, but is const evaluation actually the one single thing that const fns can do and procedural macros can't? Unless I'm missing something, the only difference is that const fns create a const context, and this issue is just pointing out that procedural macros could also create a const context and evaluate a const value. Although there is the issue IIUC that macros are expanded long before const evaluation.

[Lokathor]

I think part of the problem is also that macros expand outside-in but const fn and other code contexts want to evaluate inside-out.

[Diggsey]

foo can't actually read the resulting number without adding in a special case for from_le_bytes, nevertheless any user-provided const fn.

But foo could expand to something like this:

const N: u16 = u16::from_le_bytes([0x37, 0x13]);

use_n_in_some_way([24; N]);

... without needing any special cases for from_le_bytes. The macro can't actually use the value because macro expansion happens long before const evaluation, but it can generate code that does use the value and that also runs at compile time. This is sufficient for the vast majority of cases.

[Lokathor]

Well, unless you need the proc-macro to know the value of N as part of the code generation decision.

[Coder-256]

Well, unless you need the proc-macro to know the value of N as part of the code generation decision.

That's what I was getting at.

I think part of the problem is also that macros expand outside-in but const fn and other code contexts want to evaluate inside-out.

Hm, I didn't realize that. Still, maybe there's some way (maybe an attribute) that a procedural macro could require a const context for its body and also require that the body is pre-evaluated?

Also, probably the biggest hurdle in order to do anything useful with the input is that you would need to accept an actual, typed value rather than a token tree, which is really weird. What I'm imagining is basically a normal const fn except it returns a token tree.

The more I think about it, the more it sounds like it wouldn't work because type checking, const evaluation, and macro evaluation would be mutually recursive. I only think it might work because Rust can compile things in passes, even across files without forward declaration. One concern is, for example, a const procedural macro relies on impl const Trait1 for Foo to generate impl Trait2 for Foo, which might require running type checking before Foo has implemented all of its traits. There would likely need to be some limits if this even worked in the first place. In any case I wonder if this might be a separate issue than the original, if that's the case sorry for derailing.