Why does having inconsistent input and output dimensions in a Halide generator significantly slow down performance?

[mhua97]

`class Example1Generator : public Generator { public: Input<Buffer<uint8_t, 2>> input{"input"}; Output<Buffer<uint8_t, 3>> output{"output"};

void generate()
{
    Var x("x"), y("y"), c("c");

    Func f1;
    f1(x, y) = input(x, y) + 10;
    Func f2;
    f2(x, y) = input(x, y) + 20;
    Func f3;
    f3(x, y) = input(x, y) + 30;
    Func f4;
    f4(x, y) = input(x, y) + 40;
    Func f5;
    f5(x, y) = input(x, y) + 50;

    output(c, x, y) = mux(c, {f1(x, y), f2(x, y), f3(x, y), f4(x, y), f5(x, y)});

    if (using_autoscheduler())
    {
        input.set_estimates({{1024, 16384}, {1024, 8192}});
        output.set_estimates({{0, 10}, {1024, 16384}, {1024, 8192}});
    }
}

};

class Example2Generator : public Generator { public: Input<Buffer<uint8_t, 2>> input{"input"}; Output<Buffer<uint8_t, 2>> output{"output"};

void generate()
{
    Var x("x"), y("y");

    Func f1;
    f1(x, y) = input(x, y) + 10;

    output(x, y) = f1(x, y);

    if (using_autoscheduler())
    {
        input.set_estimates({{1024, 16384}, {1024, 8192}});
        output.set_estimates({{1024, 16384}, {1024, 8192}});
    }
}

}; ` Consider this: calling Example2Generator five times takes several times longer than calling Example1Generator once. Is this because the input to output dimensionality increased? What if my requirement necessitates going from low-dimensional input to high-dimensional output?

[abadams]

It's nothing to do with dimensionality. It's because calling Example2Generator five times reads the input from memory five times, and calling Example1Generator once reads the input from memory once (and then does five things with it). You'd see the same effect in C.

[mhua97]

Calling Example2Generator once takes about 3ms, but calling Example1Generator once takes 160ms. This is the confusing part for me.

[abadams]

In Example1Generator the innermost dimension in memory of the output buffer is 'c'. That's a small dimension - probably length five in practice given that you're muxing together five things. That means any use of SIMD (vectorization) is going to need to do a strided store to memory instead of a dense vector store, which is very bad for performance.

So if Example1Generator is slow it's probably due to the memory layout of your output buffer. It looks like you're using an autoscheduler, but this is not something an autoscheduler can help with because the memory layout of the output buffer is controlled by the caller.

A fast version of Example1Generator would look like:

    Var x("x"), y("y"), c("c");

    Func f1;
    f1(x, y) = input(x, y) + 10;
    Func f2;
    f2(x, y) = input(x, y) + 20;
    Func f3;
    f3(x, y) = input(x, y) + 30;
    Func f4;
    f4(x, y) = input(x, y) + 40;
    Func f5;
    f5(x, y) = input(x, y) + 50;

    output(x, y, c) = mux(c, {f1(x, y), f2(x, y), f3(x, y), f4(x, y), f5(x, y)});

    output.reorder(c, x, y).bound(c, 0, 5).unroll(c).vectorize(x, 16).parallel(y, 8);

Note that 'reorder' just reorders the loops, not the memory layout.

[mhua97]

Thank you very much for your suggestion, it's very helpful.