EnzymeAD / Enzyme

High-performance automatic differentiation of LLVM and MLIR.
https://enzyme.mit.edu
Other
1.28k stars 108 forks source link

Integer type support #158

Open bear24rw opened 3 years ago

bear24rw commented 3 years ago

Are integer types supported? I only get the expected answer for floats or doubles.

Correct answer with type 'float':

#include <stdio.h>
#include <stdint.h>

#define DTYPE float

extern DTYPE __enzyme_autodiff(void*, DTYPE);
DTYPE square(DTYPE x) {
    return x * x;
}
DTYPE dsquare(DTYPE x) {
    return __enzyme_autodiff(square, x);
}
int main() {
    for(DTYPE i=1; i<5; i++)
        printf("square(%f)=%f, dsquare(%f)=%f\n", (float) i, (float) square(i), (float) i, (float) dsquare(i));
}
square(1.000000)=1.000000, dsquare(1.000000)=2.000000
square(2.000000)=4.000000, dsquare(2.000000)=4.000000
square(3.000000)=9.000000, dsquare(3.000000)=6.000000
square(4.000000)=16.000000, dsquare(4.000000)=8.000000

Incorrect answer with type `int':

#include <stdio.h>
#include <stdint.h>

#define DTYPE int

extern DTYPE __enzyme_autodiff(void*, DTYPE);
DTYPE square(DTYPE x) {
    return x * x;
}
DTYPE dsquare(DTYPE x) {
    return __enzyme_autodiff(square, x);
}
int main() {
    for(DTYPE i=1; i<5; i++)
        printf("square(%f)=%f, dsquare(%f)=%f\n", (float) i, (float) square(i), (float) i, (float) dsquare(i));
}
square(1.000000)=1.000000, dsquare(1.000000)=0.000000
square(2.000000)=4.000000, dsquare(2.000000)=0.000000
square(3.000000)=9.000000, dsquare(3.000000)=0.000000
square(4.000000)=16.000000, dsquare(4.000000)=0.000000
wsmoses commented 3 years ago

Integers are presently assumed to be inactive (and thus don't impact the derivative computation).

Allowing active integers is certainly something worth including as we add support for custom accumulations

I'm curious, is there a bigger application use of integers of the differentiable types you have in mind (I haven't been searching, but haven't seen one offhand so it would be helpful to understand).

bear24rw commented 3 years ago

Integers are presently assumed to be inactive (and thus don't impact the derivative computation).

I see, is this something that is "easily" switched on or is it a bigger feature that needs implementation?

I'm curious, is there a bigger application use of integers of the differentiable types you have in mind (I haven't been searching, but haven't seen one offhand so it would be helpful to understand).

My situation is I have a large existing algorithm that I would like to differentiate. The algorithm uses a mix of types and it would be ideal if I could differentiate it "as is".

I wonder if re-writing the types to floats/doubles inside of Enzyme would be a possible solution?

wsmoses commented 3 years ago

Re active integers, I think three things would need to be added:

A codebase that uses a mix of types may not actually use active integers, even if it uses integers as part of the computation. The reason for this is a float => integer cast representing a round/floor correctly has a derivative of 0 (or technically a delta function). Thus I'm a bit curious more some more info (also feel free to email me wmoses@mit.edu if that's preferable / you don't want to share globally on github).

Re your last question, rewriting the types to float/double, however, should probably just work.

bear24rw commented 3 years ago

Thanks for the details, I will pursue the second option of converting all types to doubles and see if that works for my applications.

wsmoses commented 3 years ago

Re-opening this issue to preserve active integer type support as being on the radar. I (and several other contributors) likely won't have time to add such support until after the SuperComputing conference deadline, but hopefully can return to this afterwards.

wsmoses commented 3 years ago

Moreover, it should be relatively easy to add support via LLVM's existing fixed point intrinsics (https://llvm.org/docs/LangRef.html#fixed-point-arithmetic-intrinsics).

The question then for your case would be much simpler, assuming you can modify your code to cause those intrinsics to be emitted.