To make the later maintenance of the Flink-backend easier, we should document the compiler details while we figure them out. Maybe we should even shortly describe the instructions - and possibly do a better job with the JavaDoc there than it's the case in SystemML right now. I suggest to
[ ] Keep a Google-doc / Wiki-page with details about the compiler.
Key classes, key points in the Flink implemenation, how does it work in general...
I started a high level overwiew a while ago that can be found here.
[ ] Add JavaDoc to all instruction classes
Currently not all instruction classes in SystemML have sufficient documentation to understand right away what they do.
Additionally, some instructions perform recompilation inside their executeInstruction methods. this can lead to the effect that a SparkInstruction never actually executes a Spark job. I think it would be good to have this information somewhere.
To make the later maintenance of the Flink-backend easier, we should document the compiler details while we figure them out. Maybe we should even shortly describe the instructions - and possibly do a better job with the JavaDoc there than it's the case in SystemML right now. I suggest to
executeInstructionmethods. this can lead to the effect that a SparkInstruction never actually executes a Spark job. I think it would be good to have this information somewhere.