gyenesvi
commented
6 years ago Thanks for the review and suggestions for improvements. Indeed, there might be a mix of terminologies, as the document evolved over time some naming also changed and it could be beneficial to make it more unambiguous. For example all data is in the form of tensors, it may be better to use the word 'tensor' everywhere instead of data nodes (except where tensors are introduced, it must be specified that it represents the data in the graph).
However, I believe the 'class hierarchy' that you arrive at does not cover the picture completely, I must clarify this point. external, constant and variable are special operations, whose purpose is to introduce data (tensors) that is not the result of operations that perform actual computation (not all operations perform actual computation, some just rearrange the data for example reshape, transpose, split, concat). So there is a one-to-one mapping between external, constant and variable tensors and the operations that produce (introduce) them as a result. It is required to have these as operations because the graph definition syntax consists of a sequence of operations only, and the tensors are defined implicitly as inputs and outputs to operations. In essence, they are analogous to constructors in object oriented languages. Externals, constants and variables are the same kind of data as 'temporaries'; all of them are tensors. The only specialties are that externals come from outside the graph, variables preserve their state between executions of the graph and can change over time, and constants always have the same contents between subsequent executions of the graph. Some implementations may differentiate them for optimization possibilities, while others may treat them equally, just as regular tensors.
Let me know if this clarifies things, we will try to make the wording better to avoid confusion!
zoeoz
I’d like to draw attention to a few subtle inconsistencies about the way the document currently describes data and operations.
Section 2.1 begins with a very concise definition that a computational graph consists of “data nodes” and “operation nodes.” A corresponding figure depicts external (E), constant (C), variable (V), and tensor (T) data nodes (squares) connected to operation nodes (ellipses) and vice-versa. All of this made complete sense to me, and I thought it was well written.
Section 2.3 briefly introduces the term “computational node,” which doesn't appear to be used again. I believe this is the same as an “operation node,” since by definition a “data node” doesn’t perform an operation (or computation). Not sure what the intent was here. It seems to be a distinction without a difference.
In Section 2.5 nodes are described as “operations” and “tensors,” not as “operations” and “data.” It seems the document has made a subtle shift in terminology here. I wouldn’t necessarily conflate “tensors” with “data,” since Section 2.1 defines different kinds of data, namely external, constant, variable, and tensor (essentially, “temporary” data). I think part of the issue is that “tensor” is defined as a “data node” in Section 2.1, while in Section 2.5 it seems to be referring to data type.
Section 4 briefly uses the term “computational primitive,” analogous to the curious use of “computational node” in Section 2.3. Section 4 also uses the term “graph primitive” to describe the same thing as a “computational primitive.” I think “graph primitive” seems more appropriate to me, since by previous definitions, only operation nodes can be primitives.
Section 4.1 introduces a clash of definitions. Section 2.1 defines an “input data node” of the external, constant, and variable variety. Now Section 4.1 describes external, constant, and variable as an “operation.” I think the document should choose one definition or the other, i. e., are external, constant, and variable data nodes or operation nodes? They shouldn’t be identified as both.
Conceptually, if I was going to do a class hierarchy based off definitions from Section 2.1, it would look like this: Data and Operation are derived from Node. External, Constant, Variable, and Temporary are derived from Data. Primitive and Compound are derived from Operation. The operations in Sections 4.2-4.7 are derived from Primitive, and the operations in Section 4.8 are derived from Compound.