Please improve the documentation

[Hvass-Labs]

Hello Everyone,

I'm currently in the process of learning how to use TensorFlow (TF) and wanted to use PrettyTensor (PT) to make the graph construction much easier. The idea of PT is great! - but the documentation is short and confusing so it is difficult to learn.

Here's a few comments I have so far. I hope the comments may help you guys understand what new users find confusing so you can improve it. I might add to the list as I get further with PT.

• The documentation of PrettyTensor.md starts with "A PrettyTensor is a Tensor with a builder interface facade." I had to google search for "builder interface facade" because I don't know what it means. Perhaps it was some kind of fancy software design concept used at Google. But this is the only occurrence found by google search. So already from the beginning you're using new and confusing terminology.
• The doc for the max_pool() function (and others) says: "kernel: The size of the patch for the pool, either an int or a length 1 or 2 sequence (if length 1 or int, it is expanded)." This is hard to understand. Similarly for the stride.
• Also in the doc for max_pool() it says: "edges: Either PAD_SAME or PAD_VALID to control the padding." But the function has "edges=SAME" as default. So is it PAD_SAME or SAME? And is it a constant or is it a text-string like in TensorFlow? Looking in the code I found the answer, but that shouldn't be necessary as your doc-strings should be consistent and complete.
• softmax_classifier() returns "A tuple of the softmax's name and the loss tensor's name in m.bits." What does this mean?! What is the loss-function? cross-entropy? I couldn't figure out from the cryptic doc-string, nor is it apparent from the source-code itself.
• Your doc-strings are generally too short and cryptic and assume too much of the reader.
• reshape() seems to be convenient and the doc is more verbose than usual - but unfortunately it's still dense and cryptic so I can't get the compact string-syntax working.
• In Template Mode it says "Templates allow you to define a (potentially large) graph with some unknown values." Why is "(potentially large)" in there? Why wouldn't I be able to make large graphs? Be careful how you write things or it'll just be (unnecessarily) confusing.
• The examples for Template Mode are hard to understand. I still haven't got templates working. I just wanted the dropout-layer to work in training and not during testing. I don't want to replace the input when I call template.construct() because I just use a feed-dict for replacing the input, but then PT complains about a missing key for the template. I tried different things and ended up with errors deep inside PT.
• Here's another problem with the doc for Template Mode: "You should use caution because if a template is called with incompatible values (e.g. train and test using different widths), then it will break." You need to explain why. Perhaps it's because the variables have already been allocated and they're not re-allocated for each call to template.construct()? Please clarify in the doc.
• How is a template different from creating the graph in a model()-function and changing parameters there? When would I want to use templates vs. model-creator-functions? Is the difference that templates don't re-allocate weights?
• What is a bookkeeper? I see it passed around to functions but it's apparently only mentioned twice in the doc. Oh, there's a doc-file for this, but it doesn't really explain what it does except for this: "Small class to gather needed pieces from a Graph being built."
• Argument names are different across functions, e.g. softmax_classifier_with_sampled_loss(inputs, num_classes, ...) and softmax_classifier(input_layer, class_count). This is confusing when you want to call the functions using named args e.g. softmax_classifier(input_layer=foo, class_count=bar) which is a more safe and readable programming style than using un-named arguments, but you then need to remember different arg-names as tab-completion doesn't always work.
• The doc doesn't mention if I can override the default-scope, e.g. by having a different activation_fn for one of the layers.
• Accessing variables should be documented better. It is discussed here: https://github.com/google/prettytensor/issues/6 but when I execute: with tf.variable_scope('layer1', reuse=True): tf.get_variable('weights') I get the error: "ValueError: Variable layer1/weights does not exist, disallowed. Did you mean to set reuse=None in VarScope?" I guess it says that the variable named layer1/weights doesn't exist and reuse=True means it cannot create a new variable. I eventually got it working but the error message is cryptic and needs to be clarified. Furthermore, the design of this is quite awkward. It appears to be TensorFlow code rather than PrettyTensor. But why can't I just call tf.get_variable('layer1/weights', reuse=True)? Why do I have to enclose it in a with-block for the scope and reuse=True? From the user's perspective, that looks like poor design. Finally, the doc is also not clear that this returns a TF object which I then have to run in a session to actually get the contents of the variable. Why isn't there a function tf.get_variable_contents(session, 'layer1/weights') which returns a numpy array with the variable contents?
• I have looked at the source-code itself and there is practically no comments for the code-lines, so it takes a lot of time and effort to understand. The coding style is also very dense and code fragments are not separated by extra lines etc. Comments and line-spacings would make the code much easier to understand for others. Some high-level descriptions of the overall ideas and structures of the implementation would also help.
• You can give two layers the same name. This should raise an exception.
• There is apparently no __version__ variable.

I haven't gotten around to sequence-construction, towers, etc. in PrettyTensor, so I can't really comment on that at this point. However, from having briefly looked at the documentation it seems just as confusing as the rest.

I think it would be well worth the effort to significantly improve the documentation. Perhaps have someone else in Google go over it so it's written with a fresh set of eyes. This may take someone a week or two to complete. But this is a tiny cost compared to the combined time wasted by thousands of new users who are trying to figure out how to do even simple things in TensorFlow and PrettyTensor. If 10.000 new users waste 1 hour each on average, then that's a combined 10.000 hours they could have spent on more productive things - and I think that's a very low estimate of the gain that could be had simply by improving the documentation. That's a pretty big contribution to the community, if you look at it that way.

Also, why isn't PrettyTensor an integrated part of TensorFlow? Why would anyone want to do the low-level coding required by TF and not use something like PT? In PT I can define my graph concisely in a few lines of code and not have to worry about allocating variables of the right shape, etc. The idea of PT is great, it just needs a more polished documentation.

I also looked briefly at Keras. Although its documentation seems to be more elaborate and polished than PrettyTensor, Keras has quite different syntax and semantics. In some cases PT seems to be simpler and better than Keras. So I'm hoping you guys will greatly improve the documentation for PrettyTensor so it's easier to learn for beginners.

I will take a look at TFLearn next. The documentation appears to be more polished than PrettyTensor.

I've seen a video about Microsoft's CNTK. It appears to be much simpler and more polished than TensorFlow + PrettyTensor. It'll be interesting to see when their Python version comes out in a few months.

Anyway, I hope some of these comments were useful.

[eiderman]

Thanks for such a long and well thought out critique. I will spend a bit of time working most of these into the code base.

As part of this, I will deemphasize templates. Their use case is narrow and is usually better served by doing the following pattern:

def my_model_fn(input_, labels, phase):
  ...

with tf.variable_scope('model'):
  train_result = my_model_fn(train_input, train_labels, pt.Phase.train)

with tf.variable_scope('model', reuse=True):
  test_result = my_model_fn(test_input, test_labels, pt.Phase.test)

[Hvass-Labs]

Thanks I'm looking forward to that!

I'm currently studying TFLearn and it suffers from the same lack of documentation. It is rather gruelling to learn and there is high risk of the user misunderstanding the API.

I saw a short and good video from one of the advocates of clean code, explaining why proper API documentation is very important: https://www.youtube.com/watch?v=Y-_oYR7e9cY

TFLearn seems to be more popular than PrettyTensor, but in some ways PrettyTensor appears more elegant, so I think there's room for both. TFLearn mimics scikit-learn in some ways, but it seems rather straight-forward to add similar functionality to PrettyTensor, and perhaps tailor it more to Neural Network learning.

The other day, I made a short video tutorial on PrettyTensor because that's how I initiallity like to learn things myself and none were available: https://www.youtube.com/watch?v=GCUfJQ_dec8