flauted
commented
6 years ago @barbansonw Thanks for moving over here. Just to make sure we're on the same page, let's go back to the beginning for a second...
In maketfr we read the images and labels off of the disk into numpy arrays in the Python workspace. I recommend doing resizing and cropping at this point. That way, the images we write into the TFR file are the same shape. Then we write each example into the TFR. We call the function and voila, we have a TFR file to use. I believe you're fine with this part.
In input_pipeline we set up an object - a dataset - that will read the TFR data straight into Tensors. That's what parse_protocol_buffer does. Then some technical code to convert the image from bytes into a uint8 Tensor and restore the original shape. Right after the reshape, we could use tf.image to do some simple preprocessing. For instance, convert RGB to HSV, adjust brightness, etc. I choose to normalize my image so that each element is in the interval [-1, 1]. Whatever you do, it's important to convert images to the float datatype so we can do real-valued math with them. In uint8 format, the image tensors can only do integer math. Not very useful! I recommend not changing shape of the input image as it flows from the TFR file into the Dataset. Again, I think you're fine with this.
An aside, why did I emphasize "will"? Remember maketfr was literally changing numpy arrays in the Python workspace. But, input_pipeline isn't really doing anything in Python other than creating that Dataset object. Tensors aren't being read off of the TFR until a later time. I like to think of defining the function as making the plumbing. We install the plumbing when we call the function. We turn on the water when we get to the Session. I'm going to keep with this metaphor because it makes a lot of sense to me.
I think it's best to explain now two steps I tuck away closer to the Session. When we call input_pipeline, the returned Dataset is still missing a little bit of plumbing. We actually need to turn it into an iterable. The Dataset instance has a method called make_initializable_iterator(). So we just call that. There's step 1. Next, we get the images and labels out. The instance now has a method called get_next(). So, we call that. That's step 2. I'm not sure if you noticed this in the example or not.
train_dataset = input_pipeline(
batch_size, iterations, tfr_file=TFR_FILE)
with tf.name_scope("Input"):
train_iterator = train_dataset.make_initializable_iterator()
batch_image, batch_label = train_iterator.get_next()At this point, the input pipeline is part of the graph. The Python batch_image, batch_label returned from get_next() still aren't full of data. They're our access points to hook up the inputs to the rest of the plumbing.
Let's pick back up at def model(image_tensor, ...), which is where the handling takes place. Sometimes this is referred to as the inference part of the graph. In the example, I defined a simple convolutional neural network. Nothing particularly fancy. Here's the key to understanding the handling of the data: We're going to pass batch_image straight into model when we call model to install the convolutional neural network. That is, we've finished writing the model function. When we call it, we simply say
logits = model(batch_image, ...)Now model is part of our graph! We installed that plumbing. There's still no actual data in logits, but the graph can see how to turn batch_image into predictions. In other words, when we installed the CNN plumbing, we hooked it up to batch_image. What comes out of that chunk of pipe is a prediction.
The labels get into the cost/error/loss (all names for the same thing) function the exact same way. We write up a function - make some plumbing - called loss that compares the predictions - logits - with the labels. So,
loss_op = loss(logits=logits, labels=batch_label)We installed a loss function into our graph! The plumbing here is hooked up to the input pipes, i.e. batch_label, and the end of the CNN pipes, i.e. logits.
Now on top of it all, we put the training. I chose an AdamOptimizer because it's common, but the API for all the training operations is more or less the same. We just install the training pipes right after the loss. That is,
training_op = train(loss_op, ...)Easy enough. The training gets the error data and adjusts the variables in the model part of the graph to minimize error!
(You might be wondering how the AdamOptimizer accesses all the variables we defined in model to optimize them. After all, we didn't explicitly hook up that part of the plumbing. Don't worry too much about it - unless you're writing a GAN. If you're still curious, under the hood, when we define a variable with tf.Variable or tf.get_variable, TensorFlow automatically adds them to a list called tf.GraphKeys.TRAINABLE_VARIABLES (and tf.GraphKeys.GLOBAL_VARIABLES ... 99% of the time, sometimes a stray gets into tf.GraphKeys.LOCAL_VARIABLES, but not in my example). Objects in tf.train, including AdamOptimizer, access the tf.GraphKeys.TRAINABLE_VARIABLES list that has all the variables in it. When we call minimize, it creates the plumbing from the loss_op to each variable in that list. Specifically, for each variable, it calculates the error gradients w.r.t the variable and creates a tf.update op that reassigns the variable a new value that should minimize loss according to the *prop algorithm (backprop, RMSprop, etc). It puts all those update pipes into a list that it returns as training_op. But, that's more than we need to worry about.)
Now we have all the plumbing ready! We have something installed to transport water from the city's underground pipeline - Dataset.get_next() reads batch_image and batch_label from the TFR. We have plumbing installed to filter the water - logits = model(batch_image) makes predictions. And we have loss and training plumbing. Great! Our pipework is ready for us to turn on the water.
First thing's first, we have to put actual values into our variables. If you read the note about how training accesses variables, it should make sense why we call global_variables_initializer(). If you didn't, just know there's a list of all the variables you defined hiding under the hood. Then, we fill our input pipes with water by calling the initializer. And finally, we can run our training loop. Calling train_op evaluates the relevant parts of the graph to train (which is everything...). So, it'll call up some new data out of the iterator, run the CNN operations, calculate the loss, and update all the variables to minimize error. The trainop doesn't really return anything (actually it returns None) to Python, so we just let it unpack into ``. However, every time the model has trained on 100 examples, we want to get some insight into how it's doing. So, we tell sess we want it to run the graph up to loss_op and give us the value back in Python. We print it.
with tf.Session() as sess:
# Initialize all the tf.get_variable()s
sess.run(tf.global_variables_initializer())
sess.run(train_iterator.initializer)
for epoch in range(iterations + 1):
check_in = epoch % 100 == 0
feed_dict = {keep_prob: 0.5} # In the example I'm using dropout.
_ = sess.run(
[train_op], feed_dict=feed_dict)
train_writer.add_summary(summary, epoch)
if check_in:
curr_loss = sess.run([loss_op], feed_dict=feed_dict)
print(epoch, curr_loss)
train_writer.close()Summary/Tl;dr
To wrap it up in a sentence, the best way to handle the data that comes out of the Dataset is to construct the graph so that the data flows right into the neural network. No placeholder. No explicit calling in the session. The back-end magically handles calling up new data into batch_image and batch_label.
I hope this answers your question. But, I realize explanations are not one-size-fits-all. If this made no sense to you and didn't help you in the slightest, I'll try again. Just let me know and ask the most specific question you can.
DISCLAIMER: I'm not an expert. I haven't contributed to TensorFlow nor do I have any formal education in machine learning. My only qualifications for answering your questions are 1) I was a novice to TensorFlow 6 months ago, and 2) I've made working input pipelines and models - the old queue way, the Placeholder way, and the Dataset way.
barbansonw
Hi,
We've talked about your input pipeline in StackOverflow and since you advised me to open an issue here, here I am. You've helped me a lot already but I would like to know more about the actual handling of images and labels. With the code you provided me we end the input pipeline with a dataset, just like your code here. Your code here is a bit too much for me to fully understand so I'm wondering what's the bets way to go about handling the data we get out of that input.