Open JGoncaloPereira opened 4 years ago
I found out what the problem was, but now i have the problem of not being able to find the output layer. I printed all layer nodes and selected one, but can you tell me how to find out whats the proper name of the outpu and input layer? because i just chose the last one for the output ("layer6/logits/kernel") but the input one im not sure which one it should be or how to find it
Similar to issue #92 . Can you provide all the node names?
Here are the names that appear to me as node names: beta1_power (DT_FLOAT) [] beta2_power (DT_FLOAT) [] global_step (DT_INT64) [] layer1/conv2d/bias (DT_FLOAT) [32] layer1/conv2d/bias/Adam (DT_FLOAT) [32] layer1/conv2d/bias/Adam_1 (DT_FLOAT) [32] layer1/conv2d/kernel (DT_FLOAT) [3,3,3,32] layer1/conv2d/kernel/Adam (DT_FLOAT) [3,3,3,32] layer1/conv2d/kernel/Adam_1 (DT_FLOAT) [3,3,3,32] layer2/conv2d/bias (DT_FLOAT) [64] layer2/conv2d/bias/Adam (DT_FLOAT) [64] layer2/conv2d/bias/Adam_1 (DT_FLOAT) [64] layer2/conv2d/kernel (DT_FLOAT) [3,3,32,64] layer2/conv2d/kernel/Adam (DT_FLOAT) [3,3,32,64] layer2/conv2d/kernel/Adam_1 (DT_FLOAT) [3,3,32,64] layer2/conv2d_1/bias (DT_FLOAT) [64] layer2/conv2d_1/bias/Adam (DT_FLOAT) [64] layer2/conv2d_1/bias/Adam_1 (DT_FLOAT) [64] layer2/conv2d_1/kernel (DT_FLOAT) [3,3,64,64] layer2/conv2d_1/kernel/Adam (DT_FLOAT) [3,3,64,64] layer2/conv2d_1/kernel/Adam_1 (DT_FLOAT) [3,3,64,64] layer3/conv2d/bias (DT_FLOAT) [64] layer3/conv2d/bias/Adam (DT_FLOAT) [64] layer3/conv2d/bias/Adam_1 (DT_FLOAT) [64] layer3/conv2d/kernel (DT_FLOAT) [3,3,64,64] layer3/conv2d/kernel/Adam (DT_FLOAT) [3,3,64,64] layer3/conv2d/kernel/Adam_1 (DT_FLOAT) [3,3,64,64] layer3/conv2d_1/bias (DT_FLOAT) [64] layer3/conv2d_1/bias/Adam (DT_FLOAT) [64] layer3/conv2d_1/bias/Adam_1 (DT_FLOAT) [64] layer3/conv2d_1/kernel (DT_FLOAT) [3,3,64,64] layer3/conv2d_1/kernel/Adam (DT_FLOAT) [3,3,64,64] layer3/conv2d_1/kernel/Adam_1 (DT_FLOAT) [3,3,64,64] layer4/conv2d/bias (DT_FLOAT) [128] layer4/conv2d/bias/Adam (DT_FLOAT) [128] layer4/conv2d/bias/Adam_1 (DT_FLOAT) [128] layer4/conv2d/kernel (DT_FLOAT) [3,3,64,128] layer4/conv2d/kernel/Adam (DT_FLOAT) [3,3,64,128] layer4/conv2d/kernel/Adam_1 (DT_FLOAT) [3,3,64,128] layer4/conv2d_1/bias (DT_FLOAT) [128] layer4/conv2d_1/bias/Adam (DT_FLOAT) [128] layer4/conv2d_1/bias/Adam_1 (DT_FLOAT) [128] layer4/conv2d_1/kernel (DT_FLOAT) [3,3,128,128] layer4/conv2d_1/kernel/Adam (DT_FLOAT) [3,3,128,128] layer4/conv2d_1/kernel/Adam_1 (DT_FLOAT) [3,3,128,128] layer5/conv2d/bias (DT_FLOAT) [256] layer5/conv2d/bias/Adam (DT_FLOAT) [256] layer5/conv2d/bias/Adam_1 (DT_FLOAT) [256] layer5/conv2d/kernel (DT_FLOAT) [3,3,128,256] layer5/conv2d/kernel/Adam (DT_FLOAT) [3,3,128,256] layer5/conv2d/kernel/Adam_1 (DT_FLOAT) [3,3,128,256] layer6/dense/bias (DT_FLOAT) [1024] layer6/dense/bias/Adam (DT_FLOAT) [1024] layer6/dense/bias/Adam_1 (DT_FLOAT) [1024] layer6/dense/kernel (DT_FLOAT) [1024,1024] layer6/dense/kernel/Adam (DT_FLOAT) [1024,1024] layer6/dense/kernel/Adam_1 (DT_FLOAT) [1024,1024] layer6/logits/bias (DT_FLOAT) [14] layer6/logits/bias/Adam (DT_FLOAT) [14] layer6/logits/bias/Adam_1 (DT_FLOAT) [14] layer6/logits/kernel (DT_FLOAT) [1024,14] layer6/logits/kernel/Adam (DT_FLOAT) [1024,14] layer6/logits/kernel/Adam_1 (DT_FLOAT) [1024,14]
I set raw_tensor as true but still the same problem, when i try to set any node as input it says it does not exist in the graph
Edit: i cant freeze any layers that have Adam on it, so i was only able to freeze 21 nodes (i also couldn't freeze beta1_power and beta2_power). In all cases it says it is not in the graph when i try to freeze.
Just a little information i forgot, in order to get the names of the nodes i used the python tool called inspect_checkpoint.py because if i use the one you mentioned (butterfly) it only gives me:
butterfly/layer6/logits/kernel
Which in a way makes sense because theats the only layer it gives me access to as input or output anything else it says it does not exist
Update: i found out what the error was, i forgot to add all tensors, but i still cant figure it out which one is supposed to be the input layer.
@JGoncaloPereira Input layer will be image_tensor. Have you tried using tensorboard?
That doesnt work, i tried it also, the only node that could possibly make sense to be the input i would say it is global_step. However it appear the same problem as the issue mentioned by @yinguobing it says the size (3,128,128) is different than (), but setting raw_tensor to True (that solved the problem for the other case) makes no difference here.
@ZlaaM Have you written raw_input True while training ?
I have trained with my own dataset using the same scripts today 2 times, and for me, all things work correctly.
@Rutvik21 can you show, if possible what is your node names?
@Rutvik21
no
parser.add_argument('--raw_input', default=False, type=bool,
help='Use raw tensor as model input.')
@ZlaaM Then try with True
@Rutvik21 ok, I will try thank you
@JGoncaloPereira Here- butterfly/image_tensor butterfly/map/Shape butterfly/map/strided_slice/stack butterfly/map/strided_slice/stack_1 butterfly/map/strided_slice/stack_2 butterfly/map/strided_slice butterfly/map/TensorArray butterfly/map/TensorArrayUnstack/Shape butterfly/map/TensorArrayUnstack/strided_slice/stack butterfly/map/TensorArrayUnstack/strided_slice/stack_1 butterfly/map/TensorArrayUnstack/strided_slice/stack_2 butterfly/map/TensorArrayUnstack/strided_slice butterfly/map/TensorArrayUnstack/range/start butterfly/map/TensorArrayUnstack/range/delta butterfly/map/TensorArrayUnstack/range butterfly/map/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3 butterfly/map/Const butterfly/map/TensorArray_1 butterfly/map/while/iteration_counter butterfly/map/while/Enter butterfly/map/while/Enter_1 butterfly/map/while/Enter_2 butterfly/map/while/Merge butterfly/map/while/Merge_1 butterfly/map/while/Merge_2 butterfly/map/while/Less/Enter butterfly/map/while/Less butterfly/map/while/Less_1 butterfly/map/while/LogicalAnd butterfly/map/while/LoopCond butterfly/map/while/Switch butterfly/map/while/Switch_1 butterfly/map/while/Switch_2 butterfly/map/while/Identity butterfly/map/while/Identity_1 butterfly/map/while/Identity_2 butterfly/map/while/add/y butterfly/map/while/add butterfly/map/while/TensorArrayReadV3/Enter butterfly/map/while/TensorArrayReadV3/Enter_1 butterfly/map/while/TensorArrayReadV3 butterfly/map/while/resize/ExpandDims/dim butterfly/map/while/resize/ExpandDims butterfly/map/while/resize/size butterfly/map/while/resize/ResizeBilinear butterfly/map/while/resize/Squeeze butterfly/map/while/TensorArrayWrite/TensorArrayWriteV3/Enter butterfly/map/while/TensorArrayWrite/TensorArrayWriteV3 butterfly/map/while/add_1/y butterfly/map/while/add_1 butterfly/map/while/NextIteration butterfly/map/while/NextIteration_1 butterfly/map/while/NextIteration_2 butterfly/map/while/Exit_2 butterfly/map/TensorArrayStack/TensorArraySizeV3 butterfly/map/TensorArrayStack/range/start butterfly/map/TensorArrayStack/range/delta butterfly/map/TensorArrayStack/range butterfly/map/TensorArrayStack/TensorArrayGatherV3 butterfly/layer1/conv2d/kernel butterfly/layer1/conv2d/kernel/read butterfly/layer1/conv2d/bias butterfly/layer1/conv2d/bias/read butterfly/layer1/conv2d/Conv2D butterfly/layer1/conv2d/BiasAdd butterfly/layer1/conv2d/Relu butterfly/layer1/max_pooling2d/MaxPool butterfly/layer2/conv2d/kernel butterfly/layer2/conv2d/kernel/read butterfly/layer2/conv2d/bias butterfly/layer2/conv2d/bias/read butterfly/layer2/conv2d/Conv2D butterfly/layer2/conv2d/BiasAdd butterfly/layer2/conv2d/Relu butterfly/layer2/conv2d_1/kernel butterfly/layer2/conv2d_1/kernel/read butterfly/layer2/conv2d_1/bias butterfly/layer2/conv2d_1/bias/read butterfly/layer2/conv2d_1/Conv2D butterfly/layer2/conv2d_1/BiasAdd butterfly/layer2/conv2d_1/Relu butterfly/layer2/max_pooling2d/MaxPool butterfly/layer3/conv2d/kernel butterfly/layer3/conv2d/kernel/read butterfly/layer3/conv2d/bias butterfly/layer3/conv2d/bias/read butterfly/layer3/conv2d/Conv2D butterfly/layer3/conv2d/BiasAdd butterfly/layer3/conv2d/Relu butterfly/layer3/conv2d_1/kernel butterfly/layer3/conv2d_1/kernel/read butterfly/layer3/conv2d_1/bias butterfly/layer3/conv2d_1/bias/read butterfly/layer3/conv2d_1/Conv2D butterfly/layer3/conv2d_1/BiasAdd butterfly/layer3/conv2d_1/Relu butterfly/layer3/max_pooling2d/MaxPool butterfly/layer4/conv2d/kernel butterfly/layer4/conv2d/kernel/read butterfly/layer4/conv2d/bias butterfly/layer4/conv2d/bias/read butterfly/layer4/conv2d/Conv2D butterfly/layer4/conv2d/BiasAdd butterfly/layer4/conv2d/Relu butterfly/layer4/conv2d_1/kernel butterfly/layer4/conv2d_1/kernel/read butterfly/layer4/conv2d_1/bias butterfly/layer4/conv2d_1/bias/read butterfly/layer4/conv2d_1/Conv2D butterfly/layer4/conv2d_1/BiasAdd butterfly/layer4/conv2d_1/Relu butterfly/layer4/max_pooling2d/MaxPool butterfly/layer5/conv2d/kernel butterfly/layer5/conv2d/kernel/read butterfly/layer5/conv2d/bias butterfly/layer5/conv2d/bias/read butterfly/layer5/conv2d/Conv2D butterfly/layer5/conv2d/BiasAdd butterfly/layer5/conv2d/Relu butterfly/layer6/flatten/Shape butterfly/layer6/flatten/strided_slice/stack butterfly/layer6/flatten/strided_slice/stack_1 butterfly/layer6/flatten/strided_slice/stack_2 butterfly/layer6/flatten/strided_slice butterfly/layer6/flatten/Reshape/shape/1 butterfly/layer6/flatten/Reshape/shape butterfly/layer6/flatten/Reshape butterfly/layer6/dense/kernel butterfly/layer6/dense/kernel/read butterfly/layer6/dense/bias butterfly/layer6/dense/bias/read butterfly/layer6/dense/MatMul butterfly/layer6/dense/BiasAdd butterfly/layer6/dense/Relu butterfly/layer6/logits/kernel butterfly/layer6/logits/kernel/read butterfly/layer6/logits/bias butterfly/layer6/logits/bias/read butterfly/layer6/logits/MatMul butterfly/layer6/logits/BiasAdd butterfly/layer6/final_dense
In your case the nodes seem pretty good and it all make sense, in my case not that much, not sure why
Can you please provide all the steps that you've followed ?
python landmark.py --train_record Output.record --val_record Validacao.record --model_dir train --train_steps 500 --batch_size 32 --export_dir teste --raw_tensor True
python freeze_graph.py --input_saved_model teste/1582800913 --output_node_names beta1_power,beta2_power,global_step,layer1/conv2d/bias,layer1/conv2d/bias/Adam,layer1/conv2d/bias/Adam_1,layer1/conv2d/kernel,layer1/conv2d/kernel/Adam,layer1/conv2d/kernel/Adam_1,layer2/conv2d/bias,layer2/conv2d/bias/Adam,layer2/conv2d/bias/Adam_1,layer2/conv2d/kernel,layer2/conv2d/kernel/Adam,layer2/conv2d/kernel/Adam_1,layer2/conv2d_1/bias,layer2/conv2d_1/bias/Adam,layer2/conv2d_1/bias/Adam_1,layer2/conv2d_1/kernel,layer2/conv2d_1/kernel/Adam,layer2/conv2d_1/kernel/Adam_1,layer3/conv2d/bias,layer3/conv2d/bias/Adam,layer3/conv2d/bias/Adam_1,layer3/conv2d/kernel,layer3/conv2d/kernel/Adam,layer3/conv2d/kernel/Adam_1,layer3/conv2d_1/bias,layer3/conv2d_1/bias/Adam,layer3/conv2d_1/bias/Adam_1,layer3/conv2d_1/kernel,layer3/conv2d_1/kernel/Adam,layer3/conv2d_1/kernel/Adam_1,layer4/conv2d/bias,layer4/conv2d/bias/Adam,layer4/conv2d/bias/Adam_1,layer4/conv2d/kernel,layer4/conv2d/kernel/Adam,layer4/conv2d/kernel/Adam_1,layer4/conv2d_1/bias,layer4/conv2d_1/bias/Adam,layer4/conv2d_1/bias/Adam_1,layer4/conv2d_1/kernel,layer4/conv2d_1/kernel/Adam,layer4/conv2d_1/kernel/Adam_1,layer5/conv2d/bias,layer5/conv2d/bias/Adam,layer5/conv2d/bias/Adam_1,layer5/conv2d/kernel,layer5/conv2d/kernel/Adam,layer5/conv2d/kernel/Adam_1,layer6/dense/bias,layer6/dense/bias/Adam,layer6/dense/bias/Adam_1,layer6/dense/kernel,layer6/dense/kernel/Adam,layer6/dense/kernel/Adam_1,layer6/logits/bias,layer6/logits/bias/Adam,layer6/logits/bias/Adam_1,layer6/logits/kernel,layer6/logits/kernel/Adam,layer6/logits/kernel/Adam_1 --output_graph frozen_graphs.pb
After this i deleted all adam nodes because they didnt froze neither beta1 and beta2
after that i tried to run it and it gives the problem mentioned above
Before all this i picked my own dataset and transformed it to TFrecord using @yinguobing method
Are u using the same landmark.py script which is in this GIT. becuase I am using raw_input instead of raw_tensor ? have u changed the name ?
i used raw_input aswell i made a mistake when passing the code to here and yes im using the same script thats here.
Mine tensorboard graph look like this -
What is it for u ?
And I think you're getting node names which could be seen in my graph at the right side of in the graph.
Im trying to use tensorboard to visualize, but i never worked with it so i wasnt able to make that representation
@ZlaaM can you send to me the way you used the tensorboard?
@JGoncaloPereira python -m tensorboard.main --logdir=" path of your train file" this is working for me with tensorflow = 1.9
using Anaconda prompt
Okay. So let's start with versions of tensorflow. Which version do you use ? I am using tensorflow 1.15 for training and 2.0 for prediction.
I tried in 2 versions, 1.9.0 and 1.15.0
I suggest using 1.15 for training.
I used it
@JGoncaloPereira For printing node names which command did u use ?
The ones above: python inspect_checkpoint.py --file_name train/model.ckpt-500 --all_tensor_names
I didn't know this one. I am using the demo.py file to print the node names. Can u try with that one ?
python3 demo.py --model frozen_model.pb --list_ops true Here
I also used that one, but that only gives the ones you froze, and i needed to check which one had on it but i can make that too
butterfly/global_step butterfly/layer1/conv2d/kernel butterfly/layer1/conv2d/bias butterfly/layer2/conv2d/kernel butterfly/layer2/conv2d/bias butterfly/layer2/conv2d_1/kernel butterfly/layer2/conv2d_1/bias butterfly/layer3/conv2d/kernel butterfly/layer3/conv2d/bias butterfly/layer3/conv2d_1/kernel butterfly/layer3/conv2d_1/bias butterfly/layer4/conv2d/kernel butterfly/layer4/conv2d/bias butterfly/layer4/conv2d_1/kernel butterfly/layer4/conv2d_1/bias butterfly/layer5/conv2d/kernel butterfly/layer5/conv2d/bias butterfly/layer6/dense/kernel butterfly/layer6/dense/bias butterfly/layer6/logits/kernel butterfly/layer6/logits/bias
These are the ones i was able to freeze
I don't know what could be the problem then.
Can u send sample data(with scripts) so that I can reproduce the same error that u get.
I am using tensorflow 1.13.1 for training and prediction.
@ZlaaM Please can u send the node names after running the above command I have suggested.
@JGoncaloPereira @ZlaaM And also send the output log after the training if possible. So that I can check.
Here you can find the tfrecords of train (Output.record) and validation (Validacao.record) the rest i did just has it says in the projects:
To create saved_model: python landmark.py --train_record Output.record --val_record Validacao.record --model_dir train --train_steps 500 --batch_size 32 --export_dir /teste --raw_input True
events.out.tfevents.1583756764.zip I think this is the log but i'm not sure, that is a question that im fighting with a lot
by execute the checkpoint_info.py from python I get these nodes name
tensor_name: layer1/conv2d/kernel/Adam tensor_name: layer5/conv2d/kernel/Adam_1 tensor_name: layer5/conv2d/bias/Adam_1 tensor_name: layer1/conv2d/bias/Adam tensor_name: layer6/dense/kernel tensor_name: layer3/conv2d_1/bias tensor_name: beta1_power tensor_name: global_step tensor_name: layer4/conv2d_1/kernel/Adam_1 tensor_name: layer4/conv2d_1/kernel/Adam tensor_name: layer4/conv2d_1/bias/Adam_1 tensor_name: layer4/conv2d_1/bias/Adam tensor_name: beta2_power tensor_name: layer6/dense/bias/Adam_1 tensor_name: layer1/conv2d/kernel tensor_name: layer3/conv2d/bias/Adam_1 tensor_name: layer1/conv2d/bias tensor_name: layer6/dense/bias tensor_name: layer1/conv2d/kernel/Adam_1 tensor_name: layer5/conv2d/kernel/Adam tensor_name: layer5/conv2d/bias/Adam tensor_name: layer3/conv2d/bias tensor_name: layer1/conv2d/bias/Adam_1 tensor_name: layer6/logits/bias/Adam_1 tensor_name: layer2/conv2d/bias tensor_name: layer2/conv2d/bias/Adam tensor_name: layer2/conv2d/kernel/Adam tensor_name: layer2/conv2d/bias/Adam_1 tensor_name: layer2/conv2d/kernel tensor_name: layer2/conv2d/kernel/Adam_1 tensor_name: layer2/conv2d_1/bias tensor_name: layer6/logits/kernel tensor_name: layer2/conv2d_1/bias/Adam tensor_name: layer2/conv2d_1/bias/Adam_1 tensor_name: layer6/logits/bias/Adam tensor_name: layer2/conv2d_1/kernel tensor_name: layer2/conv2d_1/kernel/Adam tensor_name: layer2/conv2d_1/kernel/Adam_1 tensor_name: layer3/conv2d/bias/Adam tensor_name: layer3/conv2d/kernel tensor_name: layer3/conv2d/kernel/Adam_1 tensor_name: layer3/conv2d/kernel/Adam tensor_name: layer6/dense/kernel/Adam_1 tensor_name: layer6/dense/kernel/Adam tensor_name: layer3/conv2d_1/bias/Adam_1 tensor_name: layer3/conv2d_1/bias/Adam tensor_name: layer3/conv2d_1/kernel tensor_name: layer3/conv2d_1/kernel/Adam tensor_name: layer3/conv2d_1/kernel/Adam_1 tensor_name: layer4/conv2d/bias tensor_name: layer4/conv2d/bias/Adam tensor_name: layer4/conv2d_1/kernel tensor_name: layer4/conv2d_1/bias tensor_name: layer4/conv2d/bias/Adam_1 tensor_name: layer4/conv2d/kernel tensor_name: layer4/conv2d/kernel/Adam tensor_name: layer4/conv2d/kernel/Adam_1 tensor_name: layer5/conv2d/kernel tensor_name: layer5/conv2d/bias tensor_name: layer6/dense/bias/Adam tensor_name: layer6/logits/bias tensor_name: layer6/logits/kernel/Adam tensor_name: layer6/logits/kernel/Adam_1
@JGoncaloPereira try this one tensorboard --logdir="the path"
@ZlaaM just to be sure... Where do you get your log? I wanna know if im getting it from the correct place Is it cnn-facial-landmark-master/train/eval?
@JGoncaloPereira just the train file like "cnn-facial-landmark-master/train"
It still does not open anything, after you open tensorboard do you add something?
no
Here you can find the tfrecords of train (Output.record) and validation (Validacao.record) the rest i did just has it says in the projects:
To create saved_model: python landmark.py --train_record Output.record --val_record Validacao.record --model_dir train --train_steps 500 --batch_size 32 --export_dir /teste --raw_input True
@JGoncaloPereira Here you have provided train as the model_dir. So after the training is completed run the following command: tensorboard --log_dir=train
@JGoncaloPereira @ZlaaM And also send the output log after the training if possible. So that I can check.
And by logs I mean the output that you are seeing while running the train.py script.
@Rutvik21 meaning the checkpoint file ?
No the output like this -
I got a problem when trying to freeze the graph. It asks me for the output node, which you have "logits" as default. However that is not the correct name and it actually says its not in graph. So i searched in graph.pbtxt the name that included logits and that could be the answer to my problem, unfortunately i didn't find it, i tried layer6/logits/kernel/Initializer/random_uniform/shape, it was able to make a frozen pb but when i tried to run it it did not work. I hope you can help me with that. Thanks in advance for the time spent.