Result of eval_explanation_label.sh

[chandru4ni]

Hi Lisa,

We ran the script eval_explanation_label.sh and got the json file created in below path. generated_sentences/birds_fg_test/gve_models/explanation-label_1006_pred/beam1/generation_result.json

We wanted to know if this is final output for explanation labels for the birds data set you have considered. And we need to extract the labels from this json to display them in a more readable format. Or is there any additional step which we are missing to generate explanation labels? Please advice.

Rgds, Chandra.

[chandru4ni]

We have seen the json output and it contains the labels for each image in the data set.

Now we want to train the model with our data set and as per our inspection we need three files for training the model.

1. data/vocab.txt
2. data/descriptions_bird.train_noCub.fg.json
3. data/description_sentence_features.p

As of now you have stored these files in your path and using them and allowing us to download. But since want to train the model for our dataset we want to know how to generate the description_sentence_features.p file. Please suggest.

Is there anything we need to have before we train your model with our data set? We will continue to look for details from our end.

[LisaAnne]

Hi!

description_sentence_features.p should just be a label embedding for different classes. You can just use one hot vectors (performance is about the same and it will be easier to create).

Cheers,

Lisa

On Thu, Feb 22, 2018 at 4:11 AM, chandru4ni notifications@github.com wrote:

We have seen the json output and it contains the labels for each image in the data set.

Now we want to train the model with our data set and as per our inspection we need three files for training the model.

1. data/vocab.txt
2. data/descriptions_bird.train_noCub.fg.json
3. data/description_sentence_features.p

As of now you have stored these files in your path and using them and allowing us to download. But since want to train the model for our dataset we want to know how to generate the description_sentence_features.p file. Please suggest.

Is there anything we need to have before we train your model with our data set? We will continue to look for details from our end.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/LisaAnne/ECCV2016/issues/5#issuecomment-367632392, or mute the thread https://github.com/notifications/unsubscribe-auth/AFQm2iEjsqX1UCQ2vbAbIX312QXNqf_Rks5tXT05gaJpZM4SNiDj .

[chandru4ni]

Thanks Lisa.

Yes you are right that description_sentence_features.p is a label embedding for different classes (200 X 1000 numpy array). You have told to use one of the hot vectors for this. If I am not wrong, I understand from your paper that you are using Compact Bilinear Pooling to get this vector. Though I spent some time on inspecting the code base I was not able to find the code to generate this vector even for your dataset.

So I need your help here to understand the flow since I am planning to use my own data set.

regards, Chandra.

[chandru4ni]

Is my understanding that description_sentence_features.p is representation of features (1000) for 200 classes of birds correct?

[LisaAnne]

I think this is the code you want: https://github.com/LisaAnne/ECCV2016/blob/ECCV2016/examples/ECCV2016/make_class_embedding.sh

However, it is probably easier to encode classes as one hot embeddings.

On Fri, Feb 23, 2018 at 11:23 AM, chandru4ni notifications@github.com wrote:

Is my understanding that description_sentence_features.p is representation of features (1000) for 200 classes of birds correct?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/LisaAnne/ECCV2016/issues/5#issuecomment-368077858, or mute the thread https://github.com/notifications/unsubscribe-auth/AFQm2vdHyyxQ8pQjANYU_w35wCf_w2GBks5tXvQGgaJpZM4SNiDj .

[chandru4ni]

Thanks Lisa. I saw the code of extract_val.py and yes it is generating the label embeddings and saving the file as pickle file. But while I ran the script to generate the pickle file, it is crashing. While debugging I observed that it is crashing while processing through the net at net.forward() line in extract_val.py.

Below are my few observations regarding make_class_embedding.sh script. Below are the inputs to extract_val.py. image_net=examples/ECCV2016/prototxt/deploy.prototxt #word_net=examples/ECCV2016/prototxt/wtd_1000_all.prototxt word_net=examples/ECCV2016/prototxt/wtd_1000.prototxt

Chandra

#model=snapshots/description_1006 model=gve_models/description_1006

I see that word_net file name is wtd_1000_all.prototxt and it does not exist in prototxt directory as part of downloaded files. But a file named wtd_1000.prototxt file is in prototxt directory, so I have changed to that file name. And the model filename is snapshots/description_1006 but I do not see any file in snapshots directory. And description_iter_1000.cafemodel is generated when we run train_description.sh in snapshots directory. We have description_1006.caffemodel downloaded from your site at gve_models. So I have changed the model to gve_models/description_1006.caffemodel

So I wanted to check if these changes are causing the crash. If yes, are we then missing wet_1000_all.prototxt and snapshots/description_1006.caffemodel files?

If not, I will continue debugging the code to see the cause of the crash. I request your thoughts on above and if possible you can throw light on the cause of the crash.

regards, Chandra.

[chandru4ni]

So my question if the changes i have done are causing the crash? And I am unable to find the file names which you have given in the make_class_embedding.sh. Are they missing in your code base by any chance? If possible you can let me know reason of the crash for any other reason. I will continue debugging the code for further analysis.

regards, Chandra.

[chandru4ni]

Since I have changed the file names in make_class_embedding.sh as below, i want to confirm before I proceed with further debugging. The lines in bold are the original lines and below them are modified by me. As the original files are not available to compare I want your inputs in this regard.

#word_net=examples/ECCV2016/prototxt/wtd_1000_all.prototxt word_net=examples/ECCV2016/prototxt/wtd_1000.prototxt

#model=snapshots/description_1006 model=gve_models/description_1006

regards, Chandra.

[chandru4ni]

I changed the script as below and used wtd_1000_20words.prototxt file rather than wtd_1000_all.prototxt file which you had used in the script. With this make_class_embedding.sh ran successfully. word_net=examples/ECCV2016/prototxt/wtd_1000_20words.prototxt

I generated the model with generated label embeddings using train_explanation_label.sh. But when I tried to use this generated model for for evaluation script eval_explanation-label.sh, I got all captions as same for all birds and that it is not some symbols as given below.

"196.House_Wren/House_Wren_0057_187157.jpg", "caption": "\" & \"."}, {"image_id": "099.Ovenbird/Ovenbird_0070_93140.jpg", "caption": "\" & \"."}, {"image_id": "135.Bank_Swallow/Bank_Swallow_0018_129891.jpg", "caption": "\" & \"."}, {"image_id": "117.Clay_colored_Sparrow/Clay_Colored_Sparrow_0068_110706.jpg", "caption": "\" & \"."}

Looks like something wrong in the prototxt file I used or the steps I am following. Please advice with your suggestion.

regards, Chandra.

[chandru4ni]

Corrected the comments which had some typos.

I changed the script as below and used wtd_1000_20words.prototxt file rather than wtd_1000_all.prototxt file in make_class_embedding.sh script. With this make_class_embedding.sh ran successfully. word_net=examples/ECCV2016/prototxt/wtd_1000_20words.prototxt

I generated the model with label embeddings using train_explanation_label.sh. But when I tried to use this generated model for for evaluation script eval_explanation-label.sh, I got all captions as same for all bird images and that it has got some symbols rather than words as given below.

"196.House_Wren/House_Wren_0057_187157.jpg", "caption": "" & "."}, {"image_id": "099.Ovenbird/Ovenbird_0070_93140.jpg", "caption": "" & "."}, {"image_id": "135.Bank_Swallow/Bank_Swallow_0018_129891.jpg", "caption": "" & "."}, {"image_id": "117.Clay_colored_Sparrow/Clay_Colored_Sparrow_0068_110706.jpg", "caption": "" & "."}

Looks like something wrong in the prototxt file I used or the steps which I am following. Please advice with your suggestion.

regards, Chandra.

[chandru4ni]

Hi Lisa,

Now I am getting the captions correctly by using the make_class_embedding.sh script and using the deploy.prototxt and wtd_1000.prototxt files. And this script requires snapshots/description_1006 model to be available. But if we want to run this script for the first time then we will not have the model available, hence as you indicated earlier I wanted to generate the label embeddings using word2vec or Grove method. The output of your make_class_embedding.sh script is 200X1000 numpy array and I assume 200 corresponds to classes of birds. In that case word2vec and Grove provides word embeddings. Hence I am thinking on how to get the class embeddings required for our explanation model. Any suggestions from your end will be helpful.

regards, Chandra.

[chandru4ni]

Hi Lisa,

Now I am getting the captions correctly by using the make_class_embedding.sh script and using the deploy.prototxt and wtd_1000.prototxt files. And this script requires snapshots/description_1006 model to be available. But if we want to run this script for the first time then we will not have the model available, so I thought of generating the label embeddings using word2vec or Grove method. The output of your make_class_embedding.sh script is 200X1000 numpy array and I assume 200 corresponds to classes of birds. In that case using word2vec and Grove we get word embeddings and not mapped to classes. Hence I am thinking on how to get the class embeddings required for our explanation model, should I look for other tools? I will continue exploring from my end, just wrote to you so that any suggestions from your end would be helpful.

regards, Chandra.

[LisaAnne]

Hello,

You should probably retrain a model with your data and then follow the same steps I did to make my class embeddings.

On Wed, Feb 28, 2018 at 7:52 PM, chandru4ni notifications@github.com wrote:

Hi Lisa,

Now I am getting the captions correctly by using the make_class_embedding.sh script and using the deploy.prototxt and wtd_1000.prototxt files. And this script requires snapshots/description_1006 model to be available. But if we want to run this script for the first time then we will not have the model available, so I thought of generating the label embeddings using word2vec or Grove method. The output of your make_class_embedding.sh script is 200X1000 numpy array and I assume 200 corresponds to classes of birds. In that case using word2vec and Grove we get word embeddings and not mapped to classes. Hence I am thinking on how to get the class embeddings required for our explanation model, should I look for other tools? I will continue exploring from my end, just wrote to you so that any suggestions from your end would be helpful.

regards, Chandra.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/LisaAnne/ECCV2016/issues/5#issuecomment-369466166, or mute the thread https://github.com/notifications/unsubscribe-auth/AFQm2mCzekRrpTkXEIWNd4wYAaTQFdVrks5tZ3B7gaJpZM4SNiDj .

[chandru4ni]

Thanks Lisa, yeah I am setting the data set to train the model. In the mean time I found that right now the code is able to provide visual explanation for all birds that belong to bilinear_pred.p master data set. If I give a new bird which does not belong to master data set then it is not identifying it correcty. Is it a prerequisite for the project to have the bird belong to only to this data set, since your research is more specific to provide visual explanations rather the classifying the image or identifying the image. But when I go for my data set, I need to first preprocess the image to get the descriptors to identify the image. Right now the current code is not working for preprocessing the image. Let me know if my understanding is correct so that I will continue my exploration on getting my image preprocessed to get descriptors through the image net.

Regards, Chandra.

[chandru4ni]

Thanks Lisa, yeah I am working on the data set to train the model and generate the embeddings as you indicated above. In the mean time I found that right now the code is able to provide visual explanation for all birds that belong to bilinear_pred.p master data set. If I give a new bird which does not belong to master data set then it is not identifying it correctly. Is it a prerequisite for the project to have the bird belong to only to this data set, since your research is more specific to provide visual explanations rather the classifying the image or identifying the image. But when I go for my data set, I need to first preprocess the image to get the descriptors to identify the image. Right now the current code is not working for preprocessing the image. Let me know if my understanding is correct so that I will continue my exploration on getting my image preprocessed to get descriptors through the image net.

Regards, Chandra.

[chandru4ni]

Hi Lisa, Finally we are able to use your visual explanation code component for fight data set. Below are few customizations we have done. Need to check your feedback on the same.

The .p file which is used for training we have generated it using Doc2vec model of genim. We just feed in one sentence for each class and generate a 1X1000 vector or sentence embedding corresponding to each class.

We were able to generate the .p using make_class_embedding.sh script. We first gave one shot vector as input during training in explanation-label.prototxt file as below. (10 corresponds to number of classes) \'size_onehot\' : 10

And tried to use the above trained model in make_class_embedding.sh to generate .p with average of probabilities per class. But when we use this .p for training it was giving 'nan' for loss value during the 20 iteration itself. Tried changing value of size_onehot but did not help. We require your help here. What is the value of size_onehot we need to use in prototxt file and what does it correspond to?

Since make_class_embedding.sh generated .p did not work for us we used the sentence embedding vector generated by Doc2vec model and trained the explanation-label model and saw reducing loss.

We were able to successfully train and evaluate the flight dataset with good accuracy.

We want your feedback on using .p and if you can help with your advice on using size_onehot in prototxt file.

Thanks in advance.

[chandru4ni]

We are now looking into discriminative loss model of yours. This has multiple layers and looks complex. We will keep you updated on this.

regards, Chandra.

[LisaAnne]

Hi --

Yes, the discriminative loss is a bit tricky, especially in Caffe (to backprop through REINFORCE you need to be a bit creative with the existing layers).

It seems like your understanding of size_onehot is correct. It is possible that the sentence embedding for Doc2vec is better for your problem, so I would not worry about this too much. I played around with different embeddings (including just a one hot vector indicating a class) and the results were all pretty similar.

[chandru4ni]

Oh ok. That's seems convincing :-). Thanks.

Your work on discriminative loss is very interesting, we will try to explore from our end and will get back for any help.

[sandareka]

Hi @chandru4ni

I'm trying to run Lisa's code on a different dataset. But when I tried I got the error "This program requires version 3.4.0 of the Protocol Buffer runtime library, but the installed version is 2.6.1.". Did you also run into this error? If so may I please know how you could resolve it?

I hope you follow the steps given by Lisa and could train the model.

Thank you in advance!

[chandru4ni]

Hi Sandareka,

Yes I was able to run the Lisa's code for multiple dataset. I built the code with Protocol Buffer 2.6.0.

Thanks, Chandra

On Tue 4 Sep, 2018, 10:22 PM Sandareka Wickramanayake, < notifications@github.com> wrote:

Hi @chandru4ni https://github.com/chandru4ni

I'm trying to run Lisa's code on a different dataset. But when I tried I got the error "This program requires version 3.4.0 of the Protocol Buffer runtime library, but the installed version is 2.6.1.". Did you also run into this error? If so may I please know how you could resolve it?

I hope you follow the steps given by Lisa and could train the model.

Thank you in advance!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LisaAnne/ECCV2016/issues/5#issuecomment-418440585, or mute the thread https://github.com/notifications/unsubscribe-auth/Ah33kc5ZHKKIurBMG2HgfwJ4HJ0nQTy-ks5uXq_ZgaJpZM4SNiDj .

[sandareka]

Hi @chandru4ni ,

Thank you for your reply. In my case the problem was I have two python versions, 2.7 and 3.5 installed in my system and mistakenly I have had installed protobuf 3.4.0 for python 2.7. When I removed it and installed protobuf 2.6.1 for python 2.7 everything works fine.

Hope this will help someone in the future.

Thank you @chandru4ni and @LisaAnne for your help.

[LisaAnne]

Great! Good luck with your experiments.

On Tue, Sep 11, 2018, 9:58 AM Sandareka Wickramanayake < notifications@github.com> wrote:

Hi @chandru4ni https://github.com/chandru4ni ,

Thank you for your reply. In my case the problem was I have two python versions, 2.7 and 3.5 installed in my system and mistakenly I have had installed protobuf 3.4.0 for python 2.7. When I removed it and installed protobuf 2.6.1 for python 2.7 everything works fine.

Hope this will help someone in the future.

Thank you @chandru4ni https://github.com/chandru4ni and @LisaAnne https://github.com/LisaAnne for your help.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LisaAnne/ECCV2016/issues/5#issuecomment-420183085, or mute the thread https://github.com/notifications/unsubscribe-auth/AFQm2rAR3x8yM2cNm6KeGRSvFH2Q3lX_ks5uZ20NgaJpZM4SNiDj .