search

karpathy / neuraltalk

NeuralTalk is a Python+numpy project for learning Multimodal Recurrent Neural Networks that describe images with sentences.
5.4k stars 1.32k forks source link

python port of features matlab code #7

Closed subhasis256 closed 9 years ago

subhasis256 commented 9 years ago

This adds the ability to run a caffe pretrained model through Python to get the features.

karpathy commented 9 years ago

Hi Subhasis, thanks for the pull! Before we merge, could we please double check that the Matlab and Python features are identical, after image resizing? I.e. Take the Matlab image (after resizing), and run exactly that volume through ConvNet on both sides.

The problem could be that the channels / xy flips etc are somehow not correctly configured, and could be giving wrong results at the end.

EDIT: we want to do after resizing because somehow the Matlab/Python 'BICUBIC' doesn't give identical results. It's possible this is influencing final features.

EDIT2: Saw your Piazza post, which sounds like exactly what I suggested here. Thanks!

karpathy commented 9 years ago

Wait, should this also contain the def file? It's alluded to in readme but doesn't appear in the commit

subhasis256 commented 9 years ago

The def file should be the prototxt file that you download from the VGG model website or some other network. Maybe I should call it the caffe prototxt file?

Ahh sorry... now I see that you already have the prototxt files in the matlab directory. I will add them.

karpathy commented 9 years ago

I was alluding to this line:

"Note that I provide my _features deploy network def as well, which is exactly what you see on that page but I chopped off the softmax to get the 4096-D codes below."

in your readme. It makes it sound like you are attaching the deploy net to the commit?

karpathy commented 9 years ago

RE: what's inside matlab directory you have to be careful! Caffe has changed many proto fields, it might give errors if you try to use this. Or it gets automatically converted? I haven't tried any of this yet.

subhasis256 commented 9 years ago

Got it. I actually used this version of the file myself with caffe which was cloned on Jan 17 and it works.

On Tue, Feb 24, 2015 at 5:07 PM, Andrej notifications@github.com wrote:

RE: what's inside matlab directory you have to be careful! Caffe has changed many proto fields, it might give errors if you try to use this. Or it gets automatically converted? I haven't tried any of this yet.

— Reply to this email directly or view it on GitHub https://github.com/karpathy/neuraltalk/pull/7#issuecomment-75885814.

Subhasis

subhasis256 commented 9 years ago

Added new pull request with the prototxt file.

On Tue, Feb 24, 2015 at 5:09 PM, Subhasis Das subhasis256@gmail.com wrote:

Got it. I actually used this version of the file myself with caffe which was cloned on Jan 17 and it works.

On Tue, Feb 24, 2015 at 5:07 PM, Andrej notifications@github.com wrote:

RE: what's inside matlab directory you have to be careful! Caffe has changed many proto fields, it might give errors if you try to use this. Or it gets automatically converted? I haven't tried any of this yet.

— Reply to this email directly or view it on GitHub https://github.com/karpathy/neuraltalk/pull/7#issuecomment-75885814.

Subhasis

Subhasis

npow commented 9 years ago

Hi, I'm wondering why the features are extracted from net.outputs[0], which is the prob layer? Shouldn't the features be from the fc7 layer?

karpathy commented 9 years ago

@npow the deploy prototxt that's used during feature extraction has the top layer (probability layer) cut off. Maybe in more recent caffe versions there is no need to use separate deploy files and you can specify the layer directly. I haven't been keeping up with this too closely.