0x00b1 commented 7 years ago

Implementation of the Region Proposal Network (RPN) classification and regression losses.

Noteworthy items:

We’ve refactored anchor generation as Keras backend functions so they’ll run from the Keras backend libraries like TensorFlow and Theano.
The loss is computed on a Lambda layer. This is a workaround for Keras’ assumption that the y_true and y_pred values have identical shapes. We should probably refactor this into a custom layer to simplify model construction.
It’s buggy. While there’re plentiful unit tests they likely don’t provide sufficient coverage.
We need to write documentation.

JihongJu commented 7 years ago

@0x00b1

Keras’ assumption that the y_true and y_pred values have identical shapes.

That is interesting. I wasn't aware of it. Do you think we could add pre-processing to targets and generate y_true on the fly? Keras model.compile L833

codecov-io commented 7 years ago

Codecov Report

Merging #41 into master will increase coverage by 12.35%. The diff coverage is 100%.

@@             Coverage Diff             @@
##           master      #41       +/-   ##
===========================================
+ Coverage   53.64%   65.99%   +12.35%     
===========================================
  Files          17       17               
  Lines         563      644       +81     
===========================================
+ Hits          302      425      +123     
+ Misses        261      219       -42

Impacted Files	Coverage Δ
keras_rcnn/losses/rpn.py	`100% <100%> (ø)`	:arrow_up:
keras_rcnn/backend/common.py	`100% <100%> (+27.27%)`	:arrow_up:
...s_rcnn/layers/object_detection/_object_proposal.py	`100% <100%> (ø)`	:arrow_up:
keras_rcnn/backend/tensorflow_backend.py	`100% <100%> (+20.61%)`	:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update e341198...9a7a4c1. Read the comment docs.

jhung0 commented 7 years ago

The code to handle the different shapes was inspired by https://github.com/fchollet/keras/issues/4781 suggests using dummy targets

0x00b1 commented 7 years ago

I just noticed this repository yesterday and, although it is far from finished, it looks really well thought out. I'm hoping I can contribute actively to this project in the near future. I was working on my own port of py-faster-rcnn because I was unhappy with the existing ports for keras / tensorflow, but got stuck on the losses for the RPN network. I mainly believe the issue is with how Keras forces the loss to be computed in a certain way and I made an issue about it on the keras github page (fchollet/keras#7395). If you would like to contribute to that discussion, that'd be helpful. I'm curious how the RPN loss is currently handled in keras-rcnn, that part is a bit unclear to me still. I'm guessing it is some form of workaround, but which kind is it ? :p

Hi, @hgaiser! Let’s collaborate! It’s a tricky problem and we could use the help!

In this pull request, the approach:

For y_pred, the two feature maps produced by the convolutional layers that return the regression (for the reference bounding boxes) and the classification (for the reference bounding box’s objectness) are concatenated into one matrix.
For y_true, we generate the anchors as described by the Faster RCNN paper. It’s somewhat different than the py-faster-rcnn implementation by @rbgirshick as the anchors are generated on the GPU (as part of the loss computation).

I should add that it’s still wonky since we have an intermediate loss (i.e. a loss computed on an intermediate layer). However, in the past week or two I’ve been thinking about a possible RPN implementation that treats the RPN losses as regularization terms rather than traditional losses.

I’m curious, @hgaiser, is this a problem You’d like to help out with? If so, we should schedule some time to talk! This offer also includes anybody else that is lurking and wants to help!

hgaiser commented 7 years ago

Hi,

I would very much like to collaborate! I had been working on a keras port by myself and it was driving me insane ;)

I made a few changes to this PR already, I'd like to share that with you tomorrow, I'll do so in another PR. That is sort of my view on how I imagine it, I would like to hear what you had in mind and what you think of my approach. How shall we talk? I'm always online on Slack (kerasteam), maybe that could be a good channel.

mcquin commented 6 years ago

Remaining work: https://github.com/broadinstitute/keras-rcnn/pull/52#issuecomment-319687963

0x00b1 commented 6 years ago

@hgaiser Good call. Fixed compute_output_shape for the loss layers.

hgaiser commented 6 years ago

Oh that attribute was already there, nice :)

On Aug 7, 2017 10:13 PM, "Allen Goodman" notifications@github.com wrote:

@hgaiser https://github.com/hgaiser Good call. Fixed compute_output_shape for the loss layers.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/broadinstitute/keras-rcnn/pull/41#issuecomment-320767841, or mute the thread https://github.com/notifications/unsubscribe-auth/AArtakwqW5-PwjyWkgLF6q0ku9oL_14wks5sV2_MgaJpZM4OTK5M .

broadinstitute / keras-rcnn

Region Proposal Network (RPN) classification and regression losses (WIP) #41

Codecov Report