Use of texture (including color) to improve the detection of some objects

[albertodesouza]

The texture of the coke.obj model used in the Tutorials does not appear during training when I turn on LINEMOD_VIZ_IMG... Relevant code for visualization (linemod_train.cpp):

#if LINEMOD_VIZ_IMG
// Display the rendered image
if (*visualize_)
{
  cv::namedWindow("Rendering");
  if (!image.empty()) {
  cv::imshow("Rendering", image);
  cv::waitKey(1);
  }
}
#endif

I would like to use texture (including color) to improve detection of some objects...

Alberto.

[albertodesouza]

In line 212 of linemod_train.cpp, addTemplate() might return -1. This turns the indexes of the poses of the templates saved in the database out of sync with the id of the detected objects. This typically shows up in rviz as not properly oriented object detections. To correct the problem I suggest changing this peace of code:

detector_->addTemplate(sources, "object1", mask);

// Also store the pose of each template
Rs_->push_back(cv::Mat(R));
Ts_->push_back(cv::Mat(T));
distances_->push_back(distance);
Ks_->push_back(cv::Mat(K));

to that one:

int template_in = detector_->addTemplate(sources, "object1", mask);
if (template_in == -1)
{
  // Delete the status
  for (size_t j = 0; j < status.str().size(); ++j)
    std::cout << '\b';
  continue;
}

// Also store the pose of each template
Rs_->push_back(cv::Mat(R));
Ts_->push_back(cv::Mat(T));
distances_->push_back(distance);
Ks_->push_back(cv::Mat(K));

Alberto.

[bmagyar]

Hi Alberto,

regarding the title of this issue, I'd direct you to the paper of LINEMOD. http://campar.cs.tum.edu/pub/hinterstoisser2011linemod/hinterstoisser2011linemod.pdf

I would rather add a post-filtering step after the linemod module in a pipeline fashion (in the end that is what ORK is for) that gets rid of wrong detections. If you prepare such a system, I'm sure many would apprieciate using it/ reading about it on the mailing list.

Cheers,

[albertodesouza]

Hi Bence,

On Wed, Jan 28, 2015 at 7:22 AM, Bence Magyar notifications@github.com wrote:

Hi Alberto,

regarding the title of this issue, I'd direct you to the paper of LINEMOD.

http://campar.cs.tum.edu/pub/hinterstoisser2011linemod/hinterstoisser2011linemod.pdf

I have read the paper (quite a good one). If one examines it, one can see that the authors used color images (see first paragraph of page 4). Linemod is good for textureless objects, but it uses color texture (in OpenGL sense) to improve performance on color textured objects.

The current implementation of OpenCV linemod (used by ORK linemod) uses color gradients, as described in the paper (see http://docs.opencv.org/ref/master/d7/d7a/linemod_8hpp.html, class cv::linemod::ColorGradient http://docs.opencv.org/ref/master/dc/d64/classcv_1_1linemod_1_1ColorGradient.html). Actualy, ORK linemod calls OpenCV linemod passing color images as parameter (see https://github.com/wg-perception/linemod/blob/master/src/linemod_detect.cpp). They just do not have color due to some bug in the OpenGL redering code.

I would rather add a post-filtering step after the linemod module in a pipeline fashion (in the end that is what ORK is for) that gets rid of wrong detections.

The use of colors in the center of the object in a post-processing step might increase precision, but note that the color gradients in the borders (which are used by linemod) are very important for color objects in any case.

Even in the case of a post-linemod color processing pipeline step, a color (OpenGL texture) model is required and, in the current version of the code, this is not working. The code for texture processing is there, it is just not working.

If you prepare such a system, I'm sure many would apprieciate using it/ reading about it on the mailing list.

I would like to see the benefits of using color texture, as proposed in the original paper, before going for a post-processing pipeline step. The performance of the code as it is now is significantly inferior than the results obtained by the authors and, I believe, one of the reasons is the lack of color texture processing. So, any help on making it work will be greatly appreciated.

I am in for an extra pipeline step for reducing false positives as well, but we will need OpenGL texture processing for that...

Cheers,

Alberto.

Cheers,

— Reply to this email directly or view it on GitHub https://github.com/wg-perception/linemod/issues/10#issuecomment-71826819 .

[bmagyar]

Hi, Now I understand you. For the first read it wasn't clear to me in which context you used 'texture'. Now I see you meant texture in an OpeGL model sense, and color in the context of linemod.

[albertodesouza]

Hi,

On Fri, Jan 30, 2015 at 3:41 AM, Bence Magyar notifications@github.com wrote:

Hi, Now I understand you. For the first read it wasn't clear to me in which context you used 'texture'. Now I see you meant texture in an OpeGL model sense, and color in the context of linemod.

Great! Did you ever see the object model in color during linemod training? The code for texturing is there, it is just not working... The coke.obj file loads OK if you start the training command in the same directory of coke.obj.

Alberto.

— Reply to this email directly or view it on GitHub https://github.com/wg-perception/linemod/issues/10#issuecomment-72170208 .

[gypsytearss]

The solution we found involved solving two problems: (1) enabling texture in OpenGL and (2) fixing the recursive call to Model::recursive_render. Here's the quick solution we used:

Model::recursive_render(const struct aiScene *sc, const aiNode* nd, const int j) const
{
  int i;
  unsigned int n = 0, t;
  aiMatrix4x4 m = nd->mTransformation;

  // update transform
  aiTransposeMatrix4(&m);
  glPushMatrix();
  glMultMatrixf((float*) &m);

  // draw all meshes assigned to this node
  for (; n < nd->mNumMeshes; ++n)
  {
    glEnable(GL_TEXTURE_2D);
    const struct aiMesh* mesh = sc->mMeshes[nd->mMeshes[n]];

    if (n < texturesAndPaths.size())
      glBindTexture(GL_TEXTURE_2D, texturesAndPaths[j].hTexture);

    apply_material(sc->mMaterials[mesh->mMaterialIndex]);

    if (mesh->mNormals == NULL)
      glDisable (GL_LIGHTING);
    else
      glEnable(GL_LIGHTING);

    if (mesh->mColors[0] != NULL)
      glEnable (GL_COLOR_MATERIAL);
    else
      glDisable(GL_COLOR_MATERIAL);

    for (t = 0; t < mesh->mNumFaces; ++t)
    {
      const struct aiFace* face = &mesh->mFaces[t];
      GLenum face_mode;

      switch (face->mNumIndices)
      {
        case 1:
          face_mode = GL_POINTS;
          break;
        case 2:
          face_mode = GL_LINES;
          break;
        case 3:
          face_mode = GL_TRIANGLES;
          break;
        default:
          face_mode = GL_POLYGON;
          break;
      }

      glBegin(face_mode);

      for (i = 0; i < face->mNumIndices; i++)
      {
        int index = face->mIndices[i];
        if (mesh->mColors[0] != NULL)
          Color4f(&mesh->mColors[0][index]);
        if (mesh->mNormals != NULL)
          glNormal3fv(&mesh->mNormals[index].x);
        if (mesh->HasTextureCoords(0))
          glTexCoord2f(mesh->mTextureCoords[0][index].x, mesh->mTextureCoords[0][index].y);
        glVertex3fv(&mesh->mVertices[index].x);
      }

      glEnd();
    }
  }

  // draw all children
  for (n = 0; n < nd->mNumChildren; ++n)
    recursive_render(sc, nd->mChildren[n], n);

  glPopMatrix();
}

But note that with any outside calls to the recursive_render function (e.g. in Model::Draw), you'll have to specify a third parameter (typically 0).

Colin

[vrabaud]

That looks good. So that we check better the differences, can you please make a proper pull request ? Thx

[bhack]

See also @mtamburrano linergb branches at https://github.com/mtamburrano/lineRGB