Can't find NOCS visualizations and path to the runnning_outputs

[kausiksivakumar]

In the instructions mentioned for running evaluations on NOCS, what is the directory of [PATH TO THE RUNNING OUTPUTS]? Also while running predictions on NOCS, I followed #15 's instructions and updated Bundler.cpp accordingly. Even after doing that, I could not find the the visualizations of NOCS stored in the results directory. I could only find a directory called poses that contains all the homogeneous transforms in a txt file. The directory color_viz does not contain any file. Could you help me with it. Thanks

I did three things, 1) set log = 3 (in config_nocs.yml) 2) Add _fm->vizKeyPoints(frame); in Bundler.cpp 3) I did not quite understand what you were going for here https://github.com/wenbowen123/BundleTrack/issues/15#issuecomment-1004555934 . Could you clarify how to save the image? This is my output that I received at terminal while running predictions on NOCS

New frame 0280
zmq start waiting for reply
zmq got reply
finding corres between 0280(id=1) and 0000(id=0)
frame 0280 and 0000 findNN too few match, 0280 status marked as FAILcolor file: ./NOCS/real_test/scene_1//../../real_test/scene_1/0281_color.png

The above message is for all the frames Could you help me with this. Thanks

[kausiksivakumar]

Does it have something to do with the commented part of the code BundleTrack/src/FeatureManager.cpp line 98

[kausiksivakumar]

Also me changing debug_dir on config_nocs.yml doesn't seem to change the directory in which the results are stored

[kausiksivakumar]

Also I tried changing log = 3 in run_nocs.py manually Now, when I run again, I get this as feedback

[kausiksivakumar]

But me changing log = 3 in config_nocs.yml is not changing anything in the feedback. Could you please help me with this @wenbowen123 . I sincerely appreciate your time and effort. Thank you

[wenbowen123]

Hi, sorry for the late reply.

PATH TO THE RUNNING OUTPUTS: this refers to the debug_dir in config_nocs.yml

But me changing log = 3 in config_nocs.yml is not changing anything in the feedback. Could you please help me with this @wenbowen123 . I sincerely appreciate your time and effort. Thank you

What command did you run after it? Can you elaborate your complete steps. In particular, I

Also I tried changing log = 3 in run_nocs.py manually Now, when I run again, I get this as feedback

This should be the right way to do, but I'm not sure why you run into that error. Can you use some printing function to get some clues where it happens?

[wenbowen123]

Can you show me the file structures under debug_dir: /tmp/BundleTrack/ if you didn't change the debug_dir.

[kausiksivakumar]

This is my file structure in the debug dir. I changed it in run_nocs.py instead of the config_nocs.yaml file

[kausiksivakumar]

config_nocs.py i am able to get keypoints visualized but it is unable to find matches between images.

I'm basically getting this error, When I change log to 3 in run_nocs.py I am getting keypoints visualized but it is unable to match them

Debug_dir: https://drive.google.com/drive/folders/1qz2XRhPVkEL3-zns36scEWF3-RE3mBZN?usp=sharing

[wenbowen123]

Yes, the keypoints detection look good. It should be the matching problem.

Can you change this line to =1 and re-build and run? https://github.com/wenbowen123/BundleTrack/blob/476d56f38527de1029306df6d9699151907df568/CMakeLists.txt#L19

[kausiksivakumar]

Oh, it is already = 1. The outputs that are linked in this thread is after changing this line to 1

[kausiksivakumar]

I get this when I turn it to 0

[wenbowen123]

It is strange that you got frame 389 right after 0. Are the RGB files in your image folder complete and ordered?

[kausiksivakumar]

Yes they are. This matching error occurs at all the frames from 1 to 389.

This continues till frame 389

[wenbowen123]

At https://github.com/wenbowen123/BundleTrack/blob/476d56f38527de1029306df6d9699151907df568/src/FeatureManager.cpp#L247

can you replace with this code then re-build and run and copy the output here?

void SiftManager::findCorresbyNN(std::shared_ptr<Frame> frameA, std::shared_ptr<Frame> frameB)
{
  assert(frameA->_id > frameB->_id);
  if (frameA->_keypts.size()==0 || frameB->_keypts.size()==0) return;

  const float max_dist_no_neighbor = (*yml)["feature_corres"]["max_dist_no_neighbor"].as<float>();
  const float cos_max_normal_no_neighbor = std::cos((*yml)["feature_corres"]["max_normal_no_neighbor"].as<float>()/180.0*M_PI);
  const float max_dist_neighbor = (*yml)["feature_corres"]["max_dist_neighbor"].as<float>();
  const float cos_max_normal_neighbor = std::cos((*yml)["feature_corres"]["max_normal_neighbor"].as<float>()/180.0*M_PI);

  const int H = frameA->_H;
  const int W = frameA->_W;

  bool is_neighbor = std::abs(frameA->_id-frameB->_id)==1;

  std::vector<cv::DMatch> matches_AB, matches_BA;
  std::vector< std::vector<cv::DMatch> > knn_matchesAB, knn_matchesBA;
  const int k_near = 5;

#if NO_OPENCV_CUDA
  cv::Ptr<cv::DescriptorMatcher> matcher = cv::DescriptorMatcher::create(cv::DescriptorMatcher::BRUTEFORCE);
  matcher->knnMatch( frameA->_feat_des, frameB->_feat_des, knn_matchesAB, k_near);
  matcher->knnMatch( frameB->_feat_des, frameA->_feat_des, knn_matchesBA, k_near);
#else
  cv::Ptr<cv::cuda::DescriptorMatcher> matcher = cv::cuda::DescriptorMatcher::createBFMatcher(cv::NORM_L2);
  matcher->knnMatch( frameA->_feat_des_gpu, frameB->_feat_des_gpu, knn_matchesAB, k_near);
  matcher->knnMatch( frameB->_feat_des_gpu, frameA->_feat_des_gpu, knn_matchesBA, k_near);
#endif

  printf("knn_matchesAB: %d, knn_matchesBA: %d\n",knn_matchesAB.size(),knn_matchesBA.size());
  pruneMatches(frameA,frameB,knn_matchesAB,matches_AB);

  pruneMatches(frameB,frameA,knn_matchesBA,matches_BA);
  printf("111 knn_matchesAB: %d, knn_matchesBA: %d\n",knn_matchesAB.size(),knn_matchesBA.size());

  collectMutualMatches(frameA,frameB,matches_AB,matches_BA);

  if (_matches[{frameA,frameB}].size()<5 && is_neighbor)
  {
    frameA->_status = Frame::FAIL;
    printf("frame %s and %s findNN too few match, %s status marked as FAIL", frameA->_id_str.c_str(), frameB->_id_str.c_str(), frameA->_id_str.c_str());
  }

}

[kausiksivakumar]

Yes, this is what I received for all the frames

[kausiksivakumar]

Is there something going wrong with the way I have installed packages?

[wenbowen123]

This is strange. I tested a while ago and didn't have this error. Can you update with this and repeat the steps? Also make sure you didn't change the code beyond the configuration files.

void SiftManager::findCorresbyNN(std::shared_ptr<Frame> frameA, std::shared_ptr<Frame> frameB)
{
  assert(frameA->_id > frameB->_id);
  if (frameA->_keypts.size()==0 || frameB->_keypts.size()==0) return;

  const float max_dist_no_neighbor = (*yml)["feature_corres"]["max_dist_no_neighbor"].as<float>();
  const float cos_max_normal_no_neighbor = std::cos((*yml)["feature_corres"]["max_normal_no_neighbor"].as<float>()/180.0*M_PI);
  const float max_dist_neighbor = (*yml)["feature_corres"]["max_dist_neighbor"].as<float>();
  const float cos_max_normal_neighbor = std::cos((*yml)["feature_corres"]["max_normal_neighbor"].as<float>()/180.0*M_PI);

  const int H = frameA->_H;
  const int W = frameA->_W;

  bool is_neighbor = std::abs(frameA->_id-frameB->_id)==1;

  std::vector<cv::DMatch> matches_AB, matches_BA;
  std::vector< std::vector<cv::DMatch> > knn_matchesAB, knn_matchesBA;
  const int k_near = 5;

#if NO_OPENCV_CUDA
  cv::Ptr<cv::DescriptorMatcher> matcher = cv::DescriptorMatcher::create(cv::DescriptorMatcher::BRUTEFORCE);
  matcher->knnMatch( frameA->_feat_des, frameB->_feat_des, knn_matchesAB, k_near);
  matcher->knnMatch( frameB->_feat_des, frameA->_feat_des, knn_matchesBA, k_near);
#else
  cv::Ptr<cv::cuda::DescriptorMatcher> matcher = cv::cuda::DescriptorMatcher::createBFMatcher(cv::NORM_L2);
  matcher->knnMatch( frameA->_feat_des_gpu, frameB->_feat_des_gpu, knn_matchesAB, k_near);
  matcher->knnMatch( frameB->_feat_des_gpu, frameA->_feat_des_gpu, knn_matchesBA, k_near);
#endif

  printf("knn_matchesAB: %d, knn_matchesBA: %d\n",knn_matchesAB.size(),knn_matchesBA.size());
  pruneMatches(frameA,frameB,knn_matchesAB,matches_AB);

  pruneMatches(frameB,frameA,knn_matchesBA,matches_BA);
  printf("111 knn_matchesAB: %d, knn_matchesBA: %d\n",knn_matchesAB.size(),knn_matchesBA.size());

  collectMutualMatches(frameA,frameB,matches_AB,matches_BA);

  if (_matches[{frameA,frameB}].size()<5 && is_neighbor)
  {
    frameA->_status = Frame::FAIL;
    printf("frame %s and %s findNN too few match: %d, %s status marked as FAIL\n", frameA->_id_str.c_str(), frameB->_id_str.c_str(), _matches[{frameA,frameB}].size(), frameA->_id_str.c_str());
  }

}

[kausiksivakumar]

Hmm, I still get the same error But that said, I did change a few things. For instance, I changed data, mask and debug directory in the yml file I found that the logs and the debug directory were getting overwritten so I had to change it in run_nocs.py file And I added in Bundler.cpp

[kausiksivakumar]

Should I try reinstalling the process from scratch again?

[kausiksivakumar]

Hey @wenbowen123 I tried reinstalling the code from scratch. But I still received the same error as stated

[wenbowen123]

I found that the logs and the debug directory were getting overwritten so I had to change it in run_nocs.py file

Hi, can you clarify how are these over-written (or over-written by what)? It should change the output folder of BundleTrack so the results will dumped there.

Can you replace collectMutualMatches with below and repeat the previous steps? We might be close to find the problem.

void SiftManager::collectMutualMatches(std::shared_ptr<Frame> frameA, std::shared_ptr<Frame> frameB, const std::vector<cv::DMatch> &matches_AB, const std::vector<cv::DMatch> &matches_BA)
{
  auto &matches = _matches[{frameA, frameB}];

  for (int i=0;i<matches_AB.size();i++)
  {
    int Aid = matches_AB[i].queryIdx;
    int Bid = matches_AB[i].trainIdx;
    float uA = frameA->_keypts[Aid].pt.x;
    float vA = frameA->_keypts[Aid].pt.y;
    float uB = frameB->_keypts[Bid].pt.x;
    float vB = frameB->_keypts[Bid].pt.y;
    const auto &ptA = frameA->_cloud->at(std::round(uA), std::round(vA));
    const auto &ptB = frameB->_cloud->at(std::round(uB), std::round(vB));
    Correspondence corr(uA,vA,uB,vB, ptA, ptB, true);
    matches.push_back(corr);
  }
  printf("#matches=%d\n",matches.size());
  for (int i=0;i<matches_BA.size();i++)
  {
    int Aid = matches_BA[i].trainIdx;
    int Bid = matches_BA[i].queryIdx;
    float uA = frameA->_keypts[Aid].pt.x;
    float vA = frameA->_keypts[Aid].pt.y;
    float uB = frameB->_keypts[Bid].pt.x;
    float vB = frameB->_keypts[Bid].pt.y;
    const auto &ptA = frameA->_cloud->at(std::round(uA), std::round(vA));
    const auto &ptB = frameB->_cloud->at(std::round(uB), std::round(vB));
    Correspondence corr(uA,vA,uB,vB, ptA, ptB, true);
    matches.push_back(corr);
  }
  printf("111 #matches=%d\n",matches.size());
}

[kausiksivakumar]

I thinkhe run_nocs.py (in run_one_scene_one_model function) overwrites the value passed in yaml. If you notice the logs are reset to 0 and cur_out_dir is also updated Due to this whatever changes I make to config_nocs.yaml gets overwritten and I the default results directory remains in /tmp/BundleTrack and the log is always 0

[kausiksivakumar]

I changed CollectMutualMatches with the script given I got this for all frame combination

[wenbowen123]

https://github.com/wenbowen123/BundleTrack/issues/9#issuecomment-995455933 this might be the problem. Are you running with a GPU and does the cuda program compile successfully?

[kausiksivakumar]

How do I check that. I see that when building it using the script given, I get 100% successfully. Could you let me know what you meant by the cuda program? Where do I see where the depth map is set and how do I change it?

[kausiksivakumar]

`python run_server.py 
/opt/conda/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
  return f(*args, **kwds)
Loading from /home/kausik/DAIR/BundleTrack/lf-net-release/release/models/indoor/config.pkl
---------------------- OPTIONS ----------------------

                 activ_fn  leaky_relu
           aug_max_degree  180
            aug_max_scale  1.4142135623730951
               batch_size  6
               clear_logs  False
             com_strength  3.0
               conv_ksize  5
              crop_radius  16
            data_raw_size  362
                data_size  256
                  dataset  scannet
             depth_thresh  1.0
            desc_activ_fn  relu
          desc_conv_ksize  3
                 desc_dim  256
              desc_inputs  photos
         desc_leaky_alpha  0.2
                desc_loss  triplet
              desc_margin  1.0
         desc_net_channel  64
           desc_net_depth  3
                desc_norm  l2norm
          desc_perform_bn  True
         desc_train_delay  0
               descriptor  simple_desc
                 det_loss  l2loss
                 detector  mso_resnet_detector
     do_softmax_kp_refine  True
         hard_geom_thresh  False
                 hm_ksize  15
                 hm_sigma  0.5
             hpatches_dir  /scratch/trulls/yuki.ono/datasets/hpatches
            init_num_mine  64
          input_inst_norm  True
          kp_com_strength  1.0
              kp_loc_size  9
              leaky_alpha  0.2
                  log_dir  /scratch/trulls/yuki.ono/results/deep_det/desc/180705-scannet-ori-sv-3d/adam-lr-1e-3-False/mso_resnet_detector/scannet-15/aug-rTrue-180-sTrue/ori-True-5-scl-5-0.7-1.4/desc-photos-D256-topk-512/mine-rand_hard_sch-64-5-0.9/innorm-True/wdet-0.01/ori-l2loss-w-0.1/scl-0.1/nms3d-True/try-1
                       lr  0.001
                 lr_decay  False
      match_reproj_thresh  5
                  max_itr  50000
           max_seq_length  2000
           min_num_pickup  5
              mining_type  rand_hard_sch
                net_block  3
              net_channel  16
            net_max_scale  1.4142135623730951
            net_min_scale  0.7071067811865475
           net_num_scales  5
                nms_ksize  5
               nms_thresh  0.0
              num_threads  16
             optim_method  adam
                ori_ksize  5
                 ori_loss  l2loss
               ori_weight  0.1
               patch_size  32
               perform_bn  True
             pickup_delay  0.9
             pretrain_dir  
            random_offset  False
                  rot_aug  True
                scale_aug  True
       scale_com_strength  100.0
             scale_weight  0.1
              scannet_dir  /scratch/trulls/yuki.ono/datasets/scannet/dataset
             scenenet_dir  /scratch/trulls/yuki.ono/datasets/scenenet
       score_com_strength  100.0
              sfm_dpt_dir  /scratch/trulls/yuki.ono/datasets/colmap/dataset2/vis-0.4
              sfm_img_dir  /scratch/trulls/yuki.ono/datasets/colmap/colmap
                 sfm_mode  nips
                  sfm_seq  sacre_coeur
            sfm_train_seq  train.txt
            sfm_valid_seq  valid.txt
           show_histogram  False
                 sm_ksize  15
                soft_kpts  True
               soft_scale  True
                    top_k  512
           train_main_seq  0
           train_num_traj  100
                train_ori  True
        train_pair_offset  15
          train_same_time  True
                use_nms3d  True
           valid_num_traj  10
        valid_pair_offset  15
               webcam_dir  /scratch/trulls/yuki.ono/datasets/WebcamRelease
          weight_det_loss  0.01

----------------------------------------------------
Act-Fn:  <function get_activation_fn.<locals>.<lambda> at 0x7efe1b9a47b8>
Apply instance norm on input photos
Scales (0.707107~1.41 #5): [1.41421356 1.18920712 1.         0.84089642 0.70710678]
PAD=16, #conv=8, ksize=5 ori-ksize=5
Act-Fn:  <function relu at 0x7efe30855400>
===== SimpleDesc (reuse=False) =====
#1 conv-bn-act (?, 16, 16, 64)
#2 conv-bn-act (?, 8, 8, 128)
#3 conv-bn-act (?, 4, 4, 256)
FLAT (?, 4096)
Feat-Norm: L2-NORM
FEAT (?, 256)
2022-02-15 15:03:44.081732: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2022-02-15 15:03:45.259521: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-15 15:03:45.259807: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties: 
name: NVIDIA GeForce RTX 3050 Ti Laptop GPU major: 8 minor: 6 memoryClockRate(GHz): 1.485
pciBusID: 0000:01:00.0
totalMemory: 3.82GiB freeMemory: 3.74GiB
2022-02-15 15:03:45.259836: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: NVIDIA GeForce RTX 3050 Ti Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6)
Load trained models...
Checkpoint models-latest-42000
port tcp://*:5555
lfnet listending to tcp://*:5555`

This is where I get something about the gpu printed. I'm presuming that since I get some information about my gpu pritned, I believe that it is using my GPU. But I don't know where to start checking about my depthmap being set to zero. Could you help with this. Thanks, I sincerely appreciate the help

[kausiksivakumar]

https://github.com/wenbowen123/BundleTrack/issues/22#issuecomment-1028387412 This only works when I set this to 1

[wenbowen123]

Your laptop has NVIDIA GeForce RTX 3050 Ti which is relatively new. I remember some users have met similar issues using the newer GPUs. This repo has been tested on 2080 Ti. Other older GPUs may also work. I don't have newer GPUs to develop and test on my side, so there is not plan of supporting the new GPUs in near future. Another workaround will be rewriting all the cuda code by C++.

[kausiksivakumar]

Okay, we do have a computer set up with an older version of the GPU. Is there any specific graphic driver version that you want me to install. Just making sure that I cover all the bases

[wenbowen123]

We used driver 460 and CUDA 11.2

[kausiksivakumar]

Yes, it works. I believe as you said, the problem is with the newer GPUs. Also, the path to download the weights returns a 404 now. Thought I should bring it to your notice