some message can't load when in test mode

[tinylife]

mode=test时，detector.match没有返回值，导致matches为空，影响后续的相关变量值 下面是一部分输出信息： eval_calc_errors算出的误差也是空的：

[meiqua]

是master分支最新代码吗？我跑出来是这样：

/home/meiqua/anaconda3/envs/linemod/bin/python /home/meiqua/patch_linemod/patch_linemod_test.py

dep anchors:
 [509, 610, 733, 879, 1055], 
dep range: 200

test img size: (480, 640)
####################

reading detector template & info, obj: 1
load model success    
face(triangles) nums: 88740
       vertices nums: 44546
you may want tools like meshlab to simplify models to speed up rendering
------------------------------------

num templs: 21717
####################
scene: 1, im: 0
/home/meiqua/patch_linemod/pysixd/inout.py:60: FutureWarning: arrays to stack must be passed as a "sequence" type such as list or tuple. Support for non-sequence iterables such as generators is deprecated as of NumPy 1.16 and will raise an error in the future.
  im = np.vstack(map(np.uint16, r.asDirect()[2])).astype(np.float32)
candidates size before refine & nms: 305

candidates size after refine & nms: 42

local refine time: 6.45953369140625s
icp time: 4.050656080245972s

linemod time: 7.801952123641968s
matching time: 14.31969666481018s

[tinylife]

@meiqua 是最新的，我调试时一点点找，发现我上面说的那个变量空的，导致后面步骤没法进行

[meiqua]

对，看输出是match那出了问题。我看匹配时间是2，比较特殊，既不是0，也不是8，有可能是depth类型出问题了？传进去的时候是uint16吗？ 如果类型不对，相当于过了一遍模版但法向量得分都是0，所以只会有第一层金字塔的匹配时间。

[tinylife]

我print了一下depth[0][0]的数据类型是unit16。 还有就是涉及到scipy版本的问题，1.2之后的版本用不了与imread类似的图片相关的函数，我使用了matplotlib里的函数，见./pysixd/inout.py文件40-60行 **def load_im(path): im = mtplb.imread(path)

# Using PyPNG
# r = png.Reader(filename=path)
# im = np.vstack(itertools.imap(np.uint8, r.asDirect()[2]))

return im

def save_im(path, im): mtplb.imsave(path, im)

# Using PyPNG (for RGB)
# w_rgb = png.Writer(im.shape[1], im.shape[0], greyscale=False, bitdepth=8)
# with open(path, 'wb') as f:
#     w_rgb.write(f, np.reshape(im, (-1, 3 * im.shape[1])))

def load_depth(path):

PyPNG library is used since it allows to save 16-bit PNG

r = png.Reader(filename=path)
im = np.vstack(map(np.uint16, r.asDirect()[2])).astype(np.float32)
# itertools.imap is removed in py3
return im

def load_depth2(path): d = mtplb.imread(path) d = d.astype(np.float32) return d**

[meiqua]

按理说一样的代码，如果match输入一致跑出来一样才对。输入之前rgb depth显示出来是正常的吗？

[tinylife]

这是match前的图： 还有，我提到过有可能是..patch_linemod_pybind..包本身出问题了：

import pybind11_builtins as __pybind11_builtins
# no functions
# classes
class Detector(__pybind11_builtins.pybind11_object):
    # no doc
    def addTemplate(self, sources, numpy_ndarray=None, *args, **kwargs): # real signature unknown; NOTE: unreliably restored from __doc__ 
        """ addTemplate(self: patch_linemod_pybind.Detector, sources: List[numpy.ndarray], class_id: str, object_mask: numpy.ndarray=None, dep_anchors: List[int]=[]) -> List[int] """
        pass

    def clear_classes(self): # real signature unknown; restored from __doc__
        """ clear_classes(self: patch_linemod_pybind.Detector) -> None """
        pass

    def getTemplates(self, arg0, arg1): # real signature unknown; restored from __doc__
        """ getTemplates(self: patch_linemod_pybind.Detector, arg0: str, arg1: int) -> List[patch_linemod_pybind.Template] """
        return []

    def match(self, sources, numpy_ndarray=None, *args, **kwargs): # real signature unknown; NOTE: unreliably restored from __doc__ 
        """ match(self: patch_linemod_pybind.Detector, sources: List[numpy.ndarray], threshold: float, active_ratio: float, class_ids: List[str], dep_anchors: List[int], dep_range: int, masks: List[numpy.ndarray]=None) -> List[patch_linemod_pybind.Match] """
        pass

    def numTemplates(self): # real signature unknown; restored from __doc__
        """ numTemplates(self: patch_linemod_pybind.Detector) -> int """
        return 0

    def readClasses(self, arg0, p_str=None, *args, **kwargs): # real signature unknown; NOTE: unreliably restored from __doc__ 
        """ readClasses(self: patch_linemod_pybind.Detector, arg0: List[str], arg1: str) -> None """
        pass

    def read_matches(self, arg0): # real signature unknown; restored from __doc__
        """ read_matches(self: patch_linemod_pybind.Detector, arg0: str) -> List[patch_linemod_pybind.Match] """
        return []

    def writeClasses(self, arg0): # real signature unknown; restored from __doc__
        """ writeClasses(self: patch_linemod_pybind.Detector, arg0: str) -> None """
        pass

    def write_matches(self, arg0, patch_linemod_pybind_Match=None, *args, **kwargs): # real signature unknown; NOTE: unreliably restored from __doc__ 
        """ write_matches(self: patch_linemod_pybind.Detector, arg0: List[patch_linemod_pybind.Match], arg1: str) -> None """
        pass

    def __init__(self, *args, **kwargs): # real signature unknown; restored from __doc__
        """
        __init__(*args, **kwargs)
        Overloaded function.

        1. __init__(self: patch_linemod_pybind.Detector) -> None

        2. __init__(self: patch_linemod_pybind.Detector, arg0: List[int], arg1: int) -> None

        3. __init__(self: patch_linemod_pybind.Detector, arg0: int, arg1: List[int], arg2: int) -> None
        """
        pass

class Match(__pybind11_builtins.pybind11_object):
    # no doc
    def __init__(self): # real signature unknown; restored from __doc__
        """ __init__(self: patch_linemod_pybind.Match) -> None """
        pass

    class_id = property(lambda self: object(), lambda self, v: None, lambda self: None)  # default

    similarity = property(lambda self: object(), lambda self, v: None, lambda self: None)  # default

    template_id = property(lambda self: object(), lambda self, v: None, lambda self: None)  # default

    x = property(lambda self: object(), lambda self, v: None, lambda self: None)  # default

    y = property(lambda self: object(), lambda self, v: None, lambda self: None)  # default

class poseRefine(__pybind11_builtins.pybind11_object):
    # no doc
    def get_depth_edge(self, arg0): # real signature unknown; restored from __doc__
        """ get_depth_edge(self: patch_linemod_pybind.poseRefine, arg0: int) -> numpy.ndarray """
        pass

    def process(self, arg0, arg1, arg2, arg3, arg4, arg5, arg6, arg7): # real signature unknown; restored from __doc__
        """ process(self: patch_linemod_pybind.poseRefine, arg0: numpy.ndarray, arg1: numpy.ndarray, arg2: numpy.ndarray, arg3: numpy.ndarray, arg4: numpy.ndarray, arg5: int, arg6: int, arg7: float) -> None """
        pass

    def set_depth(self, depth, K=None): # real signature unknown; restored from __doc__
        """ set_depth(self: patch_linemod_pybind.poseRefine, depth: numpy.ndarray, K: numpy.ndarray=None) -> None """
        pass

    def __init__(self): # real signature unknown; restored from __doc__
        """ __init__(self: patch_linemod_pybind.poseRefine) -> None """
        pass

    fitness = property(lambda self: object(), lambda self, v: None, lambda self: None)  # default

    inlier_rmse = property(lambda self: object(), lambda self, v: None, lambda self: None)  # default

    result_refined = property(lambda self: object(), lambda self, v: None, lambda self: None)  # default

[meiqua]

不太可能吧，那在c++那边把rgb depth跟type显示出来看看？

[meiqua]

包本身不会有问题，要不然参数都对不上。倒是有一点可能，传到c++的时候类型变了。可以在c++ match函数那检查下。

[meiqua]

刚才我试了下，强行把错的类型传进去，输出的情况跟你很相似：

                matches = detector.match([rgb, depth.astype(np.float32)], 70, active_ratio,
                                         match_ids, dep_anchors, dep_range, masks=[])

/home/meiqua/anaconda3/envs/linemod/bin/python /home/meiqua/patch_linemod/patch_linemod_test.py

dep anchors:
 [509, 610, 733, 879, 1055], 
dep range: 200

test img size: (480, 640)
####################

reading detector template & info, obj: 1
load model success    
face(triangles) nums: 88740
       vertices nums: 44546
you may want tools like meshlab to simplify models to speed up rendering
------------------------------------

num templs: 21717
####################
scene: 1, im: 0
/home/meiqua/patch_linemod/pysixd/inout.py:60: FutureWarning: arrays to stack must be passed as a "sequence" type such as list or tuple. Support for non-sequence iterables such as generators is deprecated as of NumPy 1.16 and will raise an error in the future.
  im = np.vstack(map(np.uint16, r.asDirect()[2])).astype(np.float32)
candidates size before refine & nms: 0

candidates size after refine & nms: 0

local refine time: 0.00020694732666015625s
icp time: 0s

linemod time: 1.472144603729248s
matching time: 1.5175652503967285s

[tinylife]

函数调用时传了7个参数，逐一和linemod的c++文件对比了一下：

 # srcs, score for one part, active ratio, may be too low for simple objects so too many candidates?
                matches = detector.match([rgb, depth], 70, active_ratio,
                                         match_ids, dep_anchors, dep_range, masks=[])

linemod的c++里是这样的：

 std::vector<Match> Detector::match(const std::vector<Mat> &sources__, float threshold, float active_ratio,
                                   const std::vector<std::string> &class_ids,
                                   const std::vector<int>& dep_anchors, const int dep_range,
                                   const std::vector<Mat> &masks_ori)

发现只有70是int而定义是float不一样，但是改成70.0还是没有效果。

[meiqua]

不是比对这个，是看sources的类型是不是CV_16U。

[tinylife]

我输出查看了depth的类型：

 cout<<"depth's type:"<<typeid(sources[1]).name()<<endl;

输出结果：

depth's type:N2cv3MatE

应该有问题

[meiqua]

cout<<"CV_16U: "<< CV_16U <<endl;
cout<<"depth's type: "<< sources[1].type() <<endl;

[tinylife]

上面的语句输出：

CV_16U: 2
depth's type: 2
`

[meiqua]

这得详细测测看了。 把dump的两个文件放到/tmp下面； 然后在match开头结尾加这两段代码：

    const bool dump = true;
    const bool record = false;
    if(dump){  // for debug
        if(record){
            FileStorage fs("/tmp/dump_in.yml", FileStorage::WRITE);
            fs << "rgb" << sources__[0];
            fs << "dep" << sources__[1];
        }else{
            FileStorage fs("/tmp/dump_in.yml", FileStorage::READ);

            std::vector<Mat> source_test(2);
            fs["rgb"] >> source_test[0];
            fs["dep"] >> source_test[1];
            assert(cv::countNonZero(source_test[0]!=sources__[0]) == 0);
            assert(cv::countNonZero(source_test[1]!=sources__[1]) == 0);

            cout << "check in OK" << endl;
        }
    }

    if(dump){  // for debug
        if(record){
            write_matches(matches_final, "/tmp/dump_out.yml");
            assert(1 == 0 && "record end");
        }else{
            auto matches_test = read_matches("/tmp/dump_out.yml");
            cout << "match test size: " << matches_final.size() << endl;
            assert(matches_test.size() == matches_final.size());

            cout << "check out OK" << endl;
            assert(1 == 0 && "check end");
        }
    }

[tinylife]

加上后，出现错误：

 im = np.vstack(map(np.uint16, r.asDirect()[2])).astype(np.float32)
Traceback (most recent call last):
  File "/home/tiny/patch_linemod/patch_linemod_test.py", line 323, in <module>
    match_ids, dep_anchors, dep_range, masks=[])
RuntimeError: OpenCV(3.4.6) /home/tiny/opencv/modules/core/src/arithm.cpp:1265: error: (-209:Sizes of input arguments do not match) The operation is neither 'array op array' (where arrays have the same size and the same type), nor 'array op scalar', nor 'scalar op array' in function 'compare'

[meiqua]

这个错误说明还是输入的问题，不能比较，是第一个还是第二个assertion? 在中间插个cout看看

[tinylife]

第一个assertion就有问题，中间插cout跑不到

[meiqua]

rgb居然出问题了。。显示出来看看？

            cout << "CV_8UC3" << CV_8UC3 << endl;
            cout << "rgb type: " << sources__[0].type() << endl;
            cout << "rgb test type: " << source_test[0].type() << endl;
            imshow("rgb", sources__[0]);
            imshow("rgb test", source_test[0]);
            waitKey(0);

[tinylife]

果然是有问题的：

CV_8UC316
rgb type: 21
rgb test type: 16
libpng warning: Application built with libpng-1.2.54 but running with 1.6.18
libpng warning: Application built with libpng-1.2.54 but running with 1.6.18
libpng warning: Application built with libpng-1.2.54 but running with 1.6.18
libpng warning: Application built with libpng-1.2.54 but running with 1.6.18
libpng warning: Application built with libpng-1.2.54 but running with 1.6.18
libpng warning: Application built with libpng-1.2.54 but running with 1.6.18
libpng warning: Application built with libpng-1.2.54 but running with 1.6.18
libpng warning: Application built with libpng-1.2.54 but running with 1.6.18
libpng warning: Application built with libpng-1.2.54 but running with 1.6.18
libpng warning: Application built with libpng-1.2.54 but running with 1.6.18
libpng warning: Application built with libpng-1.2.54 but running with 1.6.18
libpng warning: Application built with libpng-1.2.54 but running with 1.6.18

[meiqua]

估计是python哪个读图片的库出问题了 把debug关掉，dump = false;，或者直接删 然后把python读rgb depth的换掉：

                rgb = cv2.imread(dp['test_rgb_mpath'].format(scene_id, im_id), cv2.IMREAD_COLOR)
                depth = cv2.imread(dp['test_depth_mpath'].format(scene_id, im_id), cv2.IMREAD_ANYDEPTH)

[meiqua]

我才注意到之前你说读图的换成matplotlib了，没想到这个读rgb进来竟然不是int8，是float类型

[tinylife]

改掉之后，运行正常了，不过比较卡：