About picture information acquisition and resizing

[aaaa47080]

Exceuse me, @saierd saierd, The first point is that I want to start from ITM RENDER POINT Depth information was extracted from map, but ITM was found RENDER POINT There are three channels in the map. I have looked at each channel separately and can probably confirm whether the second channel (respectively) is our real depth information.Sorry i can't find reopen choose so i recreate a issue.

Supplement: my original idea was that depth information is a single channel, but I can't seem to find depth information with only a single channel. So we can only grab the depth information in this way. If I want you to tell me how to get such information，and below is my code

from ensenso_nxlib import NxLibCommand, NxLibItem
from ensenso_nxlib.constants import *
import ensenso_nxlib
import cv2
import pandas as pd
import numpy as np

stereo_serial = "193244"
mono_serial = "4103989237"

ensenso_nxlib.api.initialize()

cmd = NxLibCommand(CMD_OPEN)
cmd.parameters()[ITM_CAMERAS][0] = stereo_serial
cmd.parameters()[ITM_CAMERAS][1] = mono_serial
cmd.execute()

stero_cam = NxLibItem()[ITM_CAMERAS][stereo_serial]
param = json.dumps(json.load(open('parameter5.json', "r")))
stero_cam << param
print('Load the stereo Camera parameter file....')

mono_cam = NxLibItem()[ITM_CAMERAS][mono_serial]
param = json.dumps(json.load(open('parameter5.json', "r")))
mono_cam << param
print('Load the  mono  Camera parameter file....')

NxLibCommand(CMD_CAPTURE).execute()
NxLibCommand(CMD_COMPUTE_DISPARITY_MAP).execute()
NxLibCommand(CMD_COMPUTE_POINT_MAP).execute()

#parameters_tree_json =  json.dumps(json.load(open('23.json', "r")))
cmd = NxLibCommand(CMD_RENDER_POINT_MAP)
cmd.parameters()[ITM_SIZE] = [256, 256]
# cmd.parameters()[ITM_SIZE] << parameters_tree_json
par = cmd.parameters().as_json()
# print(par)
cmd.execute()

depth = NxLibItem()[ITM_IMAGES][ITM_RENDER_POINT_MAP].get_binary_data()

depth_1 = depth[:,:,0]
depth_2 = depth[:,:,1]
depth_3 = depth[:,:,2]

And another question is to continue the question just now I want to get the depth channel from the code above. Its size is 768 1024, but I want to modify it to 256 256 I use cmd.parameters() [itm_size] = [256, 256] and other mentioned by Mr. before cmd.parameters()[ITM_SIZE] << parameters_numpy_Array method, but the final images are 768 * 1024, otherwise there will be errors, so I want to ask you how to correctly adjust the image size. Sorry to bother you, sir

[saierd]

RenderPointMap as you execute it does a telecentric projection of the point cloud as described in the documentation of the command. All 3 channels of the resulting image is your depth data. They contain the x, y and z coordinates of the points. There is the ZBufferOnly parameter to output only the z coordinates, if your workspace is oriented in a way that you are only interested in the height of the points.

The Size parameter for choosing the size of the rendered image works as you thought, but you did not specify the parameter correctly. The Python API does not support assignment of Python arrays to an NxLibItem at the moment (although this might be a good idea).

What you can do instead is assign the nodes separately like this:

cmd.parameters()[ITM_SIZE][0] = 256
cmd.parameters()[ITM_SIZE][1] = 256

or assign a JSON string to it like this:

cmd.parameters()[ITM_SIZE] << "[256, 256]"

[aaaa47080]

Thank you sir, but what I am wondering is that the color image result (RGB & alpha) in RENDER_POINT_MAP_TEXTURE is corresponding to our depth information, so will there be any difference from the color image of the original monocular camera? Can I use the color of the original monocular camera? Will this cause errors in depth image and color coordinate comparison?

Second, the image is combined with the depth obtained in RENDER_POINT_MAP to form one (RGBD data is fine), and if the data obtained by RENDER_POINT_MAP_TEXTURE must be used, is it necessary to keep the alpha channel, because it seems to only distinguish whether the triangle mesh can be stitched ( Is there a broken image), if so, can I use the alpha channel and mark (0 or 255) the part of the array that may appear nan due to broken image in the subsequent array?

I am sorry to you, sir I have a lot of questions...

[saierd]

• The pixel positions in the RenderPointMapTexture image always correspond to the pixels in the RenderPointMap binary (regardless of the modes I will explain below), you can simply use the same index in both images.
• The default mode of cmdRenderPointMap is to perform a telecentric projection with the given ViewPose. This is a third perspective, different from mono and stereo camera and therefore it can contain pixels for which there is neither 3D nor texture data.
• RenderPointMap contains a NaN point when there is no 3D data; RenderPointMapTexture contains alpha = 0 for points for which there is no texture data as documented here.
• If you want to see the point cloud from the perspective of the color camera, you can use the Camera parameter. In this case, the RenderPointMapTexture will be the same as the camera image and there are no missing pixels in the texture image.

[aaaa47080]

I have a little question about the first point raised by Mr.

So although the (RGB, not talking about alpha channel) in the RenderPointMap Texture is the color of the projected object, not the color of the real object, we can still use it as an independent feature (like the real RGB of the object), right?

As I mentioned above, I can take the depth single channel of RenderPointMap and combine the first three channels of RGB in RenderPointMapTexture as my physical features (in the sense, it means that the same object will have similar values), forming four RGBD meaningful feature data.

[aaaa47080]

Then I tried to make the fourth point of the single-lens camera mentioned by Mr. The code is as follows, but the image I presented seems to be a bit broken. At first, I considered whether the environment was too complicated and caused the possibility of laser propagation. The sending and receiving is not complete, so it should be changed to Dunchun’s wall and a book for testing, but it still seems to break the picture. I hope that Mr. can trouble you to point out the problem I did.

Code and Image is below:

from ensenso_nxlib import NxLibCommand, NxLibItem
from ensenso_nxlib.constants import *
import ensenso_nxlib
import cv2
import pandas as pd 
import numpy as np

stereo_serial = "193244"
mono_serial = "4103989237"

ensenso_nxlib.api.initialize()

cmd = NxLibCommand(CMD_OPEN)
cmd.parameters()[ITM_CAMERAS][0] = stereo_serial
cmd.parameters()[ITM_CAMERAS][1] = mono_serial
cmd.execute()

stero_cam = NxLibItem()[ITM_CAMERAS][stereo_serial]
param = json.dumps(json.load(open('parameter5.json', "r")))
stero_cam << param
print('Load the stereo Camera parameter file....')

mono_cam = NxLibItem()[ITM_CAMERAS][mono_serial]
param = json.dumps(json.load(open('parameter5.json', "r")))
mono_cam << param
print('Load the  mono  Camera parameter file....')

NxLibCommand(CMD_CAPTURE).execute()
NxLibCommand(CMD_COMPUTE_DISPARITY_MAP).execute()
NxLibCommand(CMD_COMPUTE_POINT_MAP).execute()

# parameters_tree_json =  json.dumps(json.load(open('23.json', "r")))
cmd = NxLibCommand(CMD_RENDER_POINT_MAP)
cmd.parameters()[ITM_CAMERA] = mono_serial
cmd.parameters()[ITM_NEAR] = 1
cmd.parameters()[ITM_FAR] = 10000
cmd.parameters()[ITM_SIZE][0] = 1280
cmd.parameters()[ITM_SIZE][1] = 1280
# cmd.parameters()[ITM_FILL_XY_COORDINATES] = True
# cmd.parameters()[ITM_Z_BUFFER_ONLY] = True
cmd.execute()

depth = NxLibItem()[ITM_IMAGES][ITM_RENDER_POINT_MAP_TEXTURE].get_binary_data()
depth_1 = depth[0:1280,0:1280]
cv2.imshow('ee',depth_1)
cv2.waitKey(0)
cv2.destroyAllWindows()

cv2.imshow('ee',depth)
cv2.waitKey(0)
cv2.destroyAllWindows()
cv2.imwrite('image_1.png', depth)

[saierd]

I think I don't really understand your first question. RenderPointMap creates an image according to the view pose you specified and fills

• the pixels of the point map with the nearest 3D point that the stereo camera observed in this direction and
• the pixels of the texture with the color that the RGB camera observed at this position.

Regarding your screenshots:

• With the Camera parameter you don't have to specify the size, it will automatically use the resolution and view of the mono camera.
• Your screenshots look like there is an offset between the mono camera and the 3D data. Did you calibrate your camera and is the Link node correct?

[aaaa47080]

• I may have thought wrong yesterday. I thought you said RenderPointMapTexture is a transformation in RenderPointMap . Very sorry.

Because I want to use the acquired features for deep training, I want to make sure that the features between them are independent, just like the RGB three channels are independent features. Just need more in-depth information now…

• I will continue to make adjustments in the follow-up. I use Nxview to perform the libration of the two cameras. The situation has improved, but there will still be some broken images. I will continue to adjust.

Below is the new image after the Clibration:

[aaaa47080]

Excuse me sir, is there a command to turn off the LED light of the Ensenso N35? Because he seems to have a different image every time he takes a photo.

This would prevent me from using the same standard when working with images.

Below is the half-processed picture, you can find that when taking pictures, one picture has LED lights, but the other one does not.

[saierd]

The projector can be disabled with the Projector node.

The pattern is required for good stereo matching, though. If you want to have both depth and color images without texture, you cannot trigger the cameras at the same time.

[aaaa47080]

Thanks for your answer, sir

If the LED is turned on all the time and not turned off, I don't know if there will be the above problems?

[saierd]

Yes, if the texture in the color image is not a problem for you, you can also leave the projector enabled.

In that case, the timing of the exposure of the mono camera is important. If you use software trigger as above, it is a bit random and that is why you sometimes see the texture and sometimes not.

To prevent this, you can use a hardware trigger for the color camera. You need a trigger cable between the devices and then set up the hardware trigger in the NxLib as described here.

[aaaa47080]

Thank you sir, I wonder if you can provide a python writing example that can keep the projector enabled, because I actually want to try it, but I can't find how to write it. It is different from the way Command has parameters directly.

[saierd]

The example from the manual shows you the nodes in the JSON tree and the commands you have to use to set up hardware trigger. For Python, you just have to translate the syntax a bit. E.g.

camera[itmParameters][itmCapture][itmTriggerMode] = valFallingEdge;

becomes

camera[ITM_PARAMETERS][ITM_CAPTURE][ITM_TRIGGER_MODE] = VAL_FALLING_EDGE

See the readme for some general information on the differences between the C++ and the Python API.

[aaaa47080]

Thanks, i will try later.

[aaaa47080]

Sir, I tried it but got an error message

    "Result": {
            "ErrorSymbol": "StereoCameraModel",
            "ErrorText": "Stereo camera model not initialized.",

I do not know why

from ensenso_nxlib import NxLibCommand, NxLibItem
from ensenso_nxlib.constants import *
import ensenso_nxlib
import cv2
import numpy as np
import time 
import warnings
warnings.filterwarnings('ignore')

path_num = 0
while path_num != 3:
    stereo_serial = "193244"
    mono_serial = "4103989237"

    ensenso_nxlib.api.initialize()

    cmd = NxLibCommand(CMD_OPEN)
    cmd.parameters()[ITM_CAMERAS][0] = stereo_serial
    cmd.parameters()[ITM_CAMERAS][1] = mono_serial
    cmd.execute()

    stero_cam = NxLibItem()[ITM_CAMERAS][stereo_serial]
    param = json.dumps(json.load(open('parameter12.json', "r")))
    stero_cam << param
    print('Load the stereo Camera parameter file....')

    mono_cam = NxLibItem()[ITM_CAMERAS][mono_serial]

    stero_cam[ITM_PARAMETERS][ITM_CAPTURE][ITM_TRIGGER_MODE] = VAL_FALLING_EDGE
    param = json.dumps(json.load(open('parameter12.json', "r")))
    mono_cam << param
    print('Load the  mono  Camera parameter file....')

    NxLibCommand(CMD_CAPTURE).execute()
    #NxLibCommand(CMD_COMPUTE_DISPARITY_MAP).execute()
    NxLibCommand(CMD_COMPUTE_POINT_MAP).execute()
    cmd = NxLibCommand(CMD_RENDER_POINT_MAP)
    cmd.parameters()[ITM_CAMERA] = mono_serial
    cmd.parameters()[ITM_NEAR] = 10
    cmd.parameters()[ITM_FAR] = 500
    cmd.execute()

[saierd]

Oh, I actually linked the wrong guide. Here ist the correct one.

You have to enable the output of one camera and set the trigger mode of the other camera, depending on how you connected them. Usually the stereo cameras gets software triggered and the RGB cameras needs a trigger mode.

You can also open both cameras in NxView and play with these parameters to make sure that the cable is connected correctly and hardware trigger works.

[aaaa47080]

Thanks, sir . If still have any other question, i will open again. Thank you so much.

[aaaa47080]

from turtle import color
from ensenso_nxlib import NxLibCommand, NxLibItem
from ensenso_nxlib.constants import *
import ensenso_nxlib
import cv2
import numpy as np
import time 
import matplotlib.pyplot as plt
import warnings
warnings.filterwarnings('ignore')

P_FAR = 900
P_NEAR = 300

path_num = 0
while path_num != 1:
    print(path_num)
    stereo_serial = "193244"
    mono_serial = "4103989237"

    ensenso_nxlib.api.initialize()

    cmd = NxLibCommand(CMD_OPEN)
    cmd.parameters()[ITM_CAMERAS][0] = stereo_serial
    cmd.parameters()[ITM_CAMERAS][1] = mono_serial
    cmd.execute()

    stero_cam = NxLibItem()[ITM_CAMERAS][stereo_serial]
    mono_cam = NxLibItem()[ITM_CAMERAS][mono_serial]

    caputer = NxLibCommand(CMD_CAPTURE)
    caputer.parameters()[ITM_WAIT_FOR_PROJECTOR] = True
    caputer.execute()

    NxLibCommand(CMD_COMPUTE_DISPARITY_MAP).execute()
    NxLibCommand(CMD_COMPUTE_POINT_MAP).execute()

    cmd = NxLibCommand(CMD_RENDER_POINT_MAP)
    cmd.parameters()[ITM_Z_BUFFER_ONLY] = True
    cmd.parameters()[ITM_FILL_XY_COORDINATES] = True
    cmd.parameters()[ITM_NEAR] = P_NEAR
    cmd.parameters()[ITM_FAR] = P_FAR
    cmd.parameters()[ITM_SIZE][0] = 512
    cmd.parameters()[ITM_SIZE][1] = 512
    cmd.parameters()[ITM_VIEW_POSE] << json.dumps(json.load(open('22.json', "r")))
    cmd.execute()
    depth = NxLibItem()[ITM_IMAGES][ITM_RENDER_POINT_MAP].get_binary_data()

I'm sorry to bother you, sir, I want to ask two questions

1. What is the unit measured by depth?

2. Why do I get a larger depth value when I swing something close, but a smaller depth when I swing it away? I've tried tilting the object back, but I've found that tilting it back reduces the depth.

[saierd]

• All distances in the NxLib are in mm.
• By default (without a workspace calibration, i.e. an identity Link node) a camera is placed in the origin and looks in positive z direction, so objects further away have bigger depth values as you expected.
• With RenderPointMap, the z values depend on the direction of the ViewPose and the near/far values. There were some bugs in older versions of the NxLib where near and far values were swapped, so you should make sure that you use a recent 3.2 version.

It is difficult to say what might be inverted in your case. You can try to verify coordinates by opening your cameras in NxView, switching to "Point Cloud (Z channel)" and "Rendered Projection (Z channel)" (this uses the RenderPointMap command) mode in the bottom right corner and hovering over the images to see the z values. In the 3D view you can also see the direction of coordinate axes from the drawn axes in the center.

[aaaa47080]

Thanks sir, i will try again.

[aaaa47080]

Sir, i have tried but i confirmed my ensenso sdk version is 3.2.489 (latest)

The python ensenso_nxlib version is 0.4.0 (latest)

But the problem of its depth and size exchange still exists. I don't know if there are other possible reasons.

[saierd]

Did you try to verify your setup in NxView?

[aaaa47080]

Right! The depth on NxView is normal and it gets deeper with the slope, but the depth captured on Python is reversed.

But I just experimented and found that the problem of depth inversion is only when I want to get a single channel Z.

cmd.parameters()[ITM_Z_BUFFER_ONLY] = True
cmd.parameters()[ITM_FILL_XY_COORDINATES] = True

But direct viewing on Nxview will not have this problem, the depth data captured with python SDK will be reversed

I don't know why there is this problem

[saierd]

Closing this ticket, since I am archiving this repository. Please contact support if you have further questions.