cdcseacave
commented
1 year ago @adricostas the code is fine, all you need to do is to match the resolution of the depth-maps when you call DenseReconstruction by using the param: --max-resolution 256
Open adricostas opened 1 year ago
cdcseacave
commented
1 year ago @adricostas the code is fine, all you need to do is to match the resolution of the depth-maps when you call DenseReconstruction by using the param: --max-resolution 256
adricostas
commented
12 months ago Thanks for the reply. Unfortunately, even changing the max_resolution parameter the densification doesn't work. Here are the values that I'm using:
MVS::OPTDENSE::init();
MVS::OPTDENSE::update();
MVS::OPTDENSE::nResolutionLevel = 0; // how many times to scale down the images before point cloud computation
MVS::OPTDENSE::nMaxResolution = 256; // do not scale images higher than this resolution
MVS::OPTDENSE::nMinResolution = 192; // do not scale images lower than this resolution
MVS::OPTDENSE::nSubResolutionLevels = 2; // number of patch-match sub-resolution iterations (0 - disabled)
MVS::OPTDENSE::nNumViews = 8; // number of views used for depth-map estimation (0 - all neighbor views available)
MVS::OPTDENSE::nMinViewsFuse = 3; // minimum number of images that agrees with an estimate during fusion in order to consider it inlier (<2 - only merge depth-maps)
MVS::OPTDENSE::nEstimationIters = 4; // number of patch-match iterations
MVS::OPTDENSE::nEstimationGeometricIters = 2; // number of geometric consistent patch-match iterations (0 - disabled)
MVS::OPTDENSE::nEstimateColors = 2; // estimate the colors for the dense point-cloud (0 - disabled, 1 - final, 2 - estimate)
MVS::OPTDENSE::nEstimateNormals = 2; // estimate the normals for the dense point-cloud (0 - disabled, 1 - final, 2 - estimate)
MVS::OPTDENSE::nOptimize = 7; // flags used to filter the depth-maps after estimation (0 - disabled, 1 - remove-speckles, 2 - fill-gaps, 4 - adjust-filter)
MVS::OPTDENSE::nIgnoreMaskLabel = -1; // label value to ignore in the image mask, stored in the MVS scene or next to each image with '.mask.png' extension (<0 - disabled)
MVS::OPTDENSE::bRemoveDmaps = false; // remove depth-maps after fusion
Util::Init();
bool densificationSuccess = mvsScene->DenseReconstruction();Any idea?
cdcseacave
commented
12 months ago I could have ideas if I know what is not working, an error message, assert, line, etc
adricostas
commented
12 months ago The same as at the initial stage:
received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff60ffd000 (LWP 80603)]
MVS::PatchMatchCUDA::EstimateDepthMap (this=<optimized out>, depthData=...)
at /data/libraries/openMVS-2.1.0/libs/MVS/PatchMatchCUDA.cpp:341
341 depthNormal.topLeftCorner<3, 1>() = Eigen::Map<const Normal::EVec>(n.ptr());
(gdb) bt
#0 MVS::PatchMatchCUDA::EstimateDepthMap (this=<optimized out>, depthData=...)
at /data/libraries/openMVS-2.1.0/libs/MVS/PatchMatchCUDA.cpp:341
#1 0x0000555555bc9fac in MVS::DepthMapsData::EstimateDepthMap (this=0x7fffffffc578,
idxImage=<optimized out>, nGeometricIter=0)
at /data/libraries/openMVS-2.1.0/libs/Common/List.h:370
#2 0x0000555555bce2e0 in MVS::Scene::DenseReconstructionEstimate (this=<optimized out>,
pData=0x7fffffffc550) at /data/libraries/openMVS-2.1.0/libs/Common/List.h:370
#3 0x0000555555bcf8df in DenseReconstructionEstimateTmp (arg=<optimized out>)
at /data/libraries/openMVS-2.1.0/libs/MVS/SceneDensify.cpp:1932
#4 0x00007ffff7eb4ea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fffd4346a2f in clone () from /lib/x86_64-linux-gnu/libc.so.6And this is because depthData.normalMap is empty when this code is executed:
// load depth-map and normal-map into CUDA memory
for (int r = 0; r < depthData.depthMap.rows; ++r) {
const int baseIndex = r * depthData.depthMap.cols;
for (int c = 0; c < depthData.depthMap.cols; ++c) {
const Normal& n = depthData.normalMap(r, c);
const int index = baseIndex + c;
Point4& depthNormal = depthNormalEstimates[index];
depthNormal.topLeftCorner<3, 1>() = Eigen::Map<const Normal::EVec>(n.ptr());
depthNormal.w() = depthData.depthMap(r, c);
}
}first of all why do you even run EstimateDepthMap if you have the depth-maps? do you want to refine them?
cdcseacave
commented
12 months ago to fuse them as they are add these params --geometric-iters 0 --postprocess-dmaps 0
adricostas
commented
12 months ago first of all why do you even run EstimateDepthMap if you have the depth-maps? do you want to refine them?
No, I don't want to refine them, I only want to take advantage of the depthmaps provided by a LIDAR to carry out the reconstruction. In other issues I read that I should convert the depthmaps that I have to .dmap files and that's all what I have done so far. I don't know if I should modify any of the parameters that I use to call the DenseReconstruction funtion regarding the ones that I used to perform the DenseReconstruction when I don't have those depthmaps. (listed in one of my previous comments). So far, I have only changed the value of the max_resolution and min_resolution based on your previous comment. I don't know if I must change more parameters in this case.
to fuse them as they are add these params
--geometric-iters 0 --postprocess-dmaps 0
Ok, I will try this!
Thanks
adricostas
commented
12 months ago Now it is working! Thank you very much for your support! Just out of curiosity, since the reconstruction that I'm getting using the LIDAR depthmaps is a little bit worse than the one returned when the depthmaps are estimated, is there a way of refining the LIDAR depthmaps as you mentioned in your previous comment?
cdcseacave
commented
12 months ago Yes, there is, run it as you tried at first, but your need to debug why the normals are not estimated when the depth map is loaded, EstimateNormals should be called
On Mon, 18 Sep 2023 at 17:54 Adriana Costas @.***> wrote:
Now it is working! Thank you very much for your support! Just out of curiosity, since the reconstruction that I'm getting using the LIDAR depthmaps is a little bit worse than the one returned when the depthmaps are estimated, is there a way of refining the LIDAR depthmaps as you mentioned in your previous comment?
— Reply to this email directly, view it on GitHub https://github.com/cdcseacave/openMVS/issues/1055#issuecomment-1723612129, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVMH3WCTJ6ZKQLWRO4JML3X3BOCHANCNFSM6AAAAAA4P6TBSI . You are receiving this because you commented.Message ID: @.***>
adricostas
commented
12 months ago Ok, thanks, your help is really appreciated. And what about using confidence maps together with the depthmaps provided by ARKit? I can get a confidence map with values [0,1,2] that indicate low, medium and high confidence respectively. Is this range compatible with the range of the confidence maps generated by OpenMVS? In case of including the confidence maps into the .dmap files, will they be used for the reconstruction if I only fuse the depthmaps as they are (without refinement)?
cdcseacave
commented
12 months ago yes, can be used, but you need to convert them in a [0,1] range, ex: 0.5, 0.7, 0.9
bkhanal-11
commented
11 months ago @adricostas Hi, you mentioned that you are using custom triangulation and bundle adjustment code to extract the sparse pointcloud from the data obtained through ARKit. I have been trying to do the same. Could you help me with this? Currently I am using poses from camera.transform, but when I try to visualize them, they are inaccurate. I would appretiate if you could share how you computed/optimized the poses obtained from ARKit.
adricostas
commented
11 months ago Hi @bkhanal-11 ,
Yes, I'm using the poses provided by camera.transform and they are not very accurate. In order to optimize them I'm applying a bundle adjustment after the triangulation, allowing the optimization of the 3D points, poses and focal length.
bkhanal-11
commented
11 months ago Hi @bkhanal-11 ,
Yes, I'm using the poses provided by camera.transform and they are not very accurate. In order to optimize them I'm applying a bundle adjustment after the triangulation, allowing the optimization of the 3D points, poses and focal length.
Could you share the resources? I am trying to do the same thing for few weeks for my academic project and not getting any result. I would appreciate if you can do that.
cdcseacave
commented
11 months ago You can also use Polycam app which records the ARKit data (RGBD images) and there is an importer already in OpenMVS for Polycam projects
adricostas
commented
9 months ago Hello,
Now I'm able to use my own depthmaps and confidence maps. The pointcloud seems to be good but the mesh is not, the edges of the screens are projected onto the wall. At first, I thought that it could be related to the fact that the depth values are not good for black objects, so I decided to set to 0 the confidence value for those black pixels, but the issue persists.
Is there any parameter of the ReconstructMesh/RefineMesh/TextureMesh functions that can be related with this issue?
cdcseacave
commented
9 months ago there is nothing wrong with meshing, the problem you see is that the input point cloud does not represent accurately the scene, the monitors are larger in reality then the point cloud represents them, so the side of them project on the existing 3D data
adricostas
commented
9 months ago Hello again,
In a previous comment you said that the confidence values must be in a [0,1] range. However, if I load the confidence maps saved into the .dmap files after executing the densification with depthmap estimation I can see that the values are not in that range. For example:
1, 5.0483069, 5.0695906, 2.0225251, 3.4381022, 3.4898958, 3.5330601, 3.4796228, 5.2911892, 5.4303646, 5.5793405, 5.5250182, 5.5812407, 6.0368762, 5.9885068, 6.1395311, 6.0218611, 7.9737659, 7.0617104, 6.3745222, 6.0646019, 6.9580593, 7.3866162, 7.9089084, 7.2736816, 7.8875203, 7.7201228, 7.8146148, 7.994205, 8.2426958, 8.1242065, 7.8560362, 7.6062765, 7.7271338, 7.6856451, 7.5575194, 7.7892895, 7.8579407, 7.6881709, 7.989953, 8.1339159, 8.224411, 8.2972584, 8.2843409, 8.257967, 8.1948452, 8.0655117, 8.1105509, 8.1168203, 7.987555, 7.4744143, 7.3470759, 7.4429359, 7.3608084, 7.524375, 7.4632635, 7.3884854, 7.255477, 7.5569077, 7.2498341, 7.6209731, 7.71593, 7.8580632, 7.9617243, 7.8899817, 8.2001476, 8.2355967, 8.3117199, 8.2758617, 8.2630291, 8.3665562, 8.2412949, 8.3400803, 8.1813316, 8.2487545, 8.2409811, 8.2706308, 8.244483, 8.1666231, 8.1991138, 8.0261993, 7.3620954, 7.1547971, 7.4995127, 7.5973654, 7.3446236, 7.4550943, 7.1218815, 7.4026203, 7.1257133, 7.2600708, 6.9128304, 6.6415625, 6.7303729, 7.2444487, 7.3918858, 7.558774, 7.2504511, 4.3551507, 0, 4.8897295, 6.5880327, 0, 3.3730402, 0, 0, 0, 0, 0, 0, 4.0555401, 4.1620922, 4.2206154, 1.6849673, 2.3533087, 0.78184462, 3.8277426, 5.5829439, 7.4815526, 7.5697012, 7.30651, 6.6364918, 6.8388777, 6.4330626, 1.8075125, 0, 2.1535058, 4.5627813, 5.3484144, 0.59444714, 5.9300184, 6.3187647, 3.9350061, 3.2421331, 5.7487864, 1.1701021, 6.9918909, 4.3732824, 5.2069545, 0, 0.38182735, 0, 0, 0, 0.87245917, 0, 2.379704, 4.2571478, 7.6528358, 6.2679367, 6.3264341, 7.4724069, 7.4385476, 7.375258, 7.5066323, 7.2340479, 7.2679281, 5.8839655, 5.7965603, 3.8879151, 3.1355267, 3.2605906, 5.933054, 6.1761336, 6.0362225, 6.0593672, 6.365809, 3.3957384, 4.1885295, 0.68125665, 5.500412, 5.6755672, 5.2084894, 4.2836995, 4.5210762, 5.9079604, 6.2796335, 6.8923545, 7.0521898, 6.7968831, 7.0922742, 6.7861562, 6.5700784, 6.0940976, 6.4174914, 6.5802846, 6.0844502, 6.2528706, 6.2330751, 5.9067507, 4.849577, 6.2422915, 5.6693168, 5.8975906, 6.0085521, 6.3508468, 5.5860386, 5.9122629, 6.1860838, 6.1318274, 6.150589, 6.239943, 5.9974422, 5.9547424, 6.0962205, 6.1653175, 6.5562329, 6.6156454, 5.1655059, 5.0708065, 6.5970111, 6.8149738, 3.6255834, 5.7750845, 5.7844472, 5.917984, 5.5443387, 3.9540286, 2.0247207, 4.19065, 0, 4.3368883, 3.7535872, 3.035743, 1.2702851, 3.3791163, 1.3348475, 2.4232755, 0.38400102, 0.07514596, 1.025833, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1.7016785, 1.7002449, 0.054858685, 0, 0, 0, 3.1563268, 3.3699477, 1.8394439, 0.42570138, 0, 0;
I think that these non-normalized values come from the function FilterDepthMaps in which confidence values are added but they are not normalized again. Are these values ok?
for (int i=0; i<sizeRef.height; ++i) {
for (int j=0; j<sizeRef.width; ++j) {
const ImageRef xRef(j,i);
const Depth depth(depthDataRef.depthMap(xRef));
if (depth == 0) {
newDepthMap(xRef) = 0;
newConfMap(xRef) = 0;
continue;
}
ASSERT(depth > 0);
#if TD_VERBOSE != TD_VERBOSE_OFF
++nProcessed;
#endif
// update best depth and confidence estimate with all estimates
float posConf(depthDataRef.confMap(xRef)), negConf(0);
Depth avgDepth(depth*posConf);
unsigned nPosViews(0), nNegViews(0);
unsigned n(N);
do {
const Depth d(depthMaps[--n](xRef));
if (d == 0) {
if (nPosViews + nNegViews + n < nMinViews)
goto DiscardDepth;
continue;
}
ASSERT(d > 0);
if (IsDepthSimilar(depth, d, thDepthDiff)) {
// average similar depths
const float c(confMaps[n](xRef));
avgDepth += d*c;
posConf += c;
++nPosViews;
} else {
// penalize confidence
if (depth > d) {
// occlusion
negConf += confMaps[n](xRef);
} else {
// free-space violation
const DepthData& depthData = arrDepthData[depthDataRef.neighbors[idxNeighbors[n]].idx.ID];
const Camera& camera = depthData.images.First().camera;
const Point3 X(cameraRef.TransformPointI2W(Point3(xRef.x,xRef.y,depth)));
const ImageRef x(ROUND2INT(camera.TransformPointW2I(X)));
if (depthData.confMap.isInside(x)) {
const float c(depthData.confMap(x));
negConf += (c > 0 ? c : confMaps[n](xRef));
} else
negConf += confMaps[n](xRef);
}
++nNegViews;
}
} while (n);
ASSERT(nPosViews+nNegViews >= nMinViews);
// if enough good views and positive confidence...
if (nPosViews >= nMinViewsAdjust && posConf > negConf && ISINSIDE(avgDepth/=posConf, depthDataRef.dMin, depthDataRef.dMax)) {
// consider this pixel an inlier
newDepthMap(xRef) = avgDepth;
newConfMap(xRef) = (posConf - negConf);
} else {
// consider this pixel an outlier
DiscardDepth:
newDepthMap(xRef) = 0;
newConfMap(xRef) = 0;
#if TD_VERBOSE != TD_VERBOSE_OFF
++nDiscarded;
#endif
}
}
}This value should be newConfMap(xRef) = (posConf - negConf)/((nPosViews + nNegViews + 1) instead of newConfMap(xRef) = (posConf - negConf)?
Thanks in advance!
Hello,
I have a software that is able to create a 3D reconstruction using images captured with a phone and their poses provided by ArKit . I'm using my own triangulation and bundle adjustment code to extract the sparse pointcloud and then OpenMVS to densify it and creating the textured mesh. The reconstruction is working perfectly, but now I'm trying to use directly the depthmaps provided by the LIDAR in order to avoid the estimation step in the OpenMVS pipeline and I'm not able to do it.
As far as I read in other reported issues, I have to create the .dmap files from the LIDAR's depthmaps. To do that, I can use
MVS::HeaderDepthDataRawand take the methodMVS::ExportDepthDataRawas an example. So, after creating myMVS::Sceneand before callingDenseReconstructionI'm doing the following:In this case the resolution of the images is 1280x960 and the resolution of the depthmaps is 256x192 but I'm indicating the last one as the resolution for both in the header because of this : "depth-map-resolution, for now only the same resolution as the image is supported". Is that ok? I guess that this shouldn't be important as depthmaps are not estimated from the images in this case. On the other hand
sfmScene->frames[i].depthMapis a cv::Mat(192,256,CV_32F) and therefore, I can assign it directly to a MVS::DepthMap.The execution after including this piece of code breaks returning a segmentation fault. This segmentation fault occurs inside the MVS::PatchMatchCUDA::EstimateDepthMap function, and it seems that is due to the fact that NormalMap is empty.
Is it necessary to add information about the normals to the dmap file? I supossed that were not necessary because I'm no indicating that I have this information with
HeaderDepthDataRaw::HAS_NORMAL. Another thing that should be pointed out is that, at the time that I'm creating the dmap files the images have not got any neighbors defined. Could it be a problem? These neighbors are only used to estimate the depthmaps, right?I would really appreciate your help on this. I don't know what I'm doing wrong.
Thank you in advance!