Some questions about using the code

[itzhuang]

I am a fresh man about cv and I found your project very interesting. So I follow the steps to install the enviroments, I used pycharm to open the code, but I have a very basic question that how can I run this project, I run the train_net.py directly but there are some errors. I know this are very basicly, but I would be very grateful if you could help me.

[itzhuang]

I run it on colab, but there are some errors. 01.03 17:14:09: Loading checkpoint from /content/drive/MyDrive/panoptic-reconstruction/tools/train_net.py Traceback (most recent call last): File "/content/drive/MyDrive/panoptic-reconstruction/tools/train_net.py", line 59, in main() File "/content/drive/MyDrive/panoptic-reconstruction/tools/train_net.py", line 53, in main trainer = engine.Trainer() File "/content/drive/MyDrive/panoptic-reconstruction/lib/engine/trainer.py", line 22, in init self.setup() File "/content/drive/MyDrive/panoptic-reconstruction/lib/engine/trainer.py", line 47, in setup checkpoint_data = self.checkpointer.load() File "/content/drive/MyDrive/panoptic-reconstruction/lib/utils/checkpoint.py", line 57, in load checkpoint = self._load_file(f) File "/content/drive/MyDrive/panoptic-reconstruction/lib/utils/checkpoint.py", line 137, in _load_file loaded = super(DetectronCheckpointer, self)._load_file(f) File "/content/drive/MyDrive/panoptic-reconstruction/lib/utils/checkpoint.py", line 108, in _load_file return torch.load(f, map_location=torch.device("cpu")) File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 608, in load return _legacy_load(opened_file, map_location, pickle_module, pickle_load_args) File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 777, in _legacy_load magic_number = pickle_module.load(f, pickle_load_args) _pickle.UnpicklingError: could not find MARK

[xheon]

Hi, thanks for you message. I'm not too familiar with Colab, but it looks like the pretrained model is missing. Please download "panoptic_front3d_v2.pth" (had to fix some layer names) from https://hidrive.ionos.com/share/nfgac71bo2#$/ and put it in the "resources" folder.

Let me know, if that was the issue and if you have further issues.

I will update the readme about download the checkpoint.

[itzhuang]

Thanks for replying me! I notice that you update the requirement, so I follow the Installation on https://github.com/xheon/PyMarchingCubes.git to install the PyMarchingCubes in my local enviroment. But there are some errors:

marching_cubes/src/../../eigen/Eigen/src/Core/MatrixBase.h:98:61: error: ‘constexpr int Eigen::internal::max_size_prefer_dynamic(A, B) [with A = Eigen::DenseBase<Eigen::CwiseBinaryOp<Eigen::internal::scalar_product_op<double, double>, const Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, const Eigen::Matrix<double, 6, 1> >, const Eigen::Matrix<double, 6, 1> > >::; B = Eigen::DenseBase<Eigen::CwiseBinaryOp<Eigen::internal::scalar_product_op<double, double>, const Eigen::CwiseNullaryOp<Eigen::internal::scalar_constant_op, const Eigen::Matrix<double, 6, 1> >, const Eigen::Matrix<double, 6, 1> > >::]’ called in a constant expression internal::max_size_prefer_dynamic(RowsAtCompileTime, ColsAtCompileTime)> SquareMatrixType;

marching_cubes/src/../../eigen/Eigen/src/Core/MatrixBase.h:123:65: note: in template argument for type ‘int’ 
                   internal::traits<Derived>::ColsAtCompileTime> BasisReturnType;
                                                                 ^~~~~~~~~~~~~~~
error: command '/usr/bin/gcc' failed with exit code 1

Looking forward to your reply.

[itzhuang]

If I don't install PyMarchingCubes but PyMCubes, there are issues like: UserWarning: This overload of nonzero is deprecated: nonzero() Consider using one of the following signatures instead: nonzero(, bool as_tuple) (Triggered internally at /opt/conda/conda-bld/pytorch_1607369981906/work/torch/csrc/utils/python_arg_parser.cpp:882.) inds = inds_all[:, j].nonzero().squeeze(1) Traceback (most recent call last): File "/home/tz/panoptic-reconstruction-main/tools/test_net_single_image.py", line 143, in main(args) File "/home/tz/panoptic-reconstruction-main/tools/test_net_single_image.py", line 65, in main results = model.inference(input_image, front3d_intrinsic, front3d_frustum_mask) File "/home/tz/panoptic-reconstruction-main/lib/modeling/panoptic_reconstruction.py", line 97, in inference frustum_result = self.frustum3d.inference(projection_result, frustum_mask) File "/home/tz/panoptic-reconstruction-main/lib/modeling/frustum/frustum_completion.py", line 524, in inference unet_output = self.model(frustum_results, 1, frustum_mask_64) File "/home/tz/anaconda3/envs/panoptic2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(input, kwargs) File "/home/tz/panoptic-reconstruction-main/lib/modeling/backbone/unet_sparse.py", line 34, in forward output = self.model(BlockContent(x, frustum_mask), batch_size) File "/home/tz/anaconda3/envs/panoptic2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, *kwargs) File "/home/tz/panoptic-reconstruction-main/lib/modeling/backbone/unet_sparse.py", line 217, in forward processed: BlockContent = self.submodule(BlockContent(encoded, x.encoding), batch_size) File "/home/tz/anaconda3/envs/panoptic2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(input, kwargs) File "/home/tz/panoptic-reconstruction-main/lib/modeling/backbone/unetsparse.py", line 392, in forward dense, , _ = encoded.dense(shape, min_coordinate=min_coordinate) File "/home/tz/anaconda3/envs/panoptic2/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiSparseTensor.py", line 489, in dense assert isinstance(min_coordinate, torch.IntTensor) AssertionError

[xheon]

Hi,

1) regarding PyMarchingCubes, what is your setup when you install it?

• Which compiler are you using?

• Can you try eigen 3.3 instead of the latest one? Another person told me yesterday, that the master branch of eigen breaks it:

  git clone https://gitlab.com/libeigen/eigen.git
  git checkout 3.3

2) Regarding the AssertionError of Minkowski, it's a bit hard to tell without additional context. Do you try to run the provided sample image "rgb_0007.png" or do you try it on another image?

[itzhuang]

Thanks for replying me again! When I switch to eigen 3.3 as you said, I install the PyMarchingCubes sucessfully, that helps me a lot!

About the Minkowski, firstly I set the --input as the followed picture; Then I run the test_net_single_image.py, it tells me AssertionError, the full contexts in the below window are:

/home/tz/anaconda3/envs/panoptic2/bin/python /home/tz/panoptic-reconstruction-main/tools/test_net_single_image.py
Load model...
Load input image...
Perform panoptic 3D scene reconstruction...
/home/tz/panoptic-reconstruction-main/lib/modeling/detector/roi_heads/box_head/inference.py:116: UserWarning: This overload of nonzero is deprecated:
    nonzero()
Consider using one of the following signatures instead:
    nonzero(*, bool as_tuple) (Triggered internally at  /opt/conda/conda-bld/pytorch_1607369981906/work/torch/csrc/utils/python_arg_parser.cpp:882.)
  inds = inds_all[:, j].nonzero().squeeze(1)
Traceback (most recent call last):
  File "/home/tz/panoptic-reconstruction-main/tools/test_net_single_image.py", line 144, in <module>
    main(args)
  File "/home/tz/panoptic-reconstruction-main/tools/test_net_single_image.py", line 66, in main
    results = model.inference(input_image, front3d_intrinsic, front3d_frustum_mask)
  File "/home/tz/panoptic-reconstruction-main/lib/modeling/panoptic_reconstruction.py", line 97, in inference
    frustum_result = self.frustum3d.inference(projection_result, frustum_mask)
  File "/home/tz/panoptic-reconstruction-main/lib/modeling/frustum/frustum_completion.py", line 524, in inference
    unet_output = self.model(frustum_results, 1, frustum_mask_64)
  File "/home/tz/anaconda3/envs/panoptic2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/tz/panoptic-reconstruction-main/lib/modeling/backbone/unet_sparse.py", line 34, in forward
    output = self.model(BlockContent(x, frustum_mask), batch_size)
  File "/home/tz/anaconda3/envs/panoptic2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/tz/panoptic-reconstruction-main/lib/modeling/backbone/unet_sparse.py", line 217, in forward
    processed: BlockContent = self.submodule(BlockContent(encoded, x.encoding), batch_size)
  File "/home/tz/anaconda3/envs/panoptic2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/tz/panoptic-reconstruction-main/lib/modeling/backbone/unet_sparse.py", line 392, in forward
    dense, _, _ = encoded.dense(shape, min_coordinate=min_coordinate)
  File "/home/tz/anaconda3/envs/panoptic2/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiSparseTensor.py", line 489, in dense
    assert isinstance(min_coordinate, torch.IntTensor)
AssertionError

Process finished with exit code 1

If I use my own png pictures, the same error occurs. Looking foward to your reply!

[czh007]

Thanks for replying me again! When I switch to eigen 3.3 as you said, I install the PyMarchingCubes sucessfully, that helps me a lot!

About the Minkowski, firstly I set the --input as the followed picture; Then I run the test_net_single_image.py, it tells me AssertionError, the full contexts in the below window are:

/home/tz/anaconda3/envs/panoptic2/bin/python /home/tz/panoptic-reconstruction-main/tools/test_net_single_image.py
Load model...
Load input image...
Perform panoptic 3D scene reconstruction...
/home/tz/panoptic-reconstruction-main/lib/modeling/detector/roi_heads/box_head/inference.py:116: UserWarning: This overload of nonzero is deprecated:
  nonzero()
Consider using one of the following signatures instead:
  nonzero(*, bool as_tuple) (Triggered internally at  /opt/conda/conda-bld/pytorch_1607369981906/work/torch/csrc/utils/python_arg_parser.cpp:882.)
  inds = inds_all[:, j].nonzero().squeeze(1)
Traceback (most recent call last):
  File "/home/tz/panoptic-reconstruction-main/tools/test_net_single_image.py", line 144, in <module>
    main(args)
  File "/home/tz/panoptic-reconstruction-main/tools/test_net_single_image.py", line 66, in main
    results = model.inference(input_image, front3d_intrinsic, front3d_frustum_mask)
  File "/home/tz/panoptic-reconstruction-main/lib/modeling/panoptic_reconstruction.py", line 97, in inference
    frustum_result = self.frustum3d.inference(projection_result, frustum_mask)
  File "/home/tz/panoptic-reconstruction-main/lib/modeling/frustum/frustum_completion.py", line 524, in inference
    unet_output = self.model(frustum_results, 1, frustum_mask_64)
  File "/home/tz/anaconda3/envs/panoptic2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/tz/panoptic-reconstruction-main/lib/modeling/backbone/unet_sparse.py", line 34, in forward
    output = self.model(BlockContent(x, frustum_mask), batch_size)
  File "/home/tz/anaconda3/envs/panoptic2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/tz/panoptic-reconstruction-main/lib/modeling/backbone/unet_sparse.py", line 217, in forward
    processed: BlockContent = self.submodule(BlockContent(encoded, x.encoding), batch_size)
  File "/home/tz/anaconda3/envs/panoptic2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/tz/panoptic-reconstruction-main/lib/modeling/backbone/unet_sparse.py", line 392, in forward
    dense, _, _ = encoded.dense(shape, min_coordinate=min_coordinate)
  File "/home/tz/anaconda3/envs/panoptic2/lib/python3.8/site-packages/MinkowskiEngine-0.5.4-py3.8-linux-x86_64.egg/MinkowskiEngine/MinkowskiSparseTensor.py", line 489, in dense
    assert isinstance(min_coordinate, torch.IntTensor)
AssertionError

Process finished with exit code 1

If I use my own png pictures, the same error occurs. Looking foward to your reply!

If I use the provided sample image "rgb_0007.png" , the same error occurs. Looking foward to your reply as well !

[czh007]

@itzhuang If use the provided sample image "rgb_0007.png" , the same error occurs？If not, how do you solve it?

[itzhuang]

@itzhuang If use the provided sample image "rgb_0007.png" , the same error occurs？If not, how do you solve it?

Yes, I have the same error.

[xheon]

Sorry for the delay. That issue is really strange and at the moment i'm not able to reproduce it.

Can you insert the following lines before unet_sparse.py:L392 and check that debug_check evaluates to True:

debug_check_int_tensor = isinstance(min_coordinate, torch.IntTensor)
debug_check_int_cuda_tensor = isinstance(min_coordinate, torch.cuda.IntTensor)
debug_check = debug_check_int_tensor or debug_check_int_cuda_tensor

It seems the following assert triggers the exception: https://github.com/xheon/MinkowskiEngine/blob/panoptic/MinkowskiEngine/MinkowskiSparseTensor.py#L477

but unet_sparse.py:L381 explicitly initializes the min_coordinate tensor as IntTensor.

Which MinkowskiEngine repository did you install?

[itzhuang]

Thanks for replying me! You said the MinkowskiEngine repository really reminds me. I found that I installed the version by following the steps in repository is not your folked version. So I replace the MinkowskiSparseTensor.py and run it again. Then I find there are no errors! Thanks! But when I run the test_net_single_image.py, there are some UserWarnings in the console, I wait for 30 minutes and there are no changes in the console and the output folder don't have an output. Should I wait more time or there are still some wrong in it?

The information in the console: /home/tz/anaconda3/envs/panoptic/bin/python /home/tz/panoptic-reconstruction-main/tools/test_net_single_image.py Load model... Load input image... Perform panoptic 3D scene reconstruction... /home/tz/anaconda3/envs/panoptic/lib/python3.9/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1639180549130/work/aten/src/ATen/native/TensorShape.cpp:2157.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

/home/tz/panoptic-reconstruction-main/lib/modeling/projection/sparse_projection.py:159: UserWarning: floordiv is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). padding_offsets = difference // 2

/home/tz/anaconda3/envs/panoptic/lib/python3.9/site-packages/MinkowskiEngine/MinkowskiSparseTensor.py:512: UserWarning: floordiv is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). coords = coords // tensor_stride

/home/tz/panoptic-reconstruction-main/lib/modeling/frustum/frustum_completion.py:531: UserWarning: floordiv is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). predicted_coordinates[:, 1:] = predicted_coordinates[:, 1:] // occupancy_prediction.tensor_stride[0]

[xheon]

The user warnings are fine, some of them origin in the MinkowskiEngine.

For the provided sample, it should not take more than few minutes until it finishes. There are 2 things to check:

• See if the process still utilizes CPU, i.e. in htop and also the GPU, i.e. using nvidia-smi or nvtop
• Debug through the script using break points and see at which point the script takes longer
• Alternatively you can put additional print output in the model to see which parts are done or not

[itzhuang]

Thanks for replying me! I follow your advice and check the process use the gpu. I run fix_checkpoint_names.py again and then I accquire the outputs. Is this the correct output? And I want to ask if I want to use my own picture, how do I need to do with the picture or change some parameters? I run it with my own picture, there are some errors:

Traceback (most recent call last):
  File "/home/tz/panoptic-reconstruction-main/tools/test_net_single_image.py", line 151, in <module>
    main(args)
  File "/home/tz/panoptic-reconstruction-main/tools/test_net_single_image.py", line 65, in main
    results = model.inference(input_image, front3d_intrinsic, front3d_frustum_mask)
  File "/home/tz/panoptic-reconstruction-main/lib/modeling/panoptic_reconstruction.py", line 82, in inference
    _, image_features = self.encoder2d(image)
  File "/home/tz/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/tz/panoptic-reconstruction-main/lib/modeling/backbone/resnet_encoder.py", line 28, in forward
    x = self.conv1(x)
  File "/home/tz/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/tz/.local/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 446, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/tz/.local/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 442, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[1, 4, 240, 320] to have 3 channels, but got 4 channels instead

If I add ' input_image = input_image.convert("RGB") ' to the code, there are some warnings as before and not go to the next step. Looking foward to your reply!

[xheon]

Hi, sorry for the late reply. Do you still have issues?

[itzhuang]

Hi, sorry for the late reply. Do you still have issues?

Thanks for replying me! I think I have run it successfully and it's really an awesome work! And I want to ask if there is an evaluation code for the project?

[xheon]

Great to hear that it works - Feel free to share your changes (here or privately) how you got it to work for your setup. It will send you the evaluation code via email as I'm doing some little clean up.

[itzhuang]

I think the problem is that the png image has four channels but the project need three. I need to learn a lot of knowledeg in the computer vision field, so maybe it's really a easy problem to a lot of people. I plan to write a summary later to note the problems I meet in this project and how I slove them, hope that could help someone. I need to thank you and your team again, from this project I acquire a lot of knowledge that I used don't konw.

[xheon]

Glad you were able to solve it! If there are further questions, feel free to open a new issue.

[viettham1998]

Thanks for replying me! You said the MinkowskiEngine repository really reminds me. I found that I installed the version by following the steps in repository is not your folked version. So I replace the MinkowskiSparseTensor.py and run it again. Then I find there are no errors! Thanks! But when I run the test_net_single_image.py, there are some UserWarnings in the console, I wait for 30 minutes and there are no changes in the console and the output folder don't have an output. Should I wait more time or there are still some wrong in it?

The information in the console: /home/tz/anaconda3/envs/panoptic/bin/python /home/tz/panoptic-reconstruction-main/tools/test_net_single_image.py Load model... Load input image... Perform panoptic 3D scene reconstruction... /home/tz/anaconda3/envs/panoptic/lib/python3.9/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1639180549130/work/aten/src/ATen/native/TensorShape.cpp:2157.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]

/home/tz/panoptic-reconstruction-main/lib/modeling/projection/sparse_projection.py:159: UserWarning: floordiv is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). padding_offsets = difference // 2

/home/tz/anaconda3/envs/panoptic/lib/python3.9/site-packages/MinkowskiEngine/MinkowskiSparseTensor.py:512: UserWarning: floordiv is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). coords = coords // tensor_stride

/home/tz/panoptic-reconstruction-main/lib/modeling/frustum/frustum_completion.py:531: UserWarning: floordiv is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). predicted_coordinates[:, 1:] = predicted_coordinates[:, 1:] // occupancy_prediction.tensor_stride[0]

I got the same error, I tried replacing MinkowskiSparseTensor.py file but still the same error. Can you guide me how to fix it? Looking forward to hearing from you