No module named ingress_srv.ingress_sr

[qingchenkanlu]

Hi, when I run the code , I meet such problems

yusheng@yusheng:~/catkin_ws/src/INVIGORATE/src/scripts$ python dataset_demo.py --scene_num 1 /home/yusheng/catkin_ws/src/INVIGORATE/src/config/../logs/08-25-2021-11-25-14 /home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/../invigorate/invigorate.py:55: UserWarning: No NLP models are loaded. warnings.warn("No NLP models are loaded.") Traceback (most recent call last): File "dataset_demo.py", line 33, in from libraries.caption_generator import caption_generator File "/home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/../libraries/caption_generator/caption_generator.py", line 4, in from ingress_srv.ingress_srv import Ingress ImportError: No module named ingress_srv.ingress_srv

Is there leave out packages? Hope your reply!

[lyfkyle]

Did you init and update the ingress_proj submodule?

[lyfkyle]

I realized that the ingress_proj submodule was not public. I just changed that, you can try to init the ingress_proj submodule again.

[qingchenkanlu]

OK, thanks, I'll have a try

[qingchenkanlu]

when I install ingress_proj , and start ingress, there is another problems, I don't know how to solve

METEOR initialized

ingress_msgs/LocalizeGoal { frame_id : 1 input : sensor_msgs/Image { header : std_msgs/Header { seq : 0 stamp : 0.000000 frame_id : "" } height : 884 width : 1479 encoding : "rgb8" is_bigendian : 0 step : 4437 data : [torch.ByteTensor of size 3922308] } }

THCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-5485/cutorch/lib/THC/generated/../generic/THCTensorMathReduce.cu line=230 error=8 : invalid device function qlua: /root/torch/install/share/lua/5.1/nn/Container.lua:67: In 3 module of nn.Sequential: In 3 module of nn.Sequential: In 1 module of nn.ConcatTable: In 2 module of nn.Sequential: ./densecap/modules/RegularizeLayer.lua:12: cuda runtime error (8) : invalid device function at /tmp/luarocks_cutorch-scm-1-5485/cutorch/lib/THC/generated/../generic/THCTensorMathReduce.cu:230 stack traceback: [C]: in function 'norm' ./densecap/modules/RegularizeLayer.lua:12: in function <./densecap/modules/RegularizeLayer.lua:11> [C]: in function 'xpcall' /root/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors' /root/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function </root/torch/install/share/lua/5.1/nn/Sequential.lua:41> [C]: in function 'xpcall' /root/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors' /root/torch/install/share/lua/5.1/nn/ConcatTable.lua:11: in function </root/torch/install/share/lua/5.1/nn/ConcatTable.lua:9> [C]: in function 'xpcall' /root/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors' ... ./densecap/DenseCapModel.lua:335: in function 'forward_test' localize_server.lua:82: in function 'goal_callback' ...rch/install/share/lua/5.1/ros/actionlib/ActionServer.lua:43: in function 'ActionServer_goalCallback' ...rch/install/share/lua/5.1/ros/actionlib/ActionServer.lua:175: in function 'f' /root/torch/install/share/lua/5.1/ros/Subscriber.lua:101: in function 'triggerCallbacks' /root/torch/install/share/lua/5.1/ros/Subscriber.lua:110: in function 'f' /root/torch/install/share/lua/5.1/ros/CallbackQueue.lua:70: in function 'callSpinCallbacks' /root/torch/install/share/lua/5.1/ros/CallbackQueue.lua:96: in function 'callAvailable' /root/torch/install/share/lua/5.1/ros/ros.lua:76: in function 'spinOnce' localize_server.lua:301: in main chunk

WARNING: If you see a stack trace below, it doesn't point to the place where this error occurred. Please use only the one above. stack traceback: [C]: at 0x7f653a3879c0 [C]: in function 'error' /root/torch/install/share/lua/5.1/nn/Container.lua:67: in function 'rethrowErrors' /root/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward' ./densecap/DenseCapModel.lua:266: in function 'forward' ./densecap/DenseCapModel.lua:335: in function 'forward_test' localize_server.lua:82: in function 'goal_callback' ...rch/install/share/lua/5.1/ros/actionlib/ActionServer.lua:43: in function 'ActionServer_goalCallback' ...rch/install/share/lua/5.1/ros/actionlib/ActionServer.lua:175: in function 'f' /root/torch/install/share/lua/5.1/ros/Subscriber.lua:101: in function 'triggerCallbacks' /root/torch/install/share/lua/5.1/ros/Subscriber.lua:110: in function 'f' /root/torch/install/share/lua/5.1/ros/CallbackQueue.lua:70: in function 'callSpinCallbacks' /root/torch/install/share/lua/5.1/ros/CallbackQueue.lua:96: in function 'callAvailable' /root/torch/install/share/lua/5.1/ros/ros.lua:76: in function 'spinOnce' localize_server.lua:301: in main chunk

when I run the example

root@yusheng:/home/yusheng/catkin_ws/src/INVIGORATE/src/ingress-proj/examples# python interactive_grounding_example.py [INFO] [1629900030.468652]: 1. Waiting for dense_refexp_load action server ... [INFO] [1629900030.542025]: 2. Waiting for dense_refexp_query action server ... [INFO] [1629900030.643989]: Ingress server found! Ready. ^CTraceback (most recent call last): File "interactive_grounding_example.py", line 121, in ground() File "interactive_grounding_example.py", line 50, in ground boxes = np.reshape(load_result.boxes, (-1, 4)) AttributeError: 'NoneType' object has no attribute 'boxes'

[qingchenkanlu]

one more question, is there need two computers to operate the experimnent? I am the first time using docker Thanks!

[lyfkyle]

Try running the following functions inside your dockers. Also, make sure you have nvidia-docker installed as mentioned in https://github.com/AdaCompNUS/ingress-proj/. $ luarocks install cutorch $ luarocks install cunn $ luarocks install cudnn

Actually, if you just want to test INVIGORATE without using INGRESS to generate questions, you can set GENERATE_CAPTION macro to False in dataset_demo.py. You should be able to test the rest.

[lyfkyle]

Regarding your second question, INVIGORATE does not need two machines, but it does need multiple environments, either from docker or conda. This is due to the unfortunate fact that different neural network modules are implemented with different environments. We are trying to port them to a unified environment.

[qingchenkanlu]

Thanks, when I run

cd src/scripts python dataset_demo.py --scene_num 1

the output of the code is a series of numbers, and don't show any result, what' wrong?

[lyfkyle]

Can you attach your output?

[qingchenkanlu]

76, 176, 176, 176, 175, 175, 175, 174, 174, 174, 174, 174, 174, 175, 175, 176, 175, 175, 176, 175, 175, 176, 175, 175, 176, 177, 175, 176, 177, 175, 176, 176, 174, 175, 176, 174, 175, 177, 175, 175, 178, 176, 176, 180, 178, 178, 181, 179, 179, 181, 179, 179, 181, 179, 179, 180, 178, 178, 180, 178, 178, 178, 176, 176, 180, 178, 178, 178, 176, 176, 176, 174, 174, 178, 176, 176, 187, 185, 185, 184, 182, 182, 180, 178, 178, 176, 176, 176, 176, 176, 176, 176, 176, 176, 176, 176, 176, 176, 176, 177, 176, 176, 177, 176, 176, 177, 176, 176, 177, 176, 176, 177, 175, 175, 176, 175, 175, 176, 178, 178, 179, 182, 182, 183, 181, 181, 182, 180, 180, 181, 181, 181, 182, 181, 181, 182, 180, 180, 181, 181, 181, 182, 181, 181, 182, 181, 181, 182, 180, 180, 181, 180, 180, 181, 180, 180, 181, 181, 181, 182, 181, 181, 182, 181, 181, 182, 180, 180, 181, 181, 179, 180, 180, 178, 179, 181, 179, 180, 182, 180, 181, 183, 181, 182, 183, 181, 182, 183, 181, 182, 182, 180, 181, 181, 179, 180, 181, 179, 180, 181, 179, 180, 182, 180, 181, 181, 179, 180, 181, 179, 180, 181, 179, 180, 180, 178, 179, 180, 178, 179, 180, 178, 179, 181, 179, 180, 180, 178, 179, 181, 179, 180, 178, 176, 177, 178, 176, 177, 178, 176, 177, 178, 176, 177, 180, 178, 179, 181, 179, 180, 181, 179, 180, 181, 179, 180, 180, 178, 179, 180, 178, 179, 180, 178, 179, 180, 178, 179, 180, 178, 179, 178, 176, 177, 177, 175, 176, 175, 175, 176, 176, 176, 177, 176, 176, 177, 175, 175, 176, 176, 176, 177, 176, 176, 177, 176, 176, 177, 176, 176, 177, 178, 178, 179, 178, 178, 179, 178, 178, 179, 173, 173, 174, 173, 173, 174, 174, 174, 175, 175, 175, 176, 175, 175, 176, 175, 175, 176, 175, 175, 176, 174, 174, 175, 174, 174, 175, 178, 178, 179, 179, 179, 180, 179, 179, 180, 178, 178, 179, 178, 178, 179, 176, 176, 177, 176, 176, 177, 175, 175, 176, 175, 175, 176, 174, 174, 175, 179, 179, 180, 179, 179, 180, 179, 179, 180, 178, 178, 179, 178, 178, 179, 179, 179, 180, 178, 178, 179, 178, 178, 179, 176, 176, 177, 176, 176, 177, 175, 175, 176, 175, 175, 176, 175, 175, 176, 175, 175, 176, 175, 175, 176, 174, 174, 175, 175, 175, 176, 176, 176, 177, 176, 176, 177, 176, 176, 177, 175, 174, 178, 174, 173, 177, 174, 173, 177, 179, 178, 182, 175, 174, 178, 176, 175, 179, 176, 175, 179, 176, 175, 179, 175, 174, 178, 176, 175, 179, 176, 175, 179, 175, 174, 178, 174, 173, 177, 175, 174, 178, 176, 175, 179, 176, 175, 179, 176, 175, 179, 176, 175, 179, 175, 174, 178, 174, 173, 177, 173, 172, 176, 173, 172, 176, 173, 172, 176, 173, 172, 176, 173, 172, 176, 173, 172, 176, 173, 172, 176, 173, 172, 176, 173, 172, 176, 173, 172, 176, 173, 172, 176, 173, 172, 176, 174, 173, 177, 176, 175, 179, 178, 177, 181, 178, 177, 181, 176, 176, 177, 174, 174, 175, 174, 174, 175, 174, 174, 175, 174, 174, 175, 173, 173, 174, 178, 178, 179, 178, 178, 179, 176, 176, 177, 173, 173, 174, 174, 174, 175, 173, 173, 174, 173, 173, 174, 172, 172, 173, 172, 172, 173, 172, 172, 173, 172, 172, 172, 173, 173, 173, 174, 174, 174, 174, 174, 174, 175, 175, 175, 175, 175, 175, 175, 175, 175, 174, 174, 174, 172, 172, 172, 172, 172, 172, 171, 171, 172, 172, 172, 173, 172, 172, 173, 172, 172, 173, 171, 171, 172, 171, 171, 172, 172, 172, 173, 173, 173, 174, 173, 173, 174, 173, 173, 174, 173, 173, 174, 172, 172, 173, 173, 173, 174, 174, 174, 175, 174, 174, 175, 173, 173, 174, 172, 172, 173, 171, 171, 172, 171, 171, 172, 171, 171, 172, 169, 169, 170, 168, 168, 169, 169, 169, 170, 169, 169, 170, 171, 171, 172, 173, 173, 174, 173, 173, 174, 173, 173, 174, 171, 171, 172, 168, 168, 169, 169, 169, 170, 169, 169, 170, 169, 169, 170, 169, 169, 170, 169, 169, 170, 169, 169, 170, 169, 169, 170, 169, 169, 170, 171, 171, 172, 171, 171, 172, 171, 171, 172, 171, 171, 172, 171, 171, 172, 171, 171, 172, 171, 171, 172, 171, 171, 172, 171, 171, 172, 169, 169, 170, 171, 169, 170, 171, 169, 170, 171, 169, 169, 170, 168, 168, 170, 168, 168, 171, 169, 169, 171, 169, 169, 173, 171, 171, 173, 171, 171, 173, 171, 171, 173, 171, 171, 173, 171, 171, 169, 169, 169, 168, 168, 168, 168, 168, 168, 168, 168, 168, 168, 168, 168, 167, 167, 167, 168, 168, 168, 169, 169, 169, 169, 169, 169, 169, 169, 169, 169, 169, 169, 168, 168, 168, 168, 168, 168, 171, 171, 171, 171, 171, 171, 171, 171, 171, 169, 169, 169, 168, 168, 168, 168, 168, 168, 169, 169, 169, 171, 171, 171, 171, 171, 171, 168, 168, 168, 165, 165, 165, 167, 167, 167, 167, 167, 167, 168, 168, 168, 169, 169, 169, 171, 171, 171, 172, 172, 172, 171, 171, 171, 172, 172, 172, 171, 171, 171, 169, 169, 169, 169, 169, 169, 169, 169, 169, 167, 167, 167, 168, 168, 168, 167, 167, 167, 168, 168, 168, 169, 169, 169, 171, 171, 171, 171, 171, 171, 169, 169, 169, 169, 169, 169, 169, 169, 169, 168, 168, 168, 168, 168, 168, 168, 168, 169, 168, 168, 169, 168, 168, 169, 168, 168, 169, 168, 168, 169, 169, 169, 170, 169, 169, 170, 168, 168, 169, 166, 166, 167, 165, 165, 166, 165, 165, 166, 166, 166, 167, 164, 168, 168, 164, 168, 168, 163, 167, 167, 164, 168, 168, 164, 168, 168, 164, 168, 168, 164, 168, 168, 163, 167, 167, 163, 167, 167, 164, 168, 168, 164, 168, 168, 161, 165, 165, 161, 165, 165, 161, 165, 165, 162, 166, 166, 165, 169, 169, 166, 170, 170, 166, 170, 170, 165, 169, 168, 164, 168, 167, 164, 168, 167, 165, 169, 168, 165, 169, 168, 165, 169, 168, 164, 168, 167, 164, 168, 167, 163, 167, 166, 163, 167, 166, 163, 167, 166, 163, 167, 166, 167, 167, 167, 166, 166, 166, 166, 166, 166, 166, 166, 166, 165, 165, 165, 164, 164, 164, 166, 166, 166, 167, 167, 167, 168, 168, 168, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 162, 166, 165, 162, 166, 165, 161, 165, 164, 161, 165, 164, 161, 165, 164, 161, 165, 164, 162, 166, 165, 162, 166, 165, 162, 166, 165, 162, 166, 165, 160, 166, 166, 159, 165, 165, 159, 165, 165, 159, 165, 165, 159, 165, 165, 160, 166, 166, 160, 166, 166, 161, 167, 167, 159, 165, 165, 159, 165, 165, 159, 165, 165, 159, 165, 165, 160, 166, 166, 160, 166, 166, 160, 166, 166, 160, 166, 166, 158, 167, 166, 158, 167, 166, 158, 167, 166, 157, 166, 165, 155, 164, 163, 154, 163, 162, 155, 164, 163, 157, 166, 165, 157, 166, 165, 155, 164, 163, 154, 163, 161, 154, 163, 161, 153, 162, 160, 153, 162, 160, 155, 161, 160, 152, 158, 157, 152, 158, 157, 152, 158, 157, 152, 158, 157, 153, 159, 158, 155, 159, 158, 155, 159, 158, 148, 152, 151, 149, 153, 152, 152, 156, 155, 152, 156, 155, 149, 153, 152, 146, 150, 149, 143, 147, 146, 141, 145, 144, 142, 142, 142, 142, 142, 142, 142, 142, 142, 142, 142, 142, 143, 143, 143, 145, 145, 145, 145, 145, 146, 145, 145, 146, 142, 146, 146, 142, 146, 146, 142, 146, 146, 142, 146, 146, 142, 146, 146, 142, 146, 146, 141, 145, 145, 140, 144, 144, 141, 145, 145, 141, 145, 145, 141, 145, 145, 141, 145, 145, 141, 145, 145, 140, 144, 144, 141, 145, 144, 146, 150, 149, 147, 151, 150, 149, 153, 152, 146, 152, 151, 144, 150, 149, 149, 155, 154, 150, 156, 155, 150, 156, 155, 152, 158, 157, 150, 156, 155, 152, 158, 157, 153, 159, 158, 153, 159, 158, 153, 159, 158, 154, 160, 159, 154, 160, 160, 154, 160, 160, 155, 161, 161, 155, 161, 161, 155, 161, 161, 156, 162, 162, 155, 161, 161, 155, 161, 161, 153, 159, 159, 153, 159, 159, 153, 159, 159, 154, 160, 160, 153, 159, 159, 153, 159, 159, 153, 159, 159, 153, 159, 159, 156, 160, 159, 156, 160, 159, 156, 160, 159, 156, 160, 159, 157, 161, 160, 156, 160, 159, 156, 160, 159, 155, 159, 158, 155, 159, 159, 155, 159, 159, 155, 159, 159, 156, 160, 160, 157, 161, 161, 157, 161, 161, 158, 162, 162, 157, 161, 161, 159, 163, 163, 159, 163, 163, 158, 162, 162, 161, 165, 165, 154, 160, 160, 160, 166, 166, 157, 163, 163, 157, 163, 163, 157, 163, 163, 157, 163, 163, 157, 163, 163, 157, 163, 163, 157, 163, 162, 157, 163, 162, 157, 163, 162, 156, 162, 161, 155, 161, 160, 156, 162, 161, 155, 161, 160, 155, 161, 160, 155, 161, 160, 155, 161, 160, 156, 162, 161, 155, 161, 160, 155, 162, 158, 154, 161, 157, 153, 160, 156, 153, 160, 156, 153, 160, 156, 153, 160, 156, 153, 160, 156, 154, 161, 157, 154, 161, 157, 154, 161, 157, 153, 160, 156, 150, 157, 153, 150, 157, 153, 150, 157, 153, 155, 162, 158, 155, 162, 158, 153, 160, 156, 153, 160, 156, 153, 160, 156, 154, 161, 157, 154, 161, 157, 153, 160, 156, 153, 160, 156, 153, 160, 156, 153, 160, 156, 153, 160, 156, 152, 159, 155, 152, 159, 155, 152, 158, 157, 152, 158, 157, 152, 158, 158, 150, 156, 156, 153, 159, 159, 153, 159, 159, 154, 160, 160, 154, 160, 160, 154, 160, 160, 152, 158, 158, 150, 156, 156, 150, 156, 156, 152, 158, 158, 153, 159, 159, 154, 160, 160, 153, 159, 159, 152, 158, 158, 153, 159, 159, 153, 159, 159, 153, 159, 159, 152, 158, 158, 153, 159, 159, 153, 159, 159, 153, 159, 159, 153, 159, 159, 153, 159, 159, 152, 158, 158, 152, 158, 158, 150, 156, 156, 149, 155, 155, 149, 155, 155, 150, 156, 156, 150, 156, 156, 150, 156, 156, 152, 158, 157, 152, 158, 157, 153, 159, 158, 153, 159, 158, 153, 159, 158, 153, 159, 158, 152, 158, 157, 150, 156, 155, 149, 156, 152, 149, 156, 152, 149, 156, 152, 149, 156, 152, 150, 157, 153, 150, 157, 153, 152, 158, 157, 153, 159, 158, 155, 161, 160, 155, 161, 160, 155, 161, 161, 155, 161, 161, 154, 160, 160, 153, 159, 159, 155, 159, 159, 155, 159, 159], None)

it is too many numbers, the teminal is full

[qingchenkanlu]

when the code run to https://github.com/AdaCompNUS/INVIGORATE/blob/03a55daa6904420cf759848c706e5a7b47112e41/src/scripts/dataset_demo.py#L106

the code stop

[lyfkyle]

The terminal shows a series of numbers because it is printing an image for some reason. I just ran the dataset demo on my machine and it works fine. Can you try running the script in a debugger to see exactly where the error occurs?

[qingchenkanlu]

/home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/dataset_demo.py(237)() -> main(args) (Pdb) 2021-08-26 11:11:35,429 - invigorate_logger - INFO - waiting for services... 2021-08-26 11:11:35,520 - invigorate_logger - INFO - img_path = /home/yusheng/catkin_ws/src/INVIGORATE/src/config/../dataset/27/origin.png 2021-08-26 11:11:35,520 - invigorate_logger - INFO - expr = remote 2021-08-26 11:11:35,521 - invigorate_logger - INFO - ---------------------------------------------------------------------------------------- 2021-08-26 11:11:35,521 - invigorate_logger - INFO - Start of iteration TypeError: TypeErro... None)',) /home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/dataset_demo.py(237)() -> main(args) (Pdb) --Return-- /home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/dataset_demo.py(237)()->None -> main(args) (Pdb) TypeError: TypeErro... None)',)

(1)()->None (Pdb) --Return-- (1)()->None

68, 168, 168, 168, 169, 169, 169, 171, 171, 171, 171, 171, 171, 168, 168, 168, 165, 165, 165, 167, 167, 167, 167, 167, 167, 168, 168, 168, 169, 169, 169, 171, 171, 171, 172, 172, 172, 171, 171, 171, 172, 172, 172, 171, 171, 171, 169, 169, 169, 169, 169, 169, 169, 169, 169, 167, 167, 167, 168, 168, 168, 167, 167, 167, 168, 168, 168, 169, 169, 169, 171, 171, 171, 171, 171, 171, 169, 169, 169, 169, 169, 169, 169, 169, 169, 168, 168, 168, 168, 168, 168, 168, 168, 169, 168, 168, 169, 168, 168, 169, 168, 168, 169, 168, 168, 169, 169, 169, 170, 169, 169, 170, 168, 168, 169, 166, 166, 167, 165, 165, 166, 165, 165, 166, 166, 166, 167, 164, 168, 168, 164, 168, 168, 163, 167, 167, 164, 168, 168, 164, 168, 168, 164, 168, 168, 164, 168, 168, 163, 167, 167, 163, 167, 167, 164, 168, 168, 164, 168, 168, 161, 165, 165, 161, 165, 165, 161, 165, 165, 162, 166, 166, 165, 169, 169, 166, 170, 170, 166, 170, 170, 165, 169, 168, 164, 168, 167, 164, 168, 167, 165, 169, 168, 165, 169, 168, 165, 169, 168, 164, 168, 167, 164, 168, 167, 163, 167, 166, 163, 167, 166, 163, 167, 166, 163, 167, 166, 167, 167, 167, 166, 166, 166, 166, 166, 166, 166, 166, 166, 165, 165, 165, 164, 164, 164, 166, 166, 166, 167, 167, 167, 168, 168, 168, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 162, 166, 165, 162, 166, 165, 161, 165, 164, 161, 165, 164, 161, 165, 164, 161, 165, 164, 162, 166, 165, 162, 166, 165, 162, 166, 165, 162, 166, 165, 160, 166, 166, 159, 165, 165, 159, 165, 165, 159, 165, 165, 159, 165, 165, 160, 166, 166, 160, 166, 166, 161, 167, 167, 159, 165, 165, 159, 165, 165, 159, 165, 165, 159, 165, 165, 160, 166, 166, 160, 166, 166, 160, 166, 166, 160, 166, 166, 158, 167, 166, 158, 167, 166, 158, 167, 166, 157, 166, 165, 155, 164, 163, 154, 163, 162, 155, 164, 163, 157, 166, 165, 157, 166, 165, 155, 164, 163, 154, 163, 161, 154, 163, 161, 153, 162, 160, 153, 162, 160, 155, 161, 160, 152, 158, 157, 152, 158, 157, 152, 158, 157, 152, 158, 157, 153, 159, 158, 155, 159, 158, 155, 159, 158, 148, 152, 151, 149, 153, 152, 152, 156, 155, 152, 156, 155, 149, 153, 152, 146, 150, 149, 143, 147, 146, 141, 145, 144, 142, 142, 142, 142, 142, 142, 142, 142, 142, 142, 142, 142, 143, 143, 143, 145, 145, 145, 145, 145, 146, 145, 145, 146, 142, 146, 146, 142, 146, 146, 142, 146, 146, 142, 146, 146, 142, 146, 146, 142, 146, 146, 141, 145, 145, 140, 144, 144, 141, 145, 145, 141, 145, 145, 141, 145, 145, 141, 145, 145, 141, 145, 145, 140, 144, 144, 141, 145, 144, 146, 150, 149, 147, 151, 150, 149, 153, 152, 146, 152, 151, 144, 150, 149, 149, 155, 154, 150, 156, 155, 150, 156, 155, 152, 158, 157, 150, 156, 155, 152, 158, 157, 153, 159, 158, 153, 159, 158, 153, 159, 158, 154, 160, 159, 154, 160, 160, 154, 160, 160, 155, 161, 161, 155, 161, 161, 155, 161, 161, 156, 162, 162, 155, 161, 161, 155, 161, 161, 153, 159, 159, 153, 159, 159, 153, 159, 159, 154, 160, 160, 153, 159, 159, 153, 159, 159, 153, 159, 159, 153, 159, 159, 156, 160, 159, 156, 160, 159, 156, 160, 159, 156, 160, 159, 157, 161, 160, 156, 160, 159, 156, 160, 159, 155, 159, 158, 155, 159, 159, 155, 159, 159, 155, 159, 159, 156, 160, 160, 157, 161, 161, 157, 161, 161, 158, 162, 162, 157, 161, 161, 159, 163, 163, 159, 163, 163, 158, 162, 162, 161, 165, 165, 154, 160, 160, 160, 166, 166, 157, 163, 163, 157, 163, 163, 157, 163, 163, 157, 163, 163, 157, 163, 163, 157, 163, 163, 157, 163, 162, 157, 163, 162, 157, 163, 162, 156, 162, 161, 155, 161, 160, 156, 162, 161, 155, 161, 160, 155, 161, 160, 155, 161, 160, 155, 161, 160, 156, 162, 161, 155, 161, 160, 155, 162, 158, 154, 161, 157, 153, 160, 156, 153, 160, 156, 153, 160, 156, 153, 160, 156, 153, 160, 156, 154, 161, 157, 154, 161, 157, 154, 161, 157, 153, 160, 156, 150, 157, 153, 150, 157, 153, 150, 157, 153, 155, 162, 158, 155, 162, 158, 153, 160, 156, 153, 160, 156, 153, 160, 156, 154, 161, 157, 154, 161, 157, 153, 160, 156, 153, 160, 156, 153, 160, 156, 153, 160, 156, 153, 160, 156, 152, 159, 155, 152, 159, 155, 152, 158, 157, 152, 158, 157, 152, 158, 158, 150, 156, 156, 153, 159, 159, 153, 159, 159, 154, 160, 160, 154, 160, 160, 154, 160, 160, 152, 158, 158, 150, 156, 156, 150, 156, 156, 152, 158, 158, 153, 159, 159, 154, 160, 160, 153, 159, 159, 152, 158, 158, 153, 159, 159, 153, 159, 159, 153, 159, 159, 152, 158, 158, 153, 159, 159, 153, 159, 159, 153, 159, 159, 153, 159, 159, 153, 159, 159, 152, 158, 158, 152, 158, 158, 150, 156, 156, 149, 155, 155, 149, 155, 155, 150, 156, 156, 150, 156, 156, 150, 156, 156, 152, 158, 157, 152, 158, 157, 153, 159, 158, 153, 159, 158, 153, 159, 158, 153, 159, 158, 152, 158, 157, 150, 156, 155, 149, 156, 152, 149, 156, 152, 149, 156, 152, 149, 156, 152, 150, 157, 153, 150, 157, 153, 152, 158, 157, 153, 159, 158, 155, 161, 160, 155, 161, 160, 155, 161, 161, 155, 161, 161, 154, 160, 160, 153, 159, 159, 155, 159, 159, 155, 159, 159], None) Uncaught exception. Entering post mortem debugging Running 'cont' or 'step' will restart the program

/opt/ros/kinetic/lib/python2.7/dist-packages/genpy/message.py(344)init() -> raise TypeError('Invalid number of arguments, args should be %s' % str(self.slots) + ' args are' + str(args))

[ZhangHanbo]

/home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/dataset_demo.py(237)() -> main(args) (Pdb) 2021-08-26 11:11:35,429 - invigorate_logger - INFO - waiting for services... 2021-08-26 11:11:35,520 - invigorate_logger - INFO - img_path = /home/yusheng/catkin_ws/src/INVIGORATE/src/config/../dataset/27/origin.png 2021-08-26 11:11:35,520 - invigorate_logger - INFO - expr = remote 2021-08-26 11:11:35,521 - invigorate_logger - INFO - ---------------------------------------------------------------------------------------- 2021-08-26 11:11:35,521 - invigorate_logger - INFO - Start of iteration TypeError: TypeErro... None)',) /home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/dataset_demo.py(237)() -> main(args) (Pdb) --Return-- /home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/dataset_demo.py(237)()->None -> main(args) (Pdb) TypeError: TypeErro... None)',) (1)()->None (Pdb) --Return-- (1)()->None

68, 168, 168, 168, 169, 169, 169, 171, 171, 171, 171, 171, 171, 168, 168, 168, 165, 165, 165, 167, 167, 167, 167, 167, 167, 168, 168, 168, 169, 169, 169, 171, 171, 171, 172, 172, 172, 171, 171, 171, 172, 172, 172, 171, 171, 171, 169, 169, 169, 169, 169, 169, 169, 169, 169, 167, 167, 167, 168, 168, 168, 167, 167, 167, 168, 168, 168, 169, 169, 169, 171, 171, 171, 171, 171, 171, 169, 169, 169, 169, 169, 169, 169, 169, 169, 168, 168, 168, 168, 168, 168, 168, 168, 169, 168, 168, 169, 168, 168, 169, 168, 168, 169, 168, 168, 169, 169, 169, 170, 169, 169, 170, 168, 168, 169, 166, 166, 167, 165, 165, 166, 165, 165, 166, 166, 166, 167, 164, 168, 168, 164, 168, 168, 163, 167, 167, 164, 168, 168, 164, 168, 168, 164, 168, 168, 164, 168, 168, 163, 167, 167, 163, 167, 167, 164, 168, 168, 164, 168, 168, 161, 165, 165, 161, 165, 165, 161, 165, 165, 162, 166, 166, 165, 169, 169, 166, 170, 170, 166, 170, 170, 165, 169, 168, 164, 168, 167, 164, 168, 167, 165, 169, 168, 165, 169, 168, 165, 169, 168, 164, 168, 167, 164, 168, 167, 163, 167, 166, 163, 167, 166, 163, 167, 166, 163, 167, 166, 167, 167, 167, 166, 166, 166, 166, 166, 166, 166, 166, 166, 165, 165, 165, 164, 164, 164, 166, 166, 166, 167, 167, 167, 168, 168, 168, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 166, 162, 166, 165, 162, 166, 165, 161, 165, 164, 161, 165, 164, 161, 165, 164, 161, 165, 164, 162, 166, 165, 162, 166, 165, 162, 166, 165, 162, 166, 165, 160, 166, 166, 159, 165, 165, 159, 165, 165, 159, 165, 165, 159, 165, 165, 160, 166, 166, 160, 166, 166, 161, 167, 167, 159, 165, 165, 159, 165, 165, 159, 165, 165, 159, 165, 165, 160, 166, 166, 160, 166, 166, 160, 166, 166, 160, 166, 166, 158, 167, 166, 158, 167, 166, 158, 167, 166, 157, 166, 165, 155, 164, 163, 154, 163, 162, 155, 164, 163, 157, 166, 165, 157, 166, 165, 155, 164, 163, 154, 163, 161, 154, 163, 161, 153, 162, 160, 153, 162, 160, 155, 161, 160, 152, 158, 157, 152, 158, 157, 152, 158, 157, 152, 158, 157, 153, 159, 158, 155, 159, 158, 155, 159, 158, 148, 152, 151, 149, 153, 152, 152, 156, 155, 152, 156, 155, 149, 153, 152, 146, 150, 149, 143, 147, 146, 141, 145, 144, 142, 142, 142, 142, 142, 142, 142, 142, 142, 142, 142, 142, 143, 143, 143, 145, 145, 145, 145, 145, 146, 145, 145, 146, 142, 146, 146, 142, 146, 146, 142, 146, 146, 142, 146, 146, 142, 146, 146, 142, 146, 146, 141, 145, 145, 140, 144, 144, 141, 145, 145, 141, 145, 145, 141, 145, 145, 141, 145, 145, 141, 145, 145, 140, 144, 144, 141, 145, 144, 146, 150, 149, 147, 151, 150, 149, 153, 152, 146, 152, 151, 144, 150, 149, 149, 155, 154, 150, 156, 155, 150, 156, 155, 152, 158, 157, 150, 156, 155, 152, 158, 157, 153, 159, 158, 153, 159, 158, 153, 159, 158, 154, 160, 159, 154, 160, 160, 154, 160, 160, 155, 161, 161, 155, 161, 161, 155, 161, 161, 156, 162, 162, 155, 161, 161, 155, 161, 161, 153, 159, 159, 153, 159, 159, 153, 159, 159, 154, 160, 160, 153, 159, 159, 153, 159, 159, 153, 159, 159, 153, 159, 159, 156, 160, 159, 156, 160, 159, 156, 160, 159, 156, 160, 159, 157, 161, 160, 156, 160, 159, 156, 160, 159, 155, 159, 158, 155, 159, 159, 155, 159, 159, 155, 159, 159, 156, 160, 160, 157, 161, 161, 157, 161, 161, 158, 162, 162, 157, 161, 161, 159, 163, 163, 159, 163, 163, 158, 162, 162, 161, 165, 165, 154, 160, 160, 160, 166, 166, 157, 163, 163, 157, 163, 163, 157, 163, 163, 157, 163, 163, 157, 163, 163, 157, 163, 163, 157, 163, 162, 157, 163, 162, 157, 163, 162, 156, 162, 161, 155, 161, 160, 156, 162, 161, 155, 161, 160, 155, 161, 160, 155, 161, 160, 155, 161, 160, 156, 162, 161, 155, 161, 160, 155, 162, 158, 154, 161, 157, 153, 160, 156, 153, 160, 156, 153, 160, 156, 153, 160, 156, 153, 160, 156, 154, 161, 157, 154, 161, 157, 154, 161, 157, 153, 160, 156, 150, 157, 153, 150, 157, 153, 150, 157, 153, 155, 162, 158, 155, 162, 158, 153, 160, 156, 153, 160, 156, 153, 160, 156, 154, 161, 157, 154, 161, 157, 153, 160, 156, 153, 160, 156, 153, 160, 156, 153, 160, 156, 153, 160, 156, 152, 159, 155, 152, 159, 155, 152, 158, 157, 152, 158, 157, 152, 158, 158, 150, 156, 156, 153, 159, 159, 153, 159, 159, 154, 160, 160, 154, 160, 160, 154, 160, 160, 152, 158, 158, 150, 156, 156, 150, 156, 156, 152, 158, 158, 153, 159, 159, 154, 160, 160, 153, 159, 159, 152, 158, 158, 153, 159, 159, 153, 159, 159, 153, 159, 159, 152, 158, 158, 153, 159, 159, 153, 159, 159, 153, 159, 159, 153, 159, 159, 153, 159, 159, 152, 158, 158, 152, 158, 158, 150, 156, 156, 149, 155, 155, 149, 155, 155, 150, 156, 156, 150, 156, 156, 150, 156, 156, 152, 158, 157, 152, 158, 157, 153, 159, 158, 153, 159, 158, 153, 159, 158, 153, 159, 158, 152, 158, 157, 150, 156, 155, 149, 156, 152, 149, 156, 152, 149, 156, 152, 149, 156, 152, 150, 157, 153, 150, 157, 153, 152, 158, 157, 153, 159, 158, 155, 161, 160, 155, 161, 160, 155, 161, 161, 155, 161, 161, 154, 160, 160, 153, 159, 159, 155, 159, 159, 155, 159, 159], None) Uncaught exception. Entering post mortem debugging Running 'cont' or 'step' will restart the program

/opt/ros/kinetic/lib/python2.7/dist-packages/genpy/message.py(344)init() -> raise TypeError('Invalid number of arguments, args should be %s' % str(self.slots) + ' args are' + str(args))

Could you provide the information of your python version, CUDA version, and the model of your graphic card?

[qingchenkanlu]

python==2.7 CUDA=9.0 GPU gtx1080Ti

[lyfkyle]

I would like to suggest you to take a look at the object_detection_service output. I suspect the print is from there.

[qingchenkanlu]

yusheng@yusheng:~/catkin_ws/src/INVIGORATE/src$ sh launch_detectron2.sh fvcore version of PathManager will be deprecated soon. Please migrate to the version in iopath repo. https://github.com/facebookresearch/iopath

/home/yusheng/catkin_ws/src/INVIGORATE/src/config/../logs/08-26-2021-10-45-57 [INFO] [1629945962.213359]: object_detection_srv inited

there is no output, and the result image is not show

[ZhangHanbo]

python==2.7 CUDA=9.0 GPU gtx1080Ti

It seems fine. But we cannot reproduce your output here. It is suggested to firstly locate where the bug appears.

You can separately run the modules, e.g., the detectron2 server, the mattnet server, and the mrt server to see whether they perform in an expected way.

[lyfkyle]

Are you using ubuntu16.04 and ROS kinetic?

[qingchenkanlu]

Are you using ubuntu16.04 and ROS kinetic?

yes

[qingchenkanlu]

yusheng@yusheng:~/catkin_ws/src/INVIGORATE/src$ sh launch_detectron2.sh fvcore version of PathManager will be deprecated soon. Please migrate to the version in iopath repo. https://github.com/facebookresearch/iopath

/home/yusheng/catkin_ws/src/INVIGORATE/src/config/../logs/08-26-2021-10-45-57 [INFO] [1629945962.213359]: object_detection_srv inited

yusheng@yusheng:~/catkin_ws/src/INVIGORATE/src$ sh launch_mattnet.sh /usr/local/lib/python2.7/dist-packages/torch/nn/modules/rnn.py:38: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.2 and num_layers=1 "num_layers={}".format(dropout, num_layers)) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'layers.joint_match.JointMatching' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.sparse.Embedding' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.dropout.Dropout' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.container.Sequential' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.linear.Linear' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.activation.ReLU' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.rnn.LSTM' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'layers.visual_encoder.SubjectEncoder' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.batchnorm.BatchNorm1d' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.activation.Tanh' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) MatNet [refcoco_unc's mrcn_cmr_with_st] loaded in 3.26 seconds. /home/yusheng/catkin_ws/src/INVIGORATE/src/mattnet_server/MAttNet/tools/../pyutils/mask-faster-rcnn/lib/model/config.py:362: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. yaml_cfg = edict(yaml.load(f)) Using config: {'ANCHOR_RATIOS': [0.5, 1, 2], 'ANCHOR_SCALES': [4, 8, 16, 32], 'DATA_DIR': '/home/yusheng/catkin_ws/src/INVIGORATE/src/mattnet_server/MAttNet/pyutils/mask-faster-rcnn/data', 'EXP_DIR': 'res101', 'MASK_SIZE': 14, 'MATLAB': 'matlab', 'MOBILENET': {'DEPTH_MULTIPLIER': 1.0, 'FIXED_LAYERS': 5, 'REGU_DEPTH': False, 'WEIGHT_DECAY': 4e-05}, 'PIXEL_MEANS': array([[[102.9801, 115.9465, 122.7717]]]), 'POOLING_ALIGN': False, 'POOLING_MODE': 'crop', 'POOLING_SIZE': 7, 'RESNET': {'FIXED_BLOCKS': 1, 'MAX_POOL': False}, 'RNG_SEED': 3, 'ROOT_DIR': '/home/yusheng/catkin_ws/src/INVIGORATE/src/mattnet_server/MAttNet/pyutils/mask-faster-rcnn', 'TEST': {'BBOX_REG': True, 'HAS_RPN': True, 'MAX_SIZE': 1000, 'MODE': 'nms', 'NMS': 0.3, 'PROPOSAL_METHOD': 'gt', 'RPN_NMS_THRESH': 0.7, 'RPN_POST_NMS_TOP_N': 300, 'RPN_PRE_NMS_TOP_N': 6000, 'RPN_TOP_N': 5000, 'SCALES': [600], 'SVM': False}, 'TRAIN': {'ASPECT_GROUPING': False, 'BATCH_SIZE': 256, 'BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0], 'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0], 'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2], 'BBOX_NORMALIZE_TARGETS': True, 'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': True, 'BBOX_REG': True, 'BBOX_THRESH': 0.5, 'BG_THRESH_HI': 0.5, 'BG_THRESH_LO': 0.0, 'BIAS_DECAY': False, 'DISPLAY': 20, 'DOUBLE_BIAS': False, 'FG_FRACTION': 0.25, 'FG_THRESH': 0.5, 'FROM_FRCN': False, 'GAMMA': 0.1, 'HAS_RPN': True, 'IMS_PER_BATCH': 1, 'LEARNING_RATE': 0.001, 'MAX_SIZE': 1000, 'MOMENTUM': 0.9, 'PROPOSAL_METHOD': 'gt', 'RPN_BATCHSIZE': 256, 'RPN_BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0], 'RPN_CLOBBER_POSITIVES': False, 'RPN_FG_FRACTION': 0.5, 'RPN_NEGATIVE_OVERLAP': 0.3, 'RPN_NMS_THRESH': 0.7, 'RPN_POSITIVE_OVERLAP': 0.7, 'RPN_POSITIVE_WEIGHT': -1.0, 'RPN_POST_NMS_TOP_N': 2000, 'RPN_PRE_NMS_TOP_N': 12000, 'SCALES': [600], 'SNAPSHOT_ITERS': 5000, 'SNAPSHOT_KEPT': 3, 'SNAPSHOT_PREFIX': 'res101_mask_rcnn', 'STEPSIZE': [30000], 'SUMMARY_INTERVAL': 180, 'TRUNCATED': False, 'USE_ALL_GT': True, 'USE_FLIPPED': True, 'USE_GT': False, 'WEIGHT_DECAY': 0.0001}, 'USE_GPU_NMS': True} loading annotations into memory... Done (t=0.65s) creating index... index created! /home/yusheng/catkin_ws/src/INVIGORATE/src/mattnet_server/MAttNet/tools/../lib/mrcn/../../pyutils/mask-faster-rcnn output/res101/coco_2014_train_minus_refer_valtest+coco_2014_valminusminival/notime/res101_mask_rcnn_iter_1250000.pth pretrained-model loaded from [/home/yusheng/catkin_ws/src/INVIGORATE/src/mattnet_server/MAttNet/tools/../lib/mrcn/../../pyutils/mask-faster-rcnn/output/res101/coco_2014_train_minus_refer_valtest+coco_2014_valminusminival/notime/res101_mask_rcnn_iter_1250000.pth]. Mask R-CNN: imdb[coco_minus_refer], tag[notime], id[res101_mask_rcnn_iter_1250000] loaded in 1.58 seconds. Ready to ground object.

yusheng@yusheng:~/catkin_ws/src/INVIGORATE/src$ sh launch_vmrn.sh model path: ../vmrn_old/model/init.pyc ../vmrn_old/data/coco/PythonAPI /home/yusheng/catkin_ws/src/INVIGORATE/src/mrt_detector/src /home/yusheng/catkin_ws/devel/lib/python2.7/dist-packages /opt/ros/kinetic/lib/python2.7/dist-packages /usr/lib/python2.7 /usr/lib/python2.7/plat-x86_64-linux-gnu /usr/lib/python2.7/lib-tk /usr/lib/python2.7/lib-old /usr/lib/python2.7/lib-dynload /usr/local/lib/python2.7/dist-packages /home/yusheng/code/Visual-new/model/roi_layers /usr/local/lib/python2.7/dist-packages/pycocotools-2.0-py2.7-linux-x86_64.egg /usr/local/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages/PILcompat /usr/lib/python2.7/dist-packages/gtk-2.0 /usr/lib/python2.7/dist-packages/wx-3.0-gtk2 ../vmrn_old ../vmrn_old/model/utils/config.py:407: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. yaml_cfg = edict(yaml.load(f)) Loading pretrained weights from /home/yusheng/catkin_ws/src/INVIGORATE/src/mrt_detector/vmrn_old/model/../data/pretrained_model/resnet101_caffe.pth ../vmrn_old/model/AllinOne.py:856: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaimingnormal. init.kaiming_normal(param, nonlinearity='relu') load checkpoint ../vmrn_old/output/res101/vmrdcompv1/all_in_one_1_25_1407_0.pth load model successfully! Ready to detect manipulation relationships.

I think there is no bugs

python==2.7 CUDA=9.0 GPU gtx1080Ti

It seems fine. But we cannot reproduce your output here. It is suggested to firstly locate where the bug appears.

You can separately run the modules, e.g., the detectron2 server, the mattnet server, and the mrt server to see whether they perform in an expected way.

[ZhangHanbo]

Are you using ubuntu16.04 and ROS kinetic?

yusheng@yusheng:~/catkin_ws/src/INVIGORATE/src$ sh launch_detectron2.sh fvcore version of PathManager will be deprecated soon. Please migrate to the version in iopath repo. https://github.com/facebookresearch/iopath

/home/yusheng/catkin_ws/src/INVIGORATE/src/config/../logs/08-26-2021-10-45-57 [INFO] [1629945962.213359]: object_detection_srv inited

yusheng@yusheng:~/catkin_ws/src/INVIGORATE/src$ sh launch_mattnet.sh /usr/local/lib/python2.7/dist-packages/torch/nn/modules/rnn.py:38: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.2 and num_layers=1 "num_layers={}".format(dropout, num_layers)) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'layers.joint_match.JointMatching' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.sparse.Embedding' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.dropout.Dropout' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.container.Sequential' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.linear.Linear' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.activation.ReLU' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.rnn.LSTM' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'layers.visual_encoder.SubjectEncoder' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.batchnorm.BatchNorm1d' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) /usr/local/lib/python2.7/dist-packages/torch/serialization.py:367: SourceChangeWarning: source code of class 'torch.nn.modules.activation.Tanh' has changed. you can retrieve the original source code by accessing the object's source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning) MatNet [refcoco_unc's mrcn_cmr_with_st] loaded in 3.26 seconds. /home/yusheng/catkin_ws/src/INVIGORATE/src/mattnet_server/MAttNet/tools/../pyutils/mask-faster-rcnn/lib/model/config.py:362: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. yaml_cfg = edict(yaml.load(f)) Using config: {'ANCHOR_RATIOS': [0.5, 1, 2], 'ANCHOR_SCALES': [4, 8, 16, 32], 'DATA_DIR': '/home/yusheng/catkin_ws/src/INVIGORATE/src/mattnet_server/MAttNet/pyutils/mask-faster-rcnn/data', 'EXP_DIR': 'res101', 'MASK_SIZE': 14, 'MATLAB': 'matlab', 'MOBILENET': {'DEPTH_MULTIPLIER': 1.0, 'FIXED_LAYERS': 5, 'REGU_DEPTH': False, 'WEIGHT_DECAY': 4e-05}, 'PIXEL_MEANS': array([[[102.9801, 115.9465, 122.7717]]]), 'POOLING_ALIGN': False, 'POOLING_MODE': 'crop', 'POOLING_SIZE': 7, 'RESNET': {'FIXED_BLOCKS': 1, 'MAX_POOL': False}, 'RNG_SEED': 3, 'ROOT_DIR': '/home/yusheng/catkin_ws/src/INVIGORATE/src/mattnet_server/MAttNet/pyutils/mask-faster-rcnn', 'TEST': {'BBOX_REG': True, 'HAS_RPN': True, 'MAX_SIZE': 1000, 'MODE': 'nms', 'NMS': 0.3, 'PROPOSAL_METHOD': 'gt', 'RPN_NMS_THRESH': 0.7, 'RPN_POST_NMS_TOP_N': 300, 'RPN_PRE_NMS_TOP_N': 6000, 'RPN_TOP_N': 5000, 'SCALES': [600], 'SVM': False}, 'TRAIN': {'ASPECT_GROUPING': False, 'BATCH_SIZE': 256, 'BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0], 'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0], 'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2], 'BBOX_NORMALIZE_TARGETS': True, 'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': True, 'BBOX_REG': True, 'BBOX_THRESH': 0.5, 'BG_THRESH_HI': 0.5, 'BG_THRESH_LO': 0.0, 'BIAS_DECAY': False, 'DISPLAY': 20, 'DOUBLE_BIAS': False, 'FG_FRACTION': 0.25, 'FG_THRESH': 0.5, 'FROM_FRCN': False, 'GAMMA': 0.1, 'HAS_RPN': True, 'IMS_PER_BATCH': 1, 'LEARNING_RATE': 0.001, 'MAX_SIZE': 1000, 'MOMENTUM': 0.9, 'PROPOSAL_METHOD': 'gt', 'RPN_BATCHSIZE': 256, 'RPN_BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0], 'RPN_CLOBBER_POSITIVES': False, 'RPN_FG_FRACTION': 0.5, 'RPN_NEGATIVE_OVERLAP': 0.3, 'RPN_NMS_THRESH': 0.7, 'RPN_POSITIVE_OVERLAP': 0.7, 'RPN_POSITIVE_WEIGHT': -1.0, 'RPN_POST_NMS_TOP_N': 2000, 'RPN_PRE_NMS_TOP_N': 12000, 'SCALES': [600], 'SNAPSHOT_ITERS': 5000, 'SNAPSHOT_KEPT': 3, 'SNAPSHOT_PREFIX': 'res101_mask_rcnn', 'STEPSIZE': [30000], 'SUMMARY_INTERVAL': 180, 'TRUNCATED': False, 'USE_ALL_GT': True, 'USE_FLIPPED': True, 'USE_GT': False, 'WEIGHT_DECAY': 0.0001}, 'USE_GPU_NMS': True} loading annotations into memory... Done (t=0.65s) creating index... index created! /home/yusheng/catkin_ws/src/INVIGORATE/src/mattnet_server/MAttNet/tools/../lib/mrcn/../../pyutils/mask-faster-rcnn output/res101/coco_2014_train_minus_refer_valtest+coco_2014_valminusminival/notime/res101_mask_rcnn_iter_1250000.pth pretrained-model loaded from [/home/yusheng/catkin_ws/src/INVIGORATE/src/mattnet_server/MAttNet/tools/../lib/mrcn/../../pyutils/mask-faster-rcnn/output/res101/coco_2014_train_minus_refer_valtest+coco_2014_valminusminival/notime/res101_mask_rcnn_iter_1250000.pth]. Mask R-CNN: imdb[coco_minus_refer], tag[notime], id[res101_mask_rcnn_iter_1250000] loaded in 1.58 seconds. Ready to ground object.

yusheng@yusheng:~/catkin_ws/src/INVIGORATE/src$ sh launch_vmrn.sh model path: ../vmrn_old/model/init.pyc ../vmrn_old/data/coco/PythonAPI /home/yusheng/catkin_ws/src/INVIGORATE/src/mrt_detector/src /home/yusheng/catkin_ws/devel/lib/python2.7/dist-packages /opt/ros/kinetic/lib/python2.7/dist-packages /usr/lib/python2.7 /usr/lib/python2.7/plat-x86_64-linux-gnu /usr/lib/python2.7/lib-tk /usr/lib/python2.7/lib-old /usr/lib/python2.7/lib-dynload /usr/local/lib/python2.7/dist-packages /home/yusheng/code/Visual-new/model/roi_layers /usr/local/lib/python2.7/dist-packages/pycocotools-2.0-py2.7-linux-x86_64.egg /usr/local/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages/PILcompat /usr/lib/python2.7/dist-packages/gtk-2.0 /usr/lib/python2.7/dist-packages/wx-3.0-gtk2 ../vmrn_old ../vmrn_old/model/utils/config.py:407: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. yaml_cfg = edict(yaml.load(f)) Loading pretrained weights from /home/yusheng/catkin_ws/src/INVIGORATE/src/mrt_detector/vmrn_old/model/../data/pretrained_model/resnet101_caffe.pth ../vmrn_old/model/AllinOne.py:856: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaimingnormal. init.kaiming_normal(param, nonlinearity='relu') load checkpoint ../vmrn_old/output/res101/vmrdcompv1/all_in_one_1_25_1407_0.pth load model successfully! Ready to detect manipulation relationships.

I think there is no bugs

python==2.7 CUDA=9.0 GPU gtx1080Ti

It seems fine. But we cannot reproduce your output here. It is suggested to firstly locate where the bug appears. You can separately run the modules, e.g., the detectron2 server, the mattnet server, and the mrt server to see whether they perform in an expected way.

I mean that you may check the input and output to see whether the data format follows the defined manner in the .srv files.

[qingchenkanlu]

yusheng@yusheng:~/catkin_ws/src/INVIGORATE/src/scripts$ python dataset_demo.py --scene_num 27 /home/yusheng/catkin_ws/src/INVIGORATE/src/config/../logs/08-26-2021-11-36-11 /home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/../invigorate/invigorate.py:55: UserWarning: No NLP models are loaded. warnings.warn("No NLP models are loaded.") 2021-08-26 11:36:11,992 - invigorate_logger - INFO - waiting for services... 2021-08-26 11:36:12,018 - invigorate_logger - INFO - img_path = /home/yusheng/catkin_ws/src/INVIGORATE/src/config/../dataset/27/origin.png 2021-08-26 11:36:12,018 - invigorate_logger - INFO - expr = remote 2021-08-26 11:36:12,018 - invigorate_logger - INFO - ---------------------------------------------------------------------------------------- 2021-08-26 11:36:12,018 - invigorate_logger - INFO - Start of iteration Traceback (most recent call last): File "dataset_demo.py", line 237, in main(args) File "dataset_demo.py", line 108, in main res = invigorate_client.estimate_state_with_img(img, expr) File "/home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/../invigorate/invigorate.py", line 119, in estimate_state_with_img res = self.multistep_object_detection(img) File "/home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/../invigorate/invigorate.py", line 141, in multistep_object_detection bboxes, classes, scores = self._object_detection(img) File "/home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/../invigorate/invigorate.py", line 789, in _object_detection num_box, bboxes, classes, class_scores = self._obj_det_client.detect_objects(img, rois) File "/home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/../libraries/ros_clients/detectron2_client.py", line 15, in detect_objects res = self._obj_det(img_msg, False, rois) File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 435, in call return self.call(*args, **kwds) File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 515, in call responses = transport.receive_once() File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_base.py", line 728, in receive_once p.read_messages(b, msg_queue, sock) File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 353, in read_messages self._read_ok_byte(b, sock) File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 336, in _read_ok_byte raise ServiceException("service [%s] responded with an error: %s"%(self.resolved_name, str)) rospy.service.ServiceException: service [/object_detection_srv] responded with an error: error processing request: call() takes 2 positional arguments but 3 were given

When I run the code the first time, there is a fault, is the version of detectron2 is wrong?

[qingchenkanlu]

python==2.7 CUDA=9.0 GPU gtx1080Ti

It seems fine. But we cannot reproduce your output here. It is suggested to firstly locate where the bug appears.

You can separately run the modules, e.g., the detectron2 server, the mattnet server, and the mrt server to see whether they perform in an expected way.

OK, I'll have a try

[lyfkyle]

We use Ubuntu18.04 and ROS melodic. We have not tested on ROS kinetic and we do not officially support it. Based on the error print, I suspect the error happens during ROS communications. Did you run source devel/setup.bash in every terminal?

[ZhangHanbo]

yusheng@yusheng:~/catkin_ws/src/INVIGORATE/src/scripts$ python dataset_demo.py --scene_num 27 /home/yusheng/catkin_ws/src/INVIGORATE/src/config/../logs/08-26-2021-11-36-11 /home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/../invigorate/invigorate.py:55: UserWarning: No NLP models are loaded. warnings.warn("No NLP models are loaded.") 2021-08-26 11:36:11,992 - invigorate_logger - INFO - waiting for services... 2021-08-26 11:36:12,018 - invigorate_logger - INFO - img_path = /home/yusheng/catkin_ws/src/INVIGORATE/src/config/../dataset/27/origin.png 2021-08-26 11:36:12,018 - invigorate_logger - INFO - expr = remote 2021-08-26 11:36:12,018 - invigorate_logger - INFO - ---------------------------------------------------------------------------------------- 2021-08-26 11:36:12,018 - invigorate_logger - INFO - Start of iteration Traceback (most recent call last): File "dataset_demo.py", line 237, in main(args) File "dataset_demo.py", line 108, in main res = invigorate_client.estimate_state_with_img(img, expr) File "/home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/../invigorate/invigorate.py", line 119, in estimate_state_with_img res = self.multistep_object_detection(img) File "/home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/../invigorate/invigorate.py", line 141, in multistep_object_detection bboxes, classes, scores = self._object_detection(img) File "/home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/../invigorate/invigorate.py", line 789, in _object_detection num_box, bboxes, classes, class_scores = self._obj_det_client.detect_objects(img, rois) File "/home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/../libraries/ros_clients/detectron2_client.py", line 15, in detect_objects res = self._obj_det(img_msg, False, rois) File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 435, in call return self.call(*args, kwds) File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 515, in call responses = transport.receive_once() File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_base.py", line 728, in receive_once p.read_messages(b, msg_queue, sock) File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 353, in read_messages self._read_ok_byte(b, sock) File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 336, in _read_ok_byte raise ServiceException("service [%s] responded with an error: %s"%(self.resolved_name, str)) rospy.service.ServiceException: service [/object_detection_srv] responded with an error: error processing request: call**() takes 2 positional arguments but 3 were given

When I run the code the first time, there is a fault, is the version of detectron2 is wrong?

Yes, this should be where the problem is.

We've modified the detectron2 indeed and maybe it is not included in this repo. We will update it soom. Thanks for your feed back.

[qingchenkanlu]

We use Ubuntu18.04 and ROS melodic. We have not tested on ROS kinetic and we do not officially support it. Based on the error print, I suspect the error happens during ROS communications. Did you run source devel/setup.bash in every terminal?

I put the source devel/setup.bash in the .bashrc, don't run it every time

[qingchenkanlu]

@lyf44 which version of detectron2 are you use? and the driver version, pytorch version

[ZhangHanbo]

@lyf44 which version of detectron2 are you use? and the driver version, pytorch version

Not an official version. We will update it soon.

Your driver version and Pytorch version might be fine.

[qingchenkanlu]

@ZhangHanbo OK thanks for your reply

[lyfkyle]

@qingchenkanlu We have updated the detectron2 submodule and also updated the Readme. Please pull the latest version and re-perform the installation steps.

Note that both VMRN and MAttNet uses cuda9.0 and pytorch 0.4. If you find it too troublesome to setup this environment by yourself, you can use our provided docker as mentioned in the Readme.

Please let me know if your problem persists.

[qingchenkanlu]

@lyf44 OK, thanks a lot, I'll try again

[qingchenkanlu]

The previous problem has been solved, there is another problem

yusheng@yusheng:~/catkin_ws/src/INVIGORATE/src/scripts$ python dataset_demo.py --scene_num 27 /home/yusheng/catkin_ws/src/INVIGORATE/src/config/../logs/08-27-2021-20-43-30 Traceback (most recent call last): File "dataset_demo.py", line 22, in from invigorate.invigorate import Invigorate File "/home/yusheng/catkin_ws/src/INVIGORATE/src/scripts/../invigorate/invigorate.py", line 30, in from torchvision.ops import nms ImportError: No module named ops

I use python=2.7 torch=0.4 torchvision=0.2, however, the from torchvision.ops import nms only avaliable at torchvision>=0.3, some thing has been left out?

[lyfkyle]

INVIGORATE requires a newer version of torch and torchvision. As mentioned in the Readme, you should create a separate conda environment for INVIGORATE in which you can install the newest available torch and torchvision.

[qingchenkanlu]

OK, thanks

[qingchenkanlu]

I have changed the code, and the network can run dataset_demo.py with python3, without conda. Thanks your help, it is a very nice work! @lyf44 @ZhangHanbo