Where is `grip_ready` pose defined?

[aPonza]

After fixing another minor issue in the panda_mvp_grasp.py file regarding logging, I'm facing this error:

$ rosrun mvp_grasping panda_mvp_grasp.py 
[ INFO] - [/moveit_python_wrappers_1579259656982902166::core::RobotModel::buildModel::93]: Loading robot model 'panda'...
[ INFO] - [/moveit_python_wrappers_1579259656982902166::core::RobotModel::buildModel::93]: Loading robot model 'panda'...
[ INFO] - [/moveit_python_wrappers_1579259656982902166::planning_interface::MoveGroupInterface::MoveGroupInterfaceImpl::MoveGroupInterfaceImpl::178]: Ready to take commands for planning group panda_arm_hand.
No weight. Success? [1/0]0
Name of this experiment? asdasd
Press Enter to Start.
[ERROR] - [/moveit_python_wrappers_1579259656982902166::planning_interface::MoveGroupInterface::setNamedTarget::1690]: The requested named target 'grip_ready' does not exist
Traceback (most recent call last):
  File "/home/ap/mvp_ws/src/mvp_grasp/mvp_grasping/ros_nodes/panda_mvp_grasp.py", line 62, in <module>
    agc.go()
  File "/home/ap/mvp_ws/src/mvp_grasp/mvp_grasping/src/mvp_grasping/panda_base_grasping_controller.py", line 307, in go
    self.pc.goto_named_pose('grip_ready', velocity=0.25)
  File "/home/ap/mvp_ws/src/mvp_grasp/franka_control_wrappers/src/franka_control_wrappers/panda_commander.py", line 146, in goto_named_pose
    self.active_group.set_named_target(pose_name)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/moveit_commander/move_group.py", line 326, in set_named_target
    raise MoveItCommanderException("Unable to set target %s. Is the target within bounds?" % name)
moveit_commander.exception.MoveItCommanderException: Unable to set target grip_ready. Is the target within bounds?

My best guess is you having this pose defined somewhere else on your system, possibly via the MoveIt Setup Assistant. The same is probably true of the drop_box pose.

Maybe we could set them inside PandaCommander instead of there, so they're available here as well for configuration?

[dougsm]

Hey @aPonza I've just had a look but unfortunately I can't find the original joint values for those two named poses. As you say, I guess they were probably both part of the moveit config in franka_ros, so didn't get saved. grip_ready was simply a pose roughly above the picking bin, and drop_box was above the destination bin. Since these are both setup specific, there's no harm in just choosing two poses that work for you.

[seivazi]

Hey @aPonza It would be great If you could push your changes any time soon :)

[aPonza]

I'm temporarily working on another thing for a couple weeks, I was editing franka_control_wrappers/src/franka_control_wrappers/panda_commander.py like this:

diff --git a/franka_control_wrappers/src/franka_control_wrappers/panda_commander.py b/franka_control_wrappers/src/franka_control_wrappers/panda_commander.py
index 6abe106..3e606ee 100644
--- a/franka_control_wrappers/src/franka_control_wrappers/panda_commander.py
+++ b/franka_control_wrappers/src/franka_control_wrappers/panda_commander.py
@@ -22,6 +22,17 @@ class PandaCommander(object):
         self.set_group(group_name)

         self.reset_publisher = rospy.Publisher('/franka_control/error_recovery/goal', ErrorRecoveryActionGoal, queue_size=1)
+        
+        set_named_poses()
+        
+    def set_named_poses(self):
+        """
+        Set poses used in the application via their name
+        """
+        #   translation then rotation                                            x y z x y z w
+        self.active_group.remember_joint_values(self, 'grip_ready', list_to_pose())
+        self.active_group.remember_joint_values(self, 'drop_box', list_to_pose())
+        

     def print_debug_info(self):
         if self.active_group:

Eevntually I'd like to make it configurable via a yaml, but as a temp fix this should work if you input the two poses.

[seivazi]

@aPonza thank you for sharing this. I will continue with it for at least one more week. Are you running whole system in one PC? If yes did you mange to get real time kernel working with NVIDIA driver? @dougsm suggested to use two PCs: One for robot control and other for GG-CNN, camera, and scale. The problem is that I don't know how to do it, Do I need to build two catkin works pace? Do you have any suggestion?

[dougsm]

@seivazi ROS has all of the functionality built in to run on separate networked machines. It's as simple as building the nodes on the machine on which they will be run, and then setting the IP address of the master machine via the ROS_MASTER_URI environment variable. See: http://wiki.ros.org/ROS/Tutorials/MultipleMachines

@aPonza thanks, that's a great way of doing it :-)

[seivazi]

Hi @dougsm and @aPonza do you remember how did you managed to get Franka working with ROS. I follow their instructions both installing via apt and build from source. I see @aPonza had encounter many errors while installing franka ros. Here is not place to discuss Franka issues but I thought maybe you have some hint for me!

Edit: I get the franka working by updating the firmware of the robot and also try an older version of panda move it config file.

[dougsm]

Hi @seivazi Sorry for the slow reply and glad you got it working. As far as I know, there should be no reason to use the older versions of the code.

We (@roboticvisionorg) have a custom installer and set of control wrappers for the panda which makes life a lot easier. You can find them here:

• https://roboticvisionorg.github.io/ (Add our repositories)
• https://roboticvisionorg.github.io/panda_setup/ (ROS/Panda/Real-time kernel setup)
• https://roboticvisionorg.github.io/rv_manipulation_driver/ (Docs for our custom controllers)

You can also ask any questions in our repo https://github.com/RoboticVisionOrg/rv_manipulation_driver

[seivazi]

Thank you for the links.

[seivazi]

Hi @aPonza,

I tried your solution for "Unable to set target grip_ready" error. I gives me some error related to number of argument for "self.active_group.remember_joint_values(self, 'grip_ready', list_to_pose()) Do we need "self" there? I just wanna make sure maybe you did some other changes too.

@dougsm do you have other suggestion were to hard code "grip_ready" and "dopbox"?

[aPonza]

Hey, just got a moment to continue here, you're right in that self is needed, and in fact this is what I currently have

diff --git a/franka_control_wrappers/src/franka_control_wrappers/panda_commander.py b/franka_control_wrappers/src/franka_control_wrappers/panda_commander.py
index 6abe106..7bd74ab 100644
--- a/franka_control_wrappers/src/franka_control_wrappers/panda_commander.py
+++ b/franka_control_wrappers/src/franka_control_wrappers/panda_commander.py
@@ -22,6 +22,20 @@ class PandaCommander(object):
         self.set_group(group_name)

         self.reset_publisher = rospy.Publisher('/franka_control/error_recovery/goal', ErrorRecoveryActionGoal, queue_size=1)
+        
+        self.set_named_poses()
+        
+    def set_named_poses(self):
+        """
+        Set poses used in the application via their name
+        """
+        this_pose = self.active_group.get_current_pose();
+        print(this_pose)
+        
+        #   translation then rotation                                 x                    y                    z                     x                    y                   z                    w
+        self.active_group.remember_joint_values(self, 'grip_ready', this_pose) #[-0.8680978268436771, -0.6478007817770287, -0.35404076143321106, -2.1504153873157840, -0.2511609577867720, 1.5078568387561373, -0.34946772768514023])
+        self.active_group.remember_joint_values(self, 'drop_box', this_pose)  #[ 0.8263036876751351,  1.0896425800490797,  0.82033907273777810, -0.8825330551727532, -0.7649754029929701, 1.7213938517864830,  2.28231075602280060])
+        

     def print_debug_info(self):
         if self.active_group:

I'm also facing the number of arguments error, so I tried to add this get_current_pose to check what should work, but here the error is:

Traceback (most recent call last):
  File "/home/ap/mvp_ws/src/mvp_grasp/mvp_grasping/ros_nodes/panda_mvp_grasp.py", line 61, in <module>
    agc = MVPGraspController()
  File "/home/ap/mvp_ws/src/mvp_grasp/mvp_grasping/ros_nodes/panda_mvp_grasp.py", line 17, in __init__
    super(MVPGraspController, self).__init__()
  File "/home/ap/mvp_ws/src/mvp_grasp/mvp_grasping/src/mvp_grasping/panda_base_grasping_controller.py", line 174, in __init__
    self.pc = PandaCommander(group_name='panda_arm_hand')
  File "/home/ap/mvp_ws/src/mvp_grasp/franka_control_wrappers/src/franka_control_wrappers/panda_commander.py", line 26, in __init__
    self.set_named_poses()
  File "/home/ap/mvp_ws/src/mvp_grasp/franka_control_wrappers/src/franka_control_wrappers/panda_commander.py", line 32, in set_named_poses
    this_pose = self.active_group.get_current_pose();
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/moveit_commander/move_group.py", line 114, in get_current_pose
    raise MoveItCommanderException("There is no end effector to get the pose of")
moveit_commander.exception.MoveItCommanderException: There is no end effector to get the pose of

Let me see what I can find.

[seivazi]

I noticed few things: to pass number of argument error we need to use self.active_group.remember_joint_values( 'grip_ready', pose). So there is no need to use self.

this_pose = self.active_group.get_current_pose(); doesn't work because of panda I believe. Because eef_link = self.active_group.get_end_effector_link() doesn't return anything. I could pass it by writing something like this this_pose = self.active_group.get_current_pose(end_effector_link = "panda_link8").pose

But now I'm confused because get_current_pose return a pose but in self.active_group.remember_joint_values we talk about join instead of pose.

So I thought to use something like this: this_pose = self.active_group.get_current_joint_values() self.active_group.remember_joint_values("grip_ready", this_pose) #) self.active_group.remember_joint_values("drop_box", this_pose)

It throw following error: catkin_ws/src/mvp_grasp/franka_control_wrappers/src/franka_control_wrappers/panda_commander.py", line 165, in goto_named_pose self.active_group.set_named_target(pose_name) File "/opt/ros/melodic/lib/python2.7/dist-packages/moveit_commander/move_group.py", line 326, in set_named_target raise MoveItCommanderException("Unable to set target %s. Is the target within bounds?" % name) moveit_commander.exception.MoveItCommanderException: Unable to set target grip_ready. Is the target within bounds?

Any Idea @aPonza @dougsm

[aPonza]

I can see where my error was: I wasn't including the two joint values for the fingers. Anyways it's weird, because it goes to "ready" but not to "grip_ready"... and then again, after going to "ready" I get a similar error to the previous one about the end effector, so maybe that part still has to be fixed before anything works. @dougsm did you have other edits on panda_moveit_config or have you ever faced these end effector issues?

[dougsm]

The only changes that I would have made to the panda_moveit_config would be to add the named poses to the srdf, and nothing else. Regarding the other issues, I have to admit that I'm unsure. However, do keep in mind that the franka_ros and panda_moveit_config code have undergone fairly major changes in the last ~12 months. If you're having problems with the moveit interface, it might be better to ask over on their repo. Also, see my above post with the ACRV panda control wrappers which may simplify things for you by abstracting some of the franka libraries.

[aPonza]

I checked panda_moveit_config and I only see trajopt changes and a couple extra default poses, but nothing that would bother a missing end effector. Similarly, for franka_ros, I can see tens of commits regarding dual arm usage, the services being moved, but nothing which should affect the end effector. Are you referring to something in particular? I tried rebuilding with an updated franka_ros (using both libfranka 0.7.0 and master) and panda_moveit_config (master) but nothing seemed to change.

EDIT: also, drivers are for melodic and I'm on kinetic.

[seivazi]

@aPonza I fixed the end effector error. The error has something to do with group_name which by defult is None in the panda_commander.py. Here what I did:

• def init(self, group_name='panda_arm_hand'): self.robot = moveit_commander.RobotCommander() self.scene = moveit_commander.PlanningSceneInterface() self.groups = {} self.active_group = None
• self.set_group("panda_arm") I don't understand why only this combination works :) Now I get: raise NotImplementedError() ---> velo_ok = self.__velo_control_loop() error

[aPonza]

Mmm you might actually want to change this line https://github.com/dougsm/mvp_grasp/blob/48330d45a135a5cf1bb058909ed1323be200069f/mvp_grasping/src/mvp_grasping/panda_base_grasping_controller.py#L170 to say 'panda_arm', instead of the other two edits you made.

It does do more than before, but since method go is not implemented in MVPGraspController it goes to call BaseGraspController's go which uses its own __velo_control_loop() instead of MVPGraspController's. Some methods need to be overridden, you can temporarily copy MVPGraspController's body in BaseGraspController's method.

This leads to tensorflow issues, which is what I suspected would happen installing tensorflow instead of tensorflow-gpu. Not sure this one is solvable:

[ERROR] - [/ap_grasping_velo::_invoke_callback::753]: bad callback: <bound method MVPGraspController.__update_callback of <__main__.MVPGraspController object at 0x7f5b41563550>>
Traceback (most recent call last):
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/topics.py", line 750, in _invoke_callback
    cb(msg)
  File "/home/ap/mvp_ws/src/mvp_grasp/mvp_grasping/src/mvp_grasping/panda_base_grasping_controller.py", line 226, in __update_callback
    res = self.entropy_srv()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 435, in __call__
    return self.call(*args, **kwds)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 515, in call
    responses = transport.receive_once()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_base.py", line 727, in receive_once
    p.read_messages(b, msg_queue, sock)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 353, in read_messages
    self._read_ok_byte(b, sock)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 336, in _read_ok_byte
    raise ServiceException("service [%s] responded with an error: %s"%(self.resolved_name, str))
ServiceException: service [/grasp_entropy_node/update_grid] responded with an error: error processing request: Error while reading resource variable conv2d_transpose_3/bias from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/conv2d_transpose_3/bias/N10tensorflow3VarE does not exist.
     [[{{node conv2d_transpose_3/BiasAdd/ReadVariableOp}}]]

[...similar errors follow...]

EDIT: At the same time, the grasp_entropy_service node returns errors like:

[ERROR] - [/grasp_entropy_node::error_handler::574]: Error processing request: Error while reading resource variable conv2d_transpose_2/bias from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/conv2d_transpose_2/bias/N10tensorflow3VarE does not exist.
     [[{{node conv2d_transpose_2/BiasAdd/ReadVariableOp}}]]
['Traceback (most recent call last):\n', '  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 625, in _handle_request\n    response = convert_return_to_response(self.handler(request), self.response_class)\n', '  File "/home/ap/mvp_ws/src/mvp_grasp/mvp_grasping/ros_nodes/grasp_entropy_node.py", line 128, in update_service_handler\n    points, angle, width_img, _ = predict(depth_crop, process_depth=False, depth_nan_mask=depth_nan_mask)\n', '  File "/home/ap/mvp_ws/src/mvp_grasp/ggcnn/src/ggcnn/ggcnn.py", line 67, in predict\n    pred_out = model.predict(depth.reshape((1, 300, 300, 1)))\n', '  File "/home/ap/.local/lib/python2.7/site-packages/tensorflow/python/keras/engine/training.py", line 1078, in predict\n    callbacks=callbacks)\n', '  File "/home/ap/.local/lib/python2.7/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 363, in model_iteration\n    batch_outs = f(ins_batch)\n', '  File "/home/ap/.local/lib/python2.7/site-packages/tensorflow/python/keras/backend.py", line 3292, in __call__\n    run_metadata=self.run_metadata)\n', '  File "/home/ap/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1458, in __call__\n    run_metadata_ptr)\n', 'FailedPreconditionError: Error while reading resource variable conv2d_transpose_2/bias from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/conv2d_transpose_2/bias/N10tensorflow3VarE does not exist.\n\t [[{{node conv2d_transpose_2/BiasAdd/ReadVariableOp}}]]\n']
[ERROR] - [/grasp_entropy_node::error_handler::574]: Error processing request: Error while reading resource variable width_out/kernel from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/width_out/kernel/N10tensorflow3VarE does not exist.

[...similar errors follow...]

Reading the error more thoroughly, it seems more of a CNN initialization issue, not necessarily a tensorflow one.

[seivazi]

I'm in the same stage with following error:

the grasp_entropy_service node returns errors like:

[ERROR] [1581593428.562427]: Error processing request: 2 root error(s) found. (0) Failed precondition: Error while reading resource variable cos_out/bias from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/cos_out/bias/N10tensorflow3VarE does not exist. [[{{node cos_out/BiasAdd/ReadVariableOp}}]] [[sin_out/BiasAdd/_39]] (1) Failed precondition: Error while reading resource variable cos_out/bias from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/cos_out/bias/N10tensorflow3VarE does not exist. [[{{node cos_out/BiasAdd/ReadVariableOp}}]] 0 successful operations. 0 derived errors ignored. ['Traceback (most recent call last):\n', ' File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 629, in _handle_request\n response = convert_return_to_response(self.handler(request), self.response_class)\n', ' File "/home/seivazi/catkin_ws/src/mvp_grasp/mvp_grasping/ros_nodes/grasp_entropy_node.py", line 128, in update_service_handler\n points, angle, widthimg, = predict(depth_crop, process_depth=False, depth_nan_mask=depth_nan_mask)\n', ' File "/home/seivazi/catkin_ws/src/mvp_grasp/ggcnn/src/ggcnn/ggcnn.py", line 67, in predict\n pred_out = model.predict(depth.reshape((1, 300, 300, 1)))\n', ' File "/home/seivazi/.local/lib/python2.7/site-packages/tensorflow/python/keras/engine/training.py", line 1078, in predict\n callbacks=callbacks)\n', ' File "/home/seivazi/.local/lib/python2.7/site-packages/tensorflow/python/keras/engine/training_arrays.py", line 363, in model_iteration\n batch_outs = f(ins_batch)\n', ' File "/home/seivazi/.local/lib/python2.7/site-packages/tensorflow/python/keras/backend.py", line 3292, in call\n run_metadata=self.run_metadata)\n', ' File "/home/seivazi/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1458, in call\n run_metadata_ptr)\n', 'FailedPreconditionError: 2 root error(s) found.\n (0) Failed precondition: Error while reading resource variable cos_out/bias from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/cos_out/bias/N10tensorflow3VarE does not exist.\n\t [[{{node cos_out/BiasAdd/ReadVariableOp}}]]\n\t [[sin_out/BiasAdd/_39]]\n (1) Failed precondition: Error while reading resource variable cos_out/bias from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/cos_out/bias/N10tensorflow3VarE does not exist.\n\t [[{{node cos_out/BiasAdd/ReadVariableOp}}]]\n0 successful operations.\n0 derived errors ignored.\n']

@aPonza you are right initially I thought it is tensorflow issue and try to follow https://github.com/tensorflow/tensorflow/issues/28287 but that is not the problem. It seems there is a missing variable.

@dougsm any suggestion?

[aPonza]

$ git diff ggcnn/src/ggcnn/ggcnn.py
diff --git a/ggcnn/src/ggcnn/ggcnn.py b/ggcnn/src/ggcnn/ggcnn.py
index 396e627..8a49dc5 100644
--- a/ggcnn/src/ggcnn/ggcnn.py
+++ b/ggcnn/src/ggcnn/ggcnn.py
@@ -8,10 +8,17 @@ import tensorflow as tf
 from tensorflow.keras.models import load_model

 from dougsm_helpers.timeit import TimeIt
+from tensorflow.python.keras.backend import set_session

+session = tf.keras.backend.get_session()
+init = tf.global_variables_initializer()
+session.run(init)
+
+graph = tf.get_default_graph()
+
+set_session(session)
 MODEL_FILE = 'models/epoch_29_model.hdf5'
 model = load_model(path.join(path.dirname(__file__), MODEL_FILE))
-graph = tf.get_default_graph()

 TimeIt.print_output = False  # For debugging/timing

@@ -58,12 +65,16 @@ def process_depth_image(depth, crop_size, out_size=300, return_mask=False, crop_

 def predict(depth, process_depth=True, crop_size=300, out_size=300, depth_nan_mask=None, filters=(2.0, 1.0, 1.0)):
+    global sess
+    global graph
+    
     if process_depth:
         depth, depth_nan_mask = process_depth_image(depth, crop_size, out_size, True)

     # Inference
     depth = np.clip((depth - depth.mean()), -1, 1)
     with graph.as_default():
+        set_session(session)
         pred_out = model.predict(depth.reshape((1, 300, 300, 1)))

     points_out = pred_out[0].squeeze()

as per this, which then leads to this:

[ERROR] - [/ap_grasping_velo::_invoke_callback::753]: bad callback: <bound method MVPGraspController.__update_callback of <__main__.MVPGraspController object at 0x7fd5dfc2fb10>>
Traceback (most recent call last):
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/topics.py", line 750, in _invoke_callback
    cb(msg)
  File "/home/ap/mvp_ws/src/mvp_grasp/mvp_grasping/src/mvp_grasping/panda_base_grasping_controller.py", line 226, in __update_callback
    res = self.entropy_srv()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 435, in __call__
    return self.call(*args, **kwds)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 515, in call
    responses = transport.receive_once()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_base.py", line 727, in receive_once
    p.read_messages(b, msg_queue, sock)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 353, in read_messages
    self._read_ok_byte(b, sock)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 336, in _read_ok_byte
    raise ServiceException("service [%s] responded with an error: %s"%(self.resolved_name, str))
ServiceException: service [/grasp_entropy_node/update_grid] responded with an error: error processing request: Weights sum to zero, can't be normalized

Please note I'm not used to tensorflow, the code can be improved.

UPDATE: gdb gave me more insight, this is the cause of the update_grid error (my linebreaks for clarity):

[ERROR] - [/grasp_entropy_node::error_handler::574]: Error processing request: Weights sum to zero, can't be normalized
['
Traceback (most recent call last):
', '  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 625, in _handle_request
    response = convert_return_to_response(self.handler(request), self.response_class)
', '  File "/home/ap/mvp_ws/src/mvp_grasp/mvp_grasping/ros_nodes/grasp_entropy_node.py", line 172, in update_service_handler
    q_am_neigh_avg = np.average(q_am_neigh, weights=neighbour_weights, axis=0)
', '  File "/home/ap/.local/lib/python2.7/site-packages/numpy/lib/function_base.py", line 422, in average
    "Weights sum to zero, can\'t be normalized")
', "ZeroDivisionError: Weights sum to zero, can't be normalized
"]

as well as:

/home/ap/mvp_ws/src/mvp_grasp/mvp_grasping/ros_nodes/grasp_entropy_node.py:231: RuntimeWarning: divide by zero encountered in true_divide
  d_from_prev_view += np.clip(1 - (np.sqrt((self._xv - x)**2 + (self._yv - y)**2 + 0*(cam_p.z - z)**2)/self.dist_from_prev_view_scale), 0, 1) * (1-kl)

[aPonza]

Ok, actually it seems to almost work, I did not have any pickable under the camera, but adding one started moving the robot and the eef went some centimeters off of it. Now it's likely just a matter of publishing the correct transform from the flange to the camera, which is bound to be wrong since we're using panda_arm instead of panda_arm_hand.

I'm guessing the division by zero errors are due to not having anything pickable, they could be handled more gracefully, I'll try tomorrow.

[seivazi]

Looking forward to test your solution tomorrow. I also noticed about offset of transformation. I couldn't figure out where should I changed it.

[seivazi]

Hi @aPonza I tried your changes to session but still get the same error as before Error processing request: 2 root error(s) found....

Could you check once again what was your last changes? I can see for example a problem with global sess (which is unused).

Could you tell me which version of tensorflow and cuda are you using? I use tf 14 and Cuda 10 an wonder whether that is the problem.

[aPonza]

I don't have a gpu on my workbench at the moment (hence

This leads to tensorflow issues, which is what I suspected would happen installing tensorflow instead of tensorflow-gpu

from here). The global sess is a mistake (use global session instead), might be useless since it's probably in the same thread as where the global variable is declared and overall is a copy-paste job at best.

I'm on tensorflow==1.14.0, no CUDA here since the GPU is elsewhere at the moment.

I have other changes on mvp_grasping/ros_nodes/panda_mvp_grasp.py and mvp_grasping/src/mvp_grasping/panda_base_grasping_controller.py but I thought they were mostly cosmetic (it's what I did instead of copying MVPController's body into the base controller). I'll send a PR with everything as soon as I can get the transform right on the final bit before the grasp, setting it up so that whoever uses panda_arm_hand gets it right still.

Can you maybe post the 2 errors you get, see if I had them and didn't realize?

[seivazi]

Thank you for your point. The problem could be tensorflow-gpu. I'm now trying to play with different version of tf. The 2 Error is as below:

[ERROR] [1581673861.842248]: Error processing request: 2 root error(s) found. (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node conv2d_1/Conv2D}}]] (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node conv2d_1/Conv2D}}]] [[sin_out/BiasAdd/_351]] 0 successful operations. 0 derived errors ignored. ['Traceback (most recent call last):\n', ' File "/opt/ros/melodic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 629, in _handle_request\n response = convert_return_to_response(self.handler(request), self.response_class)\n', ' File "/home/seivazi/catkin_ws/src/mvp_grasp/mvp_grasping/ros_nodes/grasp_entropy_node.py", line 128, in update_service_handler\n points, angle, widthimg, = predict(depth_crop, process_depth=False, depth_nan_mask=depth_nan_mask)\n', ' File "/home/seivazi/catkin_ws/src/mvp_grasp/ggcnn/src/ggcnn/ggcnn.py", line 100, in predict\n pred_out = model.predict(depth.reshape((1, 300, 300, 1)))\n', ' File "/home/seivazi/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 908, in predict\n use_multiprocessing=use_multiprocessing)\n', ' File "/home/seivazi/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training_arrays.py", line 723, in predict\n callbacks=callbacks)\n', ' File "/home/seivazi/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training_arrays.py", line 394, in model_iteration\n batch_outs = f(ins_batch)\n', ' File "/home/seivazi/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/backend.py", line 3476, in call\n run_metadata=self.run_metadata)\n', ' File "/home/seivazi/.local/lib/python2.7/site-packages/tensorflow_core/python/client/session.py", line 1472, in call\n run_metadata_ptr)\n', 'UnknownError: 2 root error(s) found.\n (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.\n\t [[{{node conv2d_1/Conv2D}}]]\n (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.\n\t [[{{node conv2d_1/Conv2D}}]]\n\t [[sin_out/BiasAdd/_351]]\n0 successful operations.\n0 derived errors ignored.\n']

[seivazi]

@aPonza with tensorflow-cpu I don't get the error as above anymore. but then I get a similar error as yours with ZeroDivisionError but for me is like below even with object located under camera view.

I noticed where my error comes but don't know how to fixed it: It is from mvp_grasp.yml and histogram: bounds:. I don't know why changing x,y cause dimension miss match:

seivazi/catkin_ws/src/mvp_grasp/mvp_grasping/ros_nodes/grasp_entropy_node.py", line 231, in update_service_handler\n d_from_prev_view += np.clip(1 - (np.sqrt((self._xv - x)2 + (self._yv - y)2 + 0*(cam_p.z - z)*2)/self.dist_from_prev_view_scale), 0, 1) (1-kl)\n', 'ValueError: operands could not be broadcast together with shapes (72,68) (68,72) (72,68) \n']

P.S I fixed the ZeroDivisionError by changing the dist_from_prev_view_scale in mvp_grasp.yml file. But I don't understand what is this distance @dougsm ?

[dougsm]

The distance from previous view was an extra loss for the next best view calculation which favoured viewpoints which were different from the current viewpoint (hence the x,y,z distance calculation), though we didn't end up using this in order to simplify the controller. You can comment out that line if you want and set d_from_prev_view to zero, or as I assume you've done, set the config value to a non-zero number.

[aPonza]

with tensorflow-cpu I don't get the error as above anymore

You might have an incompatible cuDNN, maybe?

[seivazi]

Thank you @aPonza do you have any idea which version of cuDNN should I use? @dougsm

[dougsm]

@seivazi @aPonza I have replied in the other issue which was about tensorflow versions. (I have also pushed some small updates that fix things that you were discussing). Are the moveit related issues here fixed?

Now that I have some time, I am updating the mvp_grasping package to work with our in-house panda controllers (https://roboticvisionorg.github.io/) which should help abstract away a lot of the issues that you have been having, and will definitely then be working with the newest libfranka and moveit configs.

[aPonza]

Cheers! I should probably push a PR with the changes before you start then.

[seivazi]

@aPonza it will be great if you could push your changes any time soon. I have a problem with franka movement commands: at panda_base_grasping_controller.py when I try to use self.pc.goto_named_pose('grip_ready', velocity=0.25) or any other goto pose method from panda commander (to bring the robot in top of the picking and dropping box) franka shakes a lot and stop. Somehow it works for one iteration and then such behaviour appear. I tried different poses but no chance yet.

Here are my steps in panda_base_grasping_controller.py:

1- self.pc.goto_named_pose('grip_ready', velocity=0.25) ---> works fine 2- # Execute the Grasp: grasp_ret = self.__execute_best_grasp() --->works fine 3- Here when I wanna come back to any position for example with self.pc.goto_named_pose('grip_ready', velocity=0.25) --> robot start to shake so much and fail to move any more. I wonder if it has something to do with the fact that after Execute the Grasp we need to clear memory or something :)

[aPonza]

The implemented panda controller is a demo one, they probably have a better controller on their own repository as Doug suggests. I suggest you use your own (or theirs), which is what I'll do as soon as I can finish these changes.

[dougsm]

@aPonza I'm not sure I understand what you mean by the controller being a demo one. We don't have anything particularly special control wise implemented here, just a nicer wrapper around the franka ros api and moveit.

@seivazi Those commands are just direct calls to moveit. There definitely should be nothing causing them to make the robot shake, especially if you're using the most up-to-date official version of moveit for the panda. That sounds like it could be a much more low level problem with the robot.

[seivazi]

I look to the (https://roboticvisionorg.github.io/). It is similar to the panda_commander.py in this git. It is normal Moveit commands so it should work as @dougsm pointed out. Strange enough it always works in first run of the program but then it fails. I try to use rospy.sleep() or .stop but problem still stay the same. @dougsm do you use any specific moveit planner?

@aPonza have run the program for several iteration? is it working for you with moveit commands or do you use something else for moving the robot?

[aPonza]

The controller can be improved, as you say yourself, and I haven't checked your company's implementation, I had assumed it would be an improved version. I apologize, it all came out rougher than I meant to.

I see the same oscillations/shaking Shahram sees, but it's independent of the run, it happens on the first as well. I resolved the issue in the past by enforcing some stricter soft joint limits: the oscillations happen because the robot is nearby the joint limits which triggers the safeties. You can do so in two ways:

• (easy) change the limits in franka_description/robots/panda_arm.xacro;
• (less easy) you can rebuild the JointModelGroup for Panda from scratch at runtime (I still haven't got around to doing it myself).

[seivazi]

Thank you @aPonza! I guess this is issue for me too. Could share with me your panda_arm.xacro file ? or soft limit values for all joins?

[aPonza]

I'm referring to e.g. lines 64, 97 of that file, just using lower numbers for the soft limits. This however doesn't always work and sometimes it really seems to depend on the starting position of the arm and the starting position(s) of the object(s) to pick.

[seivazi]

You are right it really depends on the arm position. A the end I use RV driver for panda. https://roboticvisionorg.github.io/