search

DavidPL1 / assembly_example

Example code for interaction with our assembly simulation ICRA 2023 challenge
7 stars 1 forks source link

Some issues on the rules #20

Open Director-of-G opened 1 year ago

Director-of-G commented 1 year ago

Hi, here are some issues on the competition rules:

  1. Will you open the portal for submitting and testing client images before the deadline, and will you open a leaderboard?
  2. In the updated v2.2-pre server image, there is a parameter called /loaded_geometry which can tell us the plug's shape in the plugging task. Will this parameter be available in final competition? Besides, what's new with the updated v2.2-pre server image?
  3. In the older v2.2-pre server image, plugs might be loaded on the ground, placed flat or upside down on the table, have you fixed this bug? According to the rules, we speculate the plugs should be placed upright on the table.
  4. Will there be an interface to get the status and scores in plugging task, just like calling /get_assembly_status in screwing task?
balandbal commented 1 year ago
  1. In the updated v2.2-pre server image, there is a parameter called /loaded_geometry which can tell us the plug's shape in the plugging task. Will this parameter be available in final competition? Besides, what's new with the updated v2.2-pre server image?

Currently, the parameter will only be populated in training mode.

  1. In the older v2.2-pre server image, plugs might be loaded on the ground, placed flat or upside down on the table, have you fixed this bug? According to the rules, we speculate the plugs should be placed upright on the table.

Yes, and yes.

  1. Will there be an interface to get the status and scores in plugging task, just like calling /get_assembly_status in screwing task?

Yes, a similar interface will be available.

DavidPL1 commented 1 year ago
  • Will you open the portal for submitting and testing client images before the deadline, and will you open a leaderboard?

A leaderboard is planned, but to my knowledge not before the deadline. @balandbal can you ask the website team about the schedule for submissions (including the leaderboard plans)?

  • In the updated v2.2-pre server image, there is a parameter called /loaded_geometry which can tell us the plug's shape in the plugging task. Will this parameter be available in final competition? Besides, what's new with the updated v2.2-pre server image?

As of today the image includes /get_assembly_status and /loaded_geometry for plugging and a velocity control issue of the panda hardware interface was fixed. We are about to fully release the version, a more extensive list of changes will then be available in the corresponding announcement.

Director-of-G commented 1 year ago

Thanks for your reply. A few more questions:

  1. Can you provide an interface to save the running scene and to reload the saved scene, just like the service reload_with_new_scene. We find such an interface very helpful for debugging.
  2. Just make sure the server image in final competition is exactly the same as the newest release with eval_mode:=True, i.e., there are no possible changes in the Mujoco scene and ROS interfaces.
DavidPL1 commented 1 year ago 
1. Can you provide an interface to save the running scene and to reload the saved scene, just like the service `reload_with_new_scene`. We find such an interface very helpful for debugging.

If you can control the randomness by setting the seed launchfile argument of the assembly_manager launchfile (started by the server). The same seed always generates the same scene. 

2. Just make sure the server image in final competition is exactly the same as the newest release with `eval_mode:=True`, i.e., there are no possible changes in the Mujoco scene and ROS interfaces.

Yes, we will evaluate the submissions by running the eval mode of the most recent server image (also using the seed argument to ensure all submissions run on the same set of scenes).

Director-of-G commented 1 year ago

Sorry to trouble, but we want to make sure the deadline of submission. It is said here that 7th May is the deadline of submission. Do you mean 23:59, 7th May, Pacific time? We failed to find ways to contact with the website team, so we turn to you for help. Thanks a lot!

balandbal commented 1 year ago

The deadline is 23:59 PST (=PDT, as Daylight Saving Time is observed) on May 7, 2023.

balandbal commented 1 year ago

Will you open the portal for submitting and testing client images before the deadline, and will you open a leaderboard?

A leaderboard is planned, but to my knowledge not before the deadline. @balandbal can you ask the website team about the schedule for submissions (including the leaderboard plans)?

To our current plan, the leaderboard will be publicly shown at the conference for the first time. The results will also be shared via our website or e-mail so those who can not attend get the update.

A platform (a file drop) for submission will be available from May 2, 2023. If we receive submissions before the deadline, we can ensure that the submitted docker images run on our evaluation system and get back to the contestants if there is a prohibitive error. We can do this with a cycle time of 24 hours, so we look at a submission sent on May 5 by May 6 to see if it runs. Please, understand that we cannot do thorough debugging on our end.

balandbal commented 1 year ago

The deadline is 23:59 CEST (UTC+2) on May 7, 2023. That would be 14:59 PDT on May 7, 2023.

Per our e-mail yesterday, we moved the submission deadline to 23:59 PST (i.e., PDT) on May 7, 2023.