Errors while creating submissions

[xtli12]

• Error 1

When I followed the Participation Guidelines, like:

# Add your submission to PYTHONPATH. Ensure that "user_solution.py" can be found to import.
export PYTHONPATH=${PATH_TO_YOUR_CODES_IN_HOST}:$PYTHONPATH

# Test whether the user solution can be imported
# python -c "from user_solution import UserPolicy"

# Run evaluation. The result will be saved to ${OUTPUT_DIR}.
ENV_ID="PickCube-v0" OUTPUT_DIR="tmp"
python -m mani_skill2.evaluation.run_evaluation -e ${ENV_ID} -o ${OUTPUT_DIR}

The path of my user_solution.py is /data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/submission So I add my submission to PYTHONPATH and test whether the user solution can be imported like this:

export PYTHONPATH=/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/submission:$PYTHONPATH
python -c "from user_solution import UserPolicy"

The error comes out like this:

Pointnet++ is not compiled
No module named 'maniskill2_learn.networks.modules.pn2_modules'

Although after I installed Pointnet++, this error still exists.

• Error 2

When I tried to run a local evaluation like this:

ENV_ID="PickCube-v0" OUTPUT_DIR="tmp"
python -m mani_skill2.evaluation.run_evaluation -e ${ENV_ID} -o ${OUTPUT_DIR}

The error comes out like this:

Fail to load evaluation configuration. (<class 'TypeError'>, TypeError("'NoneType' object cannot be interpreted as an integer"))

• Error 3

When I built my submission docker image like this:

docker build -f "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/Docker" -t maniskill2023-submission

The error comes out like this:

ERROR: "docker buildx build" requires exactly 1 argument.
See 'docker buildx build --help'.

Usage:  docker buildx build [OPTIONS] PATH | URL | -

Start a build

I've been trying to figure out how to fix those errors for a couple of days, but I've failed. I also can't tell if${YOUR_CODES_AND_WEIGHTS_IN_CONTAINER}is same as ${YOUR_CODES_AND_WEIGHTS_IN_HOST} since I haven't found an explanation for ${YOUR_CODES_AND_WEIGHTS_IN_CONTAINER} in the guideline. It would be fantastic if you could assist me in resolving those errors and provide us with more detailed Participation Guidelines. Your kind assistance is much appreciated!

[xuanlinli17]

Hi xtli12,

I've updated the docker files and submission examples for ManiSkill2-Learn. You can view it at https://github.com/haosulab/ManiSkill2-Learn/tree/main/submission_example .

Pointnet++ is not compiled
No module named 'maniskill2_learn.networks.modules.pn2_modules'

This only indicates that you haven't compiled our custom implementation of pointnet++. Programs can still run successfully without PointNet++.

Fail to load evaluation configuration. (<class 'TypeError'>, TypeError("'NoneType' object cannot be interpreted as an integer"))

If you follow my updated ManiSkill2-Learn submission example, then it shouldn't occur anymore.

ERROR: "docker buildx build" requires exactly 1 argument.

The command should be

docker build -f {path_to_docker_file} {build_context_path} -t maniskill2023-submission

Your command is missing the build_context_path. The build_context_path is the local path where docker instructions, e.g., COPY, are referenced with respect to. See here for more details.

For example,

docker build -f "/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/Docker" . -t maniskill2023-submission

Note the . in the above command.

[xtli12]

Thank you for your immediate reply! I've changed the config.file_path and model_path in the updated user_solution.py as follows:

cfg = Config.fromfile("/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/configs/mfrl/ppo/maniskill2_pn.py")  
model_path = f'/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/model_10000000.ckpt' 

And moved the user_solution.py directly under the ManiSkill2-Learn-main/, but the error still exists when I run:

export PYTHONPATH=/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main:$PYTHONPATH
ENV_ID="PickCube-v0" OUTPUT_DIR="tmp"
python -m mani_skill2.evaluation.run_evaluation -e ${ENV_ID} -o ${OUTPUT_DIR}

the error is

Fail to load evaluation configuration. (<class 'TypeError'>, TypeError("'NoneType' object cannot be interpreted as an integer"))

[xuanlinli17]

python -m mani_skill2.evaluation.run_evaluation searches the user_solution.py under the root ManiSkill2-Learn-main directory (instead of user_solution.py under submission_example). Do you have the correct path for user_solution.py (directly under root)?

Also are you running these commands in docker or outside of docker?

[xtli12]

I was running these commands outside of docker as the guideline https://github.com/haosulab/ManiSkill2/wiki/Participation-Guidelines says. First, I changed the config.file_path and model_path in user_solution.py as follows:

#original
cfg = Config.fromfile("/root/ManiSkill2-Learn/configs/mfrl/ppo/maniskill2_pn.py") # Change this
model_path = f'/root/ManiSkill2-Learn/maniskill2_learn_pretrained_models_videos/{env_id}/dapg_pointcloud/model_25000000.ckpt' # Change this

changed to the absolute path in my machine:

cfg = Config.fromfile("/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/configs/mfrl/ppo/maniskill2_pn.py")  # Change this
model_path = f'/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main/model_10000000.ckpt'  # Change this

Then I moved the user_solution.py to the directory ManiSkill2-Learn-main/ like this:

Then I followed the guideline https://github.com/haosulab/ManiSkill2/wiki/Participation-Guidelines:

export PYTHONPATH=/data/home-gxu/lxt21/SAPIEN-master/ManiSkill2-Learn-main:$PYTHONPATH
ENV_ID="PickCube-v0" OUTPUT_DIR="tmp"
python -m mani_skill2.evaluation.run_evaluation -e ${ENV_ID} -o ${OUTPUT_DIR}

But the error still exists :

Fail to load evaluation configuration. (<class 'TypeError'>, TypeError("'NoneType' object cannot be interpreted as an integer"))

[xuanlinli17]

Oh, I believe you need to refer to the new participation guideline at https://haosulab.github.io/ManiSkill2/benchmark/submission.html . Our github wiki page is obsolete. All of our documentations are at https://haosulab.github.io/ManiSkill2. Sorry about the confusion.

[xtli12]

Thank you for your reply. With your kind help, I was able to locally verify my solution ! But when I submitted the docker image to the leaderboard, my score was 0. The weird thing is that the model I employed is the official .ckpt like:

/ManiSkill2-Learn-main/maniskill2_learn_pretrained_models_videos/PickCube-v0/dapg_pointcloud_vf1/model_25000000.ckpt

and when I tested the docker image locally, the score was not 0, and the average_metrics.json file is:

{"elapsed_steps": 32.0, "is_obj_placed": 1.0, "is_robot_static": 1.0, "success": 1.0}

But the result on leaderboard is: How could I solve it?

[xuanlinli17]

Did you test the success rate locally using your submission docker?

[xtli12]

Yes, I did.

[StoneT2000]

@xtli12 Thanks for raising the issue. I will take a look into this issue, there may be bug in the evaluation system

[StoneT2000]

@xtli12 we have discovered the bug. The evaluation system has been updated now. Moreover, make sure to use absolute paths in your submission

In your code I see you used some relative paths like ./.... instead of /. Note that all code is executed from the /root/ directory, some of your code is in your own configured directory in /. I recommend using the absolute directory when possible.

Another note, make sure to not select all environment tasks to submit your image for (unless that is intended). It will slow down your evaluation, just pick the ones your code is designed for (we will also update the UI to make sure the default is not all tasks).

[xtli12]

Thanks to your kind assistance, I am now able to have a score on the leaderboard!

[StoneT2000]

@xtli12 actually we also fixed another relevant bug on the submission system. Some submissions weren't running on some tasks and was reporting some error about a pod. This has been fixed so your new submissions should evaluate on all tasks selected and not randomly fail.

[xtli12]

@StoneT2000 Hi, thanks for your kind assistance! However, there is still one issue that I can't figure out: When I got a .ckpt file after I trained locally, the success rate of it could be evaluated at nearly 0.9 (as the statistics.csv of eval_videos show):

length,reward,finish
16.00,27.09,1.00
30.00,55.04,1.00
27.00,46.54,1.00
21.00,39.88,1.00
18.00,25.68,1.00
32.00,70.89,1.00
200.00,54.32,0.00
200.00,142.26,0.00
19.00,33.22,1.00
21.00,38.99,1.00
25.00,37.49,1.00
25.00,41.71,1.00
28.00,47.78,1.00
22.00,36.20,1.00
36.00,62.98,1.00
22.00,42.51,1.00
26.00,47.45,1.00
20.00,38.91,1.00
16.00,27.86,1.00
15.00,15.59,1.00
24.00,38.76,1.00
34.00,56.16,1.00
23.00,39.97,1.00
16.00,24.94,1.00
28.00,55.13,1.00
18.00,28.16,1.00
29.00,26.65,1.00
24.00,41.36,1.00
22.00,33.08,1.00
17.00,26.10,1.00
32.00,52.22,1.00
23.00,47.61,1.00
22.00,28.18,1.00
20.00,30.57,1.00
14.00,26.25,1.00
36.00,54.62,1.00
21.00,39.93,1.00
25.00,45.10,1.00
27.00,41.78,1.00
36.00,65.30,1.00
24.00,40.19,1.00
35.00,53.46,1.00
25.00,46.08,1.00
38.00,64.58,1.00
27.00,48.21,1.00
31.00,49.23,1.00
35.00,57.30,1.00
33.00,55.16,1.00
30.00,57.10,1.00
29.00,51.07,1.00
22.00,39.08,1.00
17.00,32.24,1.00
29.00,53.18,1.00
21.00,31.36,1.00
31.00,51.91,1.00
30.00,54.70,1.00
17.00,30.73,1.00
24.00,42.72,1.00
16.00,27.93,1.00
25.00,46.75,1.00
21.00,36.03,1.00
20.00,38.31,1.00
26.00,50.52,1.00
34.00,71.51,1.00
18.00,34.77,1.00
21.00,32.74,1.00
26.00,24.26,1.00
20.00,31.10,1.00
38.00,52.84,1.00
64.00,141.20,1.00
22.00,37.52,1.00
21.00,39.26,1.00
33.00,61.53,1.00
26.00,46.36,1.00
17.00,30.52,1.00
31.00,58.11,1.00
34.00,70.25,1.00
25.00,39.19,1.00
28.00,48.81,1.00
19.00,36.75,1.00
24.00,41.37,1.00
24.00,41.12,1.00
33.00,46.83,1.00
28.00,35.95,1.00
35.00,55.76,1.00
17.00,30.58,1.00
35.00,59.21,1.00
19.00,35.51,1.00
23.00,40.27,1.00
26.00,49.69,1.00
22.00,39.70,1.00
30.00,63.27,1.00
24.00,44.66,1.00
22.00,40.48,1.00
24.00,36.81,1.00
15.00,22.36,1.00
16.00,27.23,1.00
19.00,32.93,1.00
18.00,32.40,1.00
24.00,40.25,1.00
19.00,31.77,1.00
23.00,41.42,1.00
30.00,28.32,1.00
22.00,37.12,1.00
19.00,29.72,1.00
31.00,47.59,1.00
24.00,42.54,1.00
28.00,48.80,1.00
24.00,40.05,1.00
18.00,33.08,1.00
20.00,29.66,1.00
15.00,26.05,1.00
14.00,26.88,1.00
39.00,84.47,1.00
29.00,51.09,1.00
29.00,49.55,1.00
200.00,73.68,0.00
27.00,45.85,1.00
14.00,24.01,1.00
32.00,49.84,1.00
15.00,22.88,1.00
12.00,18.75,1.00
19.00,33.10,1.00
15.00,26.14,1.00
20.00,25.97,1.00
14.00,22.45,1.00
200.00,79.64,0.00
32.00,52.45,1.00
21.00,23.57,1.00
26.00,42.50,1.00
28.00,50.71,1.00
35.00,58.86,1.00
31.00,41.77,1.00
14.00,24.70,1.00
21.00,35.15,1.00
22.00,42.51,1.00
33.00,46.44,1.00
14.00,23.89,1.00
26.00,25.57,1.00
21.00,30.96,1.00
19.00,32.06,1.00
14.00,20.12,1.00
15.00,15.79,1.00
31.00,48.35,1.00
34.00,78.64,1.00
26.00,46.72,1.00
44.00,42.71,1.00
29.00,51.72,1.00
31.00,56.81,1.00
76.00,82.25,1.00
18.00,34.17,1.00
27.00,47.76,1.00
22.00,42.31,1.00
18.00,32.68,1.00
40.00,51.47,1.00
21.00,42.09,1.00
21.00,42.62,1.00
14.00,23.53,1.00
27.00,39.50,1.00
12.00,20.75,1.00
20.00,30.41,1.00
35.00,58.70,1.00
24.00,51.75,1.00
22.00,45.00,1.00
22.00,35.31,1.00
20.00,36.31,1.00
26.00,44.10,1.00
13.00,23.01,1.00
17.00,31.28,1.00
17.00,29.20,1.00
17.00,30.68,1.00
28.00,29.18,1.00
21.00,35.53,1.00
26.00,42.27,1.00
25.00,48.19,1.00
15.00,24.92,1.00
26.00,45.94,1.00
35.00,27.94,1.00
18.00,32.05,1.00
19.00,25.48,1.00
21.00,35.41,1.00
28.00,50.24,1.00
24.00,45.98,1.00
21.00,35.20,1.00
22.00,41.48,1.00
16.00,28.18,1.00
36.00,60.58,1.00
23.00,42.88,1.00
29.00,65.18,1.00
25.00,37.11,1.00
26.00,47.13,1.00
31.00,30.39,1.00
17.00,26.66,1.00
16.00,16.37,1.00
22.00,37.15,1.00
22.00,45.53,1.00
20.00,35.19,1.00
12.00,19.97,1.00
18.00,33.18,1.00
23.00,42.42,1.00
13.00,20.98,1.00
15.00,27.37,1.00
17.00,26.32,1.00
24.00,44.16,1.00
23.00,42.53,1.00
23.00,25.47,1.00
18.00,30.27,1.00
18.00,31.68,1.00
24.00,38.49,1.00
30.00,50.92,1.00
17.00,30.32,1.00
40.00,67.57,1.00
22.00,42.43,1.00
25.00,46.81,1.00
25.00,42.17,1.00
23.00,41.89,1.00
31.00,52.64,1.00
32.00,45.36,1.00
28.00,50.13,1.00
29.00,37.51,1.00
20.00,35.07,1.00
18.00,30.43,1.00
22.00,36.31,1.00
21.00,41.24,1.00
23.00,40.41,1.00
27.00,49.71,1.00
34.00,58.82,1.00
13.00,21.66,1.00
25.00,46.83,1.00
15.00,25.76,1.00
21.00,36.89,1.00
19.00,30.49,1.00
26.00,44.97,1.00
16.00,24.61,1.00
14.00,22.44,1.00
21.00,37.26,1.00
24.00,40.71,1.00
22.00,33.69,1.00
49.00,78.87,1.00
42.00,71.75,1.00
31.00,46.28,1.00
18.00,27.85,1.00
20.00,35.42,1.00
20.00,37.93,1.00
34.00,48.01,1.00
22.00,29.76,1.00
17.00,30.02,1.00
15.00,15.99,1.00
30.00,50.20,1.00
28.00,52.79,1.00
25.00,48.07,1.00
14.00,25.65,1.00
33.00,57.88,1.00
31.00,48.26,1.00
27.00,46.15,1.00
26.00,42.86,1.00
23.00,35.27,1.00
35.00,46.36,1.00
26.00,47.01,1.00
73.00,96.45,1.00
23.00,41.99,1.00
14.00,26.26,1.00
30.00,47.25,1.00
12.00,18.94,1.00
24.00,37.01,1.00
38.00,59.17,1.00
29.00,46.39,1.00
39.00,61.66,1.00
23.00,31.29,1.00
34.00,58.86,1.00
18.00,37.57,1.00
200.00,83.20,0.00
20.00,40.74,1.00
12.00,17.78,1.00
32.00,37.41,1.00
62.00,70.61,1.00
17.00,26.63,1.00
22.00,39.68,1.00
24.00,26.04,1.00
58.00,72.89,1.00
34.00,53.15,1.00
28.00,51.34,1.00
31.00,52.10,1.00
24.00,39.92,1.00
38.00,59.99,1.00
27.00,49.32,1.00
30.00,51.98,1.00
21.00,30.69,1.00
18.00,29.23,1.00
16.00,25.07,1.00
29.00,54.99,1.00
20.00,41.45,1.00
31.00,59.07,1.00
35.00,55.60,1.00
28.00,50.28,1.00
35.00,57.08,1.00
21.00,28.22,1.00
25.00,41.92,1.00
22.00,45.31,1.00
19.00,25.44,1.00
23.00,43.70,1.00
22.00,24.92,1.00
57.00,80.43,1.00
26.00,42.72,1.00
23.00,23.00,1.00
37.00,67.64,1.00
29.00,44.46,1.00
21.00,42.02,1.00
22.00,30.81,1.00
17.00,29.65,1.00
40.00,55.80,1.00
23.00,44.98,1.00
27.00,58.54,1.00
155.00,126.08,1.00
31.00,29.77,1.00
16.00,28.56,1.00
12.00,19.08,1.00
17.00,32.03,1.00
28.00,53.48,1.00
31.00,56.41,1.00
33.00,39.11,1.00
24.00,43.42,1.00
31.00,39.48,1.00
25.00,44.01,1.00
25.00,48.72,1.00
24.00,45.73,1.00
21.00,38.08,1.00
18.00,37.44,1.00
26.00,37.24,1.00
19.00,34.84,1.00
24.00,44.10,1.00
101.00,95.12,1.00
25.00,28.45,1.00
22.00,37.77,1.00
18.00,27.75,1.00
24.00,41.10,1.00
69.00,77.68,1.00
24.00,44.38,1.00
24.00,40.66,1.00
25.00,37.41,1.00
21.00,38.31,1.00
22.00,37.10,1.00
19.00,31.02,1.00
28.00,48.91,1.00
20.00,29.51,1.00
22.00,36.85,1.00
26.00,50.80,1.00
28.00,44.52,1.00
18.00,18.22,1.00
16.00,26.99,1.00
21.00,39.33,1.00
22.00,39.26,1.00
37.00,64.34,1.00
16.00,28.90,1.00
28.00,49.94,1.00
31.00,43.18,1.00
25.00,38.68,1.00
15.00,26.31,1.00
28.00,44.25,1.00
27.00,48.84,1.00
149.00,116.92,1.00
28.00,46.06,1.00
25.00,22.95,1.00
36.00,59.43,1.00
15.00,26.89,1.00
34.00,58.41,1.00
32.00,48.31,1.00
19.00,35.64,1.00
20.00,36.34,1.00
14.00,24.92,1.00
21.00,39.67,1.00
19.00,30.47,1.00
30.00,51.02,1.00
26.00,48.49,1.00
27.00,47.60,1.00
36.00,79.77,1.00
27.00,39.21,1.00
19.00,35.03,1.00
25.00,44.85,1.00
28.00,46.73,1.00
15.00,24.81,1.00
18.00,29.94,1.00
14.00,24.91,1.00
28.00,52.74,1.00
17.00,26.08,1.00
101.00,85.64,1.00
17.00,32.39,1.00
33.00,47.91,1.00
22.00,43.30,1.00
27.00,47.68,1.00
200.00,113.97,0.00
27.00,47.66,1.00
35.00,55.59,1.00
21.00,38.53,1.00
26.00,44.32,1.00
16.00,27.20,1.00
20.00,34.87,1.00
33.00,56.56,1.00
16.00,26.49,1.00
33.00,52.67,1.00
16.00,16.93,1.00
29.00,64.23,1.00
14.00,25.53,1.00
32.00,56.26,1.00
28.00,50.18,1.00
16.00,23.22,1.00
25.00,43.49,1.00
17.00,29.97,1.00
23.00,36.14,1.00
30.00,51.53,1.00
31.00,55.51,1.00
15.00,27.04,1.00
17.00,32.93,1.00
39.00,57.28,1.00
19.00,29.06,1.00
18.00,29.48,1.00
26.00,45.48,1.00
26.00,39.48,1.00
15.00,26.02,1.00
19.00,35.00,1.00
26.00,50.46,1.00
16.00,25.47,1.00
23.00,39.17,1.00
19.00,34.81,1.00
26.00,47.02,1.00
23.00,41.52,1.00
27.00,45.33,1.00
21.00,20.56,1.00
28.00,41.28,1.00
29.00,49.74,1.00
22.00,31.76,1.00
24.00,39.90,1.00
30.00,40.28,1.00
30.00,51.54,1.00
28.00,48.10,1.00
28.00,52.50,1.00
18.00,28.43,1.00
27.00,44.12,1.00
16.00,23.20,1.00
34.00,54.25,1.00
34.00,60.75,1.00
13.00,19.38,1.00
17.00,25.26,1.00
14.00,18.18,1.00
17.00,17.81,1.00
18.00,26.91,1.00
16.00,29.16,1.00
29.00,48.90,1.00
18.00,32.33,1.00
14.00,19.70,1.00
26.00,45.34,1.00
21.00,39.48,1.00
25.00,34.49,1.00
28.00,47.21,1.00
31.00,52.68,1.00
17.00,17.51,1.00
16.00,23.33,1.00
22.00,35.00,1.00
23.00,33.65,1.00
15.00,27.88,1.00
24.00,44.66,1.00
32.00,47.15,1.00
20.00,30.27,1.00
20.00,31.94,1.00
20.00,36.35,1.00
23.00,39.11,1.00
19.00,29.18,1.00
23.00,42.68,1.00
27.00,47.53,1.00
27.00,44.08,1.00
22.00,39.57,1.00
27.00,50.09,1.00
37.00,82.54,1.00
15.00,21.94,1.00
60.00,62.16,1.00
20.00,39.00,1.00
18.00,30.67,1.00
27.00,46.50,1.00
13.00,23.11,1.00
21.00,33.53,1.00
15.00,26.22,1.00
17.00,28.36,1.00
35.00,55.92,1.00
25.00,46.77,1.00
23.00,43.28,1.00
18.00,35.07,1.00
26.00,41.23,1.00
24.00,41.70,1.00
32.00,58.87,1.00
18.00,30.94,1.00
112.00,129.10,1.00
33.00,67.45,1.00
14.00,25.67,1.00
22.00,28.69,1.00
32.00,72.20,1.00
43.00,71.23,1.00
200.00,61.88,0.00
51.00,60.59,1.00
16.00,29.60,1.00
28.00,44.72,1.00

but when I upload it, the score is 0, maybe this is because of an overfitting problem, but it's weird that the score is totally 0, like: only when I upload the official pretrained .ckpt, the score is not 0. How can I solve this issue?

[xuanlinli17]

Could you double check user_solution.py on e.g., are controller and observation mode correct?

[xtli12]

The problem had been solved, thank you very much!