Farama-Foundation / Gymnasium-Robotics

A collection of robotics simulation environments for reinforcement learning
https://robotics.farama.org/
MIT License
529 stars 85 forks source link

Adroit hand dense reward fixes #220

Open 7oponaut opened 3 months ago

7oponaut commented 3 months ago

Description

This PR fixes the hand-to-object dense reward component signs in the adroit hand door, hammer, and relocate scenarios.

The bugs cause the agent to maximize distance between the hand and the object of interest.

The bugs were introduced in 7b5aa90 with a refactor of the relevant code sections. I ran trainings before and after the fix to confirm that it works.

As far as I can tell the pen scenario is not affected, so no changes there.

I couldn't find the relevant CONTRIBUTING.md file to set stuff up for precommit and unit testing.

Type of change

Checklist:

Kallinteris-Andreas commented 3 months ago

@7oponaut We use the CONTRIBUTING.md, from gymnasium

this change (reward function adjustment) requires a new version to be made (''AdroitHand*-v2''), this can be done by creating new files with the fixes and registering them in https://github.com/Farama-Foundation/Gymnasium-Robotics/blob/0a213bb29e031a3a8a0612c4b08b32569954164a/gymnasium_robotics/__init__.py#L1200, for reference, check how the versions are handled for other environments

7oponaut commented 3 months ago

@Kallinteris-Andreas Taking a closer look, 7b5aa90 changed reward behavior in the sparse reward setting in addition to the accidental change that causes issues. It also added termination conditions, this affects both the dense and sparse cases. The version numbers weren't bumped to "AdroitHand*-v2" though.

Which is to be considered v1, 7b5aa90 or the version before that?

I will take a look at CONTRIBUTING.md from gymnasium, thank you.

Kallinteris-Andreas commented 3 months ago

@7oponaut you are correct, the reward function was changed then without a version bump, which is a mistake, this should at least be documented (we likely remove those environment versions later because of that) for reference, the Adroid*-v1 env was added in gymnasium_robotics==1.2.0 and the reward function was changed in gymnasium_robotics==1.2.1

regardless, the change proposed here requires a version bump