A problem about timestep in Deepbots

[speedhawk]

Hi, Sorry for trouble you again and hope you are well. I got a problem in creating the environments of reinforcement learning training. I just remember that you have proposed me before a thought that with deepbots we can construct a similar world as many cases of gym environments, in which the the agent moves by not continuous driving but actions moving one space in a different direction (such as cliff walking which is a typical grid world). Thanks to your enlightment, I am now trying to create a grid world with arena map with black and white plaid over it. Now, my main thought is to create a space including actions by which the robot agent can move a constant distance, just like the same distance to one grid step ahead to different direction (forward, back, left, right). However, I got a problem with timestep. As you know, the timestep is an attribute of Supervisor class, which is the super class of even all of relative subclasses. When the iteration starts, the function step(), which is the function of Supervisor class, will be called by the step() function in subclass and automatically call the timestep attribute from Supervisor class so that the 3D simulating animation can push forward along the timeline. The trouble is, if I create a space with 'grid moving style' actions (such as moving for a grid distance, or rotate a given angle in one step of iteration), the function also use timestep. For example, this is the original step function in Supervisor class:

    def step(self, action):
        self.handle_emitter(action)
        if super(Supervisor, self).step(self.timestep) == -1:
            exit()

        return (
            self.get_observations(),
            self.get_reward(action),
            self.is_done(),
            self.get_info(),
        )

And based on some resources online, one method to implement 'grid moving style' action can be achieved like this:

def turn_left(angle):
    l_speed = 1.0
    r_speed = -1.0
    leftMotor.setVelocity(l_speed)
    rightMotor.setVelocity(r_speed)

    while robot.step(self.timestep) != -1:
        if robot.getFromDef('agent').getField('rotation').getSFRotation[3] >= 1.5708:
            leftMotor.setVelocity(0.0)
            rightMotor.setVelocity(0.0)
        break

Here, timestep cannot be called simultanerously by two processes. Therefore, do you have any ideas to solve the problem? Or any better solutions for create grid world in Webots? Many thanks for any help or advice!

[tsampazk]

Hey @speedhawk! From what i understand from your description, the way you are approaching the problem is not ideal if i am not mistaken. Instead of implementing movement through the motors, I would suggest to use the supervisor to move the robot to the required grid position instantly, or rotate it instantly by setting its position and rotation. This would probably require some groundwork for the grid so you would know exactly at which world coordinates each grid square is.

I am not sure whether this solution fits your problem, hope it gives you some ideas. Let me know if i can offer any additional help.

[speedhawk]

Hi @tsampazk ! Really graceful for your reply with some advice! Do you have any idea for the implementation of grid-world construction groundworks (such as occupancy grid mapping)? Is it able to be achieved by deepbots or requiring other plugins or resources?

Instead of implementing movement through the motors, I would suggest to use the supervisor to move the robot to the required grid position instantly, or rotate it instantly by setting its position and rotation.

Additionally, this is my first time to hear that the robot agent can move like this in Webots without motor LoL. It is really interesting. Could you please give me some enlighting or advice about how to implement it? Many thanks!