reset

[waynezw0618]

Hello again @simensov I see for reset you randomly select the value for the states for both pose and velocity of the bounds. https://github.com/simensov/ml4ca/blob/e2e75f3785455a29faa92881870605590e07425f/src/rl/windows_workspace/specific/customEnv.py#L141-L150. it is easy to understand that you have a random initial velocities. but I don't understand what is the meaning of random pose, is it in the error frame or in global frame. I mean how to set the initial points for each episode in the training. do I need take the random pose into the consideration. since I suppose for each episode you randomly peak up a point some distance and angle away from the set point. do you need to add the random pose to that point?

Best Regards Wei

[simensov]

Hi,

The selection of randomized starting pose (position and heading) was done in the NED-frame (or the "global frame"). But I simply used the same point in the NED-frame as set point for DP in each episode, meaning that if I chose random pose in the NED-frame, it also meant choosing random pose in the error frame.

[waynezw0618]

Hi I may not well understand yet. if you select a random starting pose, then what is the reference for the start point? if it is the same, the state in error frame is zeros for all. is it ?

[simensov]

The setpoint in error-frame is always [0, 0, 0] -> eliminate all errors. This was set to a fixed point in the NED-frame, native to the simulator that I used from DNV. I believe the coordinates were along the lines of 63°26' N, 10°25' E, and the heading was towards true north (0 deg) - that is in the Dorabassenget bay area in Trondheim, Norway. So the real, random pose was selected to be a deviation from this NED-frame pose, limited by the size of the elements in self.real_ss_bounds

[waynezw0618]

• Since during training, you randomly select a start point for each episode. do you control the distance between start point and setpoint?Since when the distance is different the number of steps could be difference. so that the accumulated reward as well the discounted reward or return is different for each. does that affect training?
• what is the state of pose in error frame. is it different between pose in NED-frame and the setpoint. or it is different between pose and the reference? I mean the in section 5.4.2 above eqn.5.7a) is a fixed value of setpoint pose in NED-frame or an array from guidance system?

[simensov]

• The distance is controlled / limited by the state space definition I made. Yes; the number of steps could be different. The trajectories are in fact almost never the same size until the DRL model has become good enough to always avoid terminal states. But remember that we are not looking for the optimal trajectory itself, so the length and accumulated return of each episode is not the main concern, rather the optimal action selection in each state! Yes, the chance of the learning being improved by long episodes that ends up with high rewards is higher, but the DRL model itself only knows of what state it is being presented with at every time instant.
• The state in the error frame is the error-state in body-frame between the current pose and the setpoint/reference (I have used these two words for the same thing, coming from the main literature). See the class ErrorFrame, and how I used the transformation matrix to calculate the error state: https://github.com/simensov/ml4ca/blob/e2e75f3785455a29faa92881870605590e07425f/src/rl/windows_workspace/specific/errorFrame.py#L25-L32