search

krumiaa / MindMaker

MindMaker UE4 Machine Learning Toolkit
255 stars 31 forks source link

Terminating episode early #24

Open rswlkr opened 2 years ago

rswlkr commented 2 years ago

I'm setting the done flag when I reach an early termination event with the expectation that the episode will end. I track the number of steps and episodes within UE but it seems even after setting the done flag to true and passing it back through mindmaker, the episode continues until the defined number of timesteps is complete - how can I terminate early? I'm using PPO2.

rswlkr commented 2 years ago

looks like it might have been thought about before #11 ?

krumiaa commented 2 years ago

Apologies, I dont think the done flag is currently working, but it's easy to terminate early with blueprints, just check for a certain number of episodes to have elapsed with a C branch node, and if it has, use the quit game node to exit out. I'm not sure what added value there would be to using the done nose since once you quit, it closes the connection with the mindmaker server and halts training.

On Mon, Sep 26, 2022 at 2:25 PM Ross Walker @.***> wrote:

I'm setting the done flag when I reach an early termination event with the expectation that the episode will end. I track the number of steps and episodes within UE but it seems even after setting the done flag to true and passing it back through mindmaker, the episode continues until the defined number of timesteps is complete - how can I terminate early? I'm using PPO2.

— Reply to this email directly, view it on GitHub https://github.com/krumiaa/MindMaker/issues/24, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABI6N2L22FKENGOXM3WNSIDWAHTBPANCNFSM6AAAAAAQWBU3XY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

rswlkr commented 2 years ago

I'm not talking about terminating the program, 'done' is about finishing the episode as far as I understand it?

On further inspection...setting the done flag in mindmaker is doing something, the summary printout includes additional line items when using done to terminate an episode early, notably 'average num steps per episode' which is less than n_steps 256 that is set in the params.

I was getting thrown off because I was expecting to see that summary printout after every episode ends but it appears it happens after n_steps regardless of early termination which could make sense. Early terminated episodes also don't show up in tensorboard, maybe because the reward is set to 0?

krumiaa commented 1 year ago

you are correct, the dones are referring to episode termination. You will need to reference the source code which is included in the mindmaker drl project under mindmaker/source to see what specifically is happening with them, I thought I had forgot to implement this but maybe it is being passed to mindmaker after all if you are seeing those results. Il have a look before the next release and maybe add something in documentation clarifying this

On Tue, Sep 27, 2022 at 11:36 AM Ross Walker @.***> wrote:

I'm not talking about terminating the program, 'done' is about finishing the episode as far as I understand it?

On further inspection...setting the done flag in mindmaker is doing something, the summary printout includes additional line items when using done to terminate an episode early, notably 'average num steps per episode' which is less than n_steps 256 that is set in the params.

I was getting thrown off because I was expecting to see that summary printout after every episode ends but it appears it happens after n_steps regardless of early termination which could make sense. Early terminated episodes also don't show up in tensorboard, maybe because the reward is set to 0?

— Reply to this email directly, view it on GitHub https://github.com/krumiaa/MindMaker/issues/24#issuecomment-1259902173, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABI6N2N7D2G34IORXZVXYMLWAM5BLANCNFSM6AAAAAAQWBU3XY . You are receiving this because you commented.Message ID: @.***>