In pearl, qf1 and qf2 should be differently initialized

rlworkgroup / garage

A toolkit for reproducible reinforcement learning research.

MIT License

1.86k stars 309 forks source link

In pearl, qf1 and qf2 should be differently initialized #2215

Open JeongHyunho opened 3 years ago

JeongHyunho commented 3 years ago

As implemented in github.com/katerakelly/oyster, qf1 and qf2 should be differently initialized. Otherwise, two functions are always same because those are updated identically. Or, let me know what I am wrong.

https://github.com/katerakelly/oyster/blob/44e20fddf181d8ca3852bdf9b6927d6b8c6f48fc/launch_experiment.py#L45-L53

https://github.com/rlworkgroup/garage/blob/82b5c33ae0796489a00391f80cb94e41657f5962/src/garage/torch/algos/pearl.py#L137-L138

ryanjulian commented 3 years ago

@JeongHyunho thank you for this issue report!

This indeed looks like a bug, but I'd like to verify. Does self._qf2 have the same parameters as self._qf1?