search

alexhernandezgarcia / gflownet

Generative Flow Networks - GFlowNet
https://gflownet.readthedocs.io/en/latest/
Apache License 2.0
161 stars 10 forks source link

The proxy should not be copied into each environment instance #288

Closed alexhernandezgarcia closed 3 months ago

alexhernandezgarcia commented 7 months ago

Currently, the proxy is set as an attribute of the environments and the base environment implements the methods proxy2reward() and reward2proxy() that determine the conversion between proxy outputs and reward. The environment also implements the methods reward() and reward_batch(), which call the proxy and the conversion methods. This is probably not ideal for various reasons.

I do not see any longer a good reason to keep the proxy and these methods within the environment. It seems possible and a good idea to completely detach the environment and the proxy. Some proxies need information from the environment, which is currently set via the call to Env.setup_proxy(), which calls the proxy's setup() method. But this could just be done elsewhere.

Now, in terms of alternatives, I am not completely settled on what the best option would be. In particular, where should the methods that convert between proxy and reward go?

alexhernandezgarcia commented 6 months ago

It seems that it would be better to have the methods in the base proxy class, so as to make it easier for non-GFN baselines to re-use the GFlowNet code.

alexhernandezgarcia commented 3 months ago

Done in PR https://github.com/alexhernandezgarcia/gflownet/pull/299