Torch support

[mohanksriram]

Can we use reverb to store and retrieve PyTorch tensors? If yes, are there any performance implications?

[acassirer]

Hey,

I'm not very familiar with PyTorch tbh but Reverb is compatible with numpy ndarrays which PyTorch presumably also is? I know that PyTorch networks have been used together with Acme and Reverb in the past so there shouldn't be any hard blockers here. My best advice is to simply try it out and see how it performs!

[mohanksriram]

Hi,

Thanks for the update. I was concerned because I could not run a simple reverb server(the one in the official readme) without installing tensorflow as the ensure_tf_install.ensure_tf_version() method throws an error. I wonder whether it would be possible to isolate the tensorflow dependency for people who don't really need tensorflow.

Anyway, I'll try out reverb with pytorch. I really appreciate your effort in building a library specifically for experience replay! I enjoyed the core design paper too.

[ethanluoyc]

@mohanksriram I think reverb has a hard dependency on TensorFlow, e.g. The specification of the replay content needs to be specified as tf.TensorSpec, so you have to have tf installed.

Nevertheless, you don't have to use tf to implement your training. Since you can load data from reverb as tf.Dataset, you can create a dataset dataset and then use it as numpy iterator by calling dataset.as_numpy_iterator(), from that point onwards the data will just be numpy arrays. This is, for example, how Acme's JAX agents take advantage of reverb as their replay store. In that case, you may want to take a look at the examples in Acme for how they do this.

[mohanksriram]

Hi @ethanluoyc, Thank you so much for the detailed response. That helps a lot and also answers my concerns. I'm closing this issue for now.

[araonblake]

Hi @mohanksriram , I'm also trying to use PyTorch, did you successfully make Reverb work with PyTorch? If yes would you mind sharing an example? Thank you

[mohanksriram]

Hi @happyhackingit, As @ethanluoyc suggested you can use numpy_iterator() to obtain numpy arrays. You can later convert the numpy array into a torch tensor if you need to feed a torch model. Here's an example of how acme uses numpy_iterator.

[araonblake]

Got it, thanks @mohanksriram