This is highly dependent on many factors. How much memory does your machine have? The easiest thing to do would be to reduce the size of the replay buffer. Alternatively you could re-write the replay buffer code to be more memory efficient (e.g. not store obs and next_obs separately but instead have a list of indices that point to the next observation, storing actions as single ints rather than one-hot, etc)
This is highly dependent on many factors. How much memory does your machine have? The easiest thing to do would be to reduce the size of the replay buffer. Alternatively you could re-write the replay buffer code to be more memory efficient (e.g. not store obs and next_obs separately but instead have a list of indices that point to the next observation, storing actions as single ints rather than one-hot, etc)