search

JuliaReinforcementLearning / ReinforcementLearningTrajectories.jl

A generalized experience replay buffer for reinforcement learning
MIT License
8 stars 8 forks source link

Document whether each Trajectory supports CUDA.jl or not #3

Closed findmyway closed 2 years ago

ludvigk commented 2 years ago

NStepBatchSampler and BatchSampler return non-contiguous SubArrays for state and next_state, which have to be collected before being transferred to the gpu. This seems to be very slow. Will incorporating the old consecutive_view function help with this issue?

findmyway commented 2 years ago

Will incorporating the old consecutive_view function help with this issue?

I'm afraid not. If I remember it correctly, the old consecutive_view also has the same issue.

The best we can do here is to reuse the collected array buffer instead of allocating a new one every time.

ludvigk commented 2 years ago

Yeah, I thought about that. Right now, it’s easy to do on the policy side, but it seems like a feature that is better to keep on the trajectory side of things. Question is if there are some algorithms that won’t benefit from a contiguous array, and thus getting an unnecessary slowdown from transferring the data to the contiguous array buffer. Perhaps it would be worth sacrificing those algorithms (if they exist) for the convenience factor?

findmyway commented 2 years ago

Question is if there are some algorithms that won’t benefit from a contiguous array, and thus getting an unnecessary slowdown from transferring the data to the contiguous array buffer.

This can be made pluggable. I may add an example soon.