search

LambdaColdStorage / lambda-tensorflow-benchmark

BSD 3-Clause "New" or "Revised" License
234 stars 60 forks source link

NVLink memory pooling #2

Open mratsim opened 5 years ago

mratsim commented 5 years ago

Congrats on the article.

Regarding this sentence:

Benchmark the 2080 Ti with multiple GPUs, with and without the NVLINK connector.

It sounds like even without NVLink we can pool memory of the GeForce, are you sure? (i.e. with 2 cards we can pass bigger batch size).

chuanli11 commented 5 years ago

@mratsim It appears 2080Ti does not support gpu peering without NVlink. We are currently investigating the impact of this on training. Initial results showed that it hurts replicated training for models with large number of weights (VGG16) or frequent weights update (AlexNet).