hi, I'm training a model with 1000 samples, where the GT size is 10241024 and the LR input size is 256\256. I haven't changed any other parameters, but I'm only able to train for 5 epochs in a day on a GPU with 8GB of memory. I've never trained a super-resolution model before, so I'm wondering if this is normal, or if it's a machine issue, or something else. I noticed that the training time statistics only cover the "model(lr)" step, which is quite fast, taking only a few seconds. It seems like other unaccounted-for steps are taking longer,"loss.backward()" cost long time.
vs code notifications：pydevd warning:computing repr of gt(Tensor) was slow(took 16.42s) Customize report timeout by setting the 'PYDEVD_WARN_SLOW_RESOLVE_TIMEOUT‘ environment variable to a higher timeout(default is 0.5s)
hi, I'm training a model with 1000 samples, where the GT size is 10241024 and the LR input size is 256\256. I haven't changed any other parameters, but I'm only able to train for 5 epochs in a day on a GPU with 8GB of memory. I've never trained a super-resolution model before, so I'm wondering if this is normal, or if it's a machine issue, or something else. I noticed that the training time statistics only cover the "model(lr)" step, which is quite fast, taking only a few seconds. It seems like other unaccounted-for steps are taking longer,"loss.backward()" cost long time. vs code notifications：pydevd warning:computing repr of gt(Tensor) was slow(took 16.42s) Customize report timeout by setting the 'PYDEVD_WARN_SLOW_RESOLVE_TIMEOUT‘ environment variable to a higher timeout(default is 0.5s)