VITA-Group / EnlightenGAN

[IEEE TIP] "EnlightenGAN: Deep Light Enhancement without Paired Supervision" by Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, Zhangyang Wang
Other
890 stars 198 forks source link

RuntimeError: cuda runtime error (10) : invalid device ordinal at torch/csrc/cuda/Module.cpp:88 #83

Open Joge-jiaye opened 3 years ago

Joge-jiaye commented 3 years ago

Traceback (most recent call last): File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connection.py", line 170, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/util/connection.py", line 96, in create_connection raise err File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/util/connection.py", line 86, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 706, in urlopen chunked=chunked, File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 394, in _make_request conn.request(method, url, **httplib_request_kw) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connection.py", line 234, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/http/client.py", line 1107, in request self._send_request(method, url, body, headers) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/http/client.py", line 1152, in _send_request self.endheaders(body) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/http/client.py", line 1103, in endheaders self._send_output(message_body) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/http/client.py", line 934, in _send_output self.send(msg) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/http/client.py", line 877, in send self.connect() File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connection.py", line 200, in connect conn = self._new_conn() File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connection.py", line 182, in _new_conn self, "Failed to establish a new connection: %s" % e urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f688e044b70>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 756, in urlopen method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/urllib3/util/retry.py", line 574, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8081): Max retries exceeded with url: /env/main (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f688e044b70>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/visdom/init.py", line 711, in _send data=json.dumps(msg), File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/visdom/init.py", line 677, in _handle_post r = self.session.post(url, data=data) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/requests/sessions.py", line 590, in post return self.request('POST', url, data=data, json=json, kwargs) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/requests/sessions.py", line 542, in request resp = self.send(prep, send_kwargs) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/requests/sessions.py", line 655, in send r = adapter.send(request, *kwargs) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8081): Max retries exceeded with url: /env/main (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f688e044b70>: Failed to establish a new connection: [Errno 111] Connection refused',)) [Errno 111] Connection refused create web directory ./checkpoints/enlightening/web... THCudaCheck FAIL file=torch/csrc/cuda/Module.cpp line=88 error=10 : invalid device ordinal Traceback (most recent call last): File "train.py", line 31, in model.optimize_parameters(epoch) File "/home/fjy/work/EnlightenGAN/models/single_model.py", line 394, in optimize_parameters self.forward() File "/home/fjy/work/EnlightenGAN/models/single_model.py", line 244, in forward self.fake_B, self.latent_real_A = self.netG_A.forward(self.real_img, self.real_A_gray) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 69, in forward inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 80, in scatter return scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 38, in scatter_kwargs inputs = scatter(inputs, target_gpus, dim) if inputs else [] File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 31, in scatter return scatter_map(inputs) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 18, in scatter_map return list(zip(map(scatter_map, obj))) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 15, in scatter_map return Scatter.apply(target_gpus, None, dim, obj) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/nn/parallel/_functions.py", line 74, in forward outputs = comm.scatter(input, ctx.target_gpus, ctx.chunk_sizes, ctx.dim, streams) File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/cuda/comm.py", line 188, in scatter with torch.cuda.device(device), torch.cuda.stream(stream): File "/home/fjy/.conda/envs/EG_py3.5/lib/python3.5/site-packages/torch/cuda/init.py", line 209, in enter torch._C._cuda_setDevice(self.idx) RuntimeError: cuda runtime error (10) : invalid device ordinal at torch/csrc/cuda/Module.cpp:88