Closed KumoLiu closed 3 months ago
[2024-05-20T04:03:05.050Z] Running ./3d_registration/learn2reg_oasis_unpaired_brain_mr.ipynb [2024-05-20T04:03:05.050Z] Checking PEP8 compliance... [2024-05-20T04:03:05.983Z] Running notebook... [2024-05-20T04:03:06.240Z] Before: [2024-05-20T04:03:06.240Z] "max_epochs = 50\n", [2024-05-20T04:03:06.240Z] "- Train the network longer, i.e., use `max_epochs = 200` in cell [Hyperparameters](#hyperparameters). \n", [2024-05-20T04:03:06.240Z] After: [2024-05-20T04:03:06.240Z] "max_epochs = 1\n", [2024-05-20T04:03:06.240Z] "- Train the network longer, i.e., use `max_epochs = 1` in cell [Hyperparameters](#hyperparameters). \n", [2024-05-20T04:03:06.240Z] Before: [2024-05-20T04:03:06.240Z] "val_interval = 1\n", [2024-05-20T04:03:06.240Z] After: [2024-05-20T04:03:06.240Z] "val_interval = 1\n", [2024-05-20T04:03:14.315Z] MONAI version: 1.3.1rc9 [2024-05-20T04:03:14.315Z] Numpy version: 1.24.4 [2024-05-20T04:03:14.315Z] Pytorch version: 2.3.0a0+40ec155e58.nv24.03 [2024-05-20T04:03:14.315Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False [2024-05-20T04:03:14.315Z] MONAI rev id: b16f54ab82e665f8643043ffba5c917fa417121b [2024-05-20T04:03:14.315Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py [2024-05-20T04:03:14.315Z] [2024-05-20T04:03:14.315Z] Optional dependencies: [2024-05-20T04:03:14.315Z] Pytorch Ignite version: 0.4.11 [2024-05-20T04:03:14.315Z] ITK version: 5.3.0 [2024-05-20T04:03:14.315Z] Nibabel version: 5.2.1 [2024-05-20T04:03:14.315Z] scikit-image version: 0.22.0 [2024-05-20T04:03:14.315Z] scipy version: 1.12.0 [2024-05-20T04:03:14.315Z] Pillow version: 10.2.0 [2024-05-20T04:03:14.315Z] Tensorboard version: 2.16.2 [2024-05-20T04:03:14.315Z] gdown version: 5.2.0 [2024-05-20T04:03:14.315Z] TorchVision version: 0.18.0a0 [2024-05-20T04:03:14.315Z] tqdm version: 4.66.2 [2024-05-20T04:03:14.315Z] lmdb version: 1.4.1 [2024-05-20T04:03:14.315Z] psutil version: 5.9.4 [2024-05-20T04:03:14.315Z] pandas version: 1.5.3 [2024-05-20T04:03:14.315Z] einops version: 0.7.0 [2024-05-20T04:03:14.315Z] transformers version: 4.40.2 [2024-05-20T04:03:14.315Z] mlflow version: 2.13.0 [2024-05-20T04:03:14.315Z] pynrrd version: 1.0.0 [2024-05-20T04:03:14.315Z] clearml version: 1.16.1 [2024-05-20T04:03:14.315Z] [2024-05-20T04:03:14.315Z] For details about installing the optional dependencies, please visit: [2024-05-20T04:03:14.315Z] https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies [2024-05-20T04:03:14.315Z] [2024-05-20T04:03:16.826Z] papermill --progress-bar --log-output -k python3 [2024-05-20T04:03:17.081Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:149: UserWarning: the file is not specified with any extension : - [2024-05-20T04:03:17.081Z] warnings.warn(f"the file is not specified with any extension : {os.path.basename(path)}") [2024-05-20T04:03:37.507Z] Executing: 0%| | 0/37 [00:00<?, ?cell/s] Executing: 3%|▎ | 1/37 [00:01<00:38, 1.06s/cell] Executing: 8%|▊ | 3/37 [00:10<02:09, 3.80s/cell] Executing: 14%|█▎ | 5/37 [00:16<01:48, 3.38s/cell] Executing: 24%|██▍ | 9/37 [00:17<00:44, 1.59s/cell] Executing: 24%|██▍ | 9/37 [00:20<01:02, 2.22s/cell] [2024-05-20T04:03:37.507Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:149: UserWarning: the file is not specified with any extension : - [2024-05-20T04:03:37.507Z] warnings.warn(f"the file is not specified with any extension : {os.path.basename(path)}") [2024-05-20T04:03:37.507Z] Traceback (most recent call last): [2024-05-20T04:03:37.507Z] File "/usr/local/bin/papermill", line 8, in <module> [2024-05-20T04:03:37.507Z] sys.exit(papermill()) [2024-05-20T04:03:37.507Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__ [2024-05-20T04:03:37.507Z] return self.main(*args, **kwargs) [2024-05-20T04:03:37.507Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main [2024-05-20T04:03:37.507Z] rv = self.invoke(ctx) [2024-05-20T04:03:37.507Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke [2024-05-20T04:03:37.507Z] return ctx.invoke(self.callback, **ctx.params) [2024-05-20T04:03:37.507Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke [2024-05-20T04:03:37.507Z] return __callback(*args, **kwargs) [2024-05-20T04:03:37.507Z] File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func [2024-05-20T04:03:37.507Z] return f(get_current_context(), *args, **kwargs) [2024-05-20T04:03:37.507Z] File "/usr/local/lib/python3.10/dist-packages/papermill/cli.py", line 235, in papermill [2024-05-20T04:03:37.507Z] execute_notebook( [2024-05-20T04:03:37.507Z] File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 131, in execute_notebook [2024-05-20T04:03:37.507Z] raise_for_execution_errors(nb, output_path) [2024-05-20T04:03:37.507Z] File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 251, in raise_for_execution_errors [2024-05-20T04:03:37.507Z] raise error [2024-05-20T04:03:37.507Z] papermill.exceptions.PapermillExecutionError: [2024-05-20T04:03:37.507Z] --------------------------------------------------------------------------- [2024-05-20T04:03:37.507Z] Exception encountered at "In [4]": [2024-05-20T04:03:37.507Z] --------------------------------------------------------------------------- [2024-05-20T04:03:37.507Z] SSLCertVerificationError Traceback (most recent call last) [2024-05-20T04:03:37.507Z] File /usr/lib/python3.10/urllib/request.py:1348, in AbstractHTTPHandler.do_open(self, http_class, req, **http_conn_args) [2024-05-20T04:03:37.507Z] 1347 try: [2024-05-20T04:03:37.507Z] -> 1348 h.request(req.get_method(), req.selector, req.data, headers, [2024-05-20T04:03:37.507Z] 1349 encode_chunked=req.has_header('Transfer-encoding')) [2024-05-20T04:03:37.507Z] 1350 except OSError as err: # timeout error [2024-05-20T04:03:37.507Z] [2024-05-20T04:03:37.507Z] File /usr/lib/python3.10/http/client.py:1283, in HTTPConnection.request(self, method, url, body, headers, encode_chunked) [2024-05-20T04:03:37.507Z] 1282 """Send a complete request to the server.""" [2024-05-20T04:03:37.507Z] -> 1283 self._send_request(method, url, body, headers, encode_chunked) [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /usr/lib/python3.10/http/client.py:1329, in HTTPConnection._send_request(self, method, url, body, headers, encode_chunked) [2024-05-20T04:03:37.508Z] 1328 body = _encode(body, 'body') [2024-05-20T04:03:37.508Z] -> 1329 self.endheaders(body, encode_chunked=encode_chunked) [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /usr/lib/python3.10/http/client.py:1278, in HTTPConnection.endheaders(self, message_body, encode_chunked) [2024-05-20T04:03:37.508Z] 1277 raise CannotSendHeader() [2024-05-20T04:03:37.508Z] -> 1278 self._send_output(message_body, encode_chunked=encode_chunked) [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /usr/lib/python3.10/http/client.py:1038, in HTTPConnection._send_output(self, message_body, encode_chunked) [2024-05-20T04:03:37.508Z] 1037 del self._buffer[:] [2024-05-20T04:03:37.508Z] -> 1038 self.send(msg) [2024-05-20T04:03:37.508Z] 1040 if message_body is not None: [2024-05-20T04:03:37.508Z] 1041 [2024-05-20T04:03:37.508Z] 1042 # create a consistent interface to message_body [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /usr/lib/python3.10/http/client.py:976, in HTTPConnection.send(self, data) [2024-05-20T04:03:37.508Z] 975 if self.auto_open: [2024-05-20T04:03:37.508Z] --> 976 self.connect() [2024-05-20T04:03:37.508Z] 977 else: [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /usr/lib/python3.10/http/client.py:1455, in HTTPSConnection.connect(self) [2024-05-20T04:03:37.508Z] 1453 server_hostname = self.host [2024-05-20T04:03:37.508Z] -> 1455 self.sock = self._context.wrap_socket(self.sock, [2024-05-20T04:03:37.508Z] 1456 server_hostname=server_hostname) [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /usr/lib/python3.10/ssl.py:513, in SSLContext.wrap_socket(self, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, session) [2024-05-20T04:03:37.508Z] 507 def wrap_socket(self, sock, server_side=False, [2024-05-20T04:03:37.508Z] 508 do_handshake_on_connect=True, [2024-05-20T04:03:37.508Z] 509 suppress_ragged_eofs=True, [2024-05-20T04:03:37.508Z] 510 server_hostname=None, session=None): [2024-05-20T04:03:37.508Z] 511 # SSLSocket class handles server_hostname encoding before it calls [2024-05-20T04:03:37.508Z] 512 # ctx._wrap_socket() [2024-05-20T04:03:37.508Z] --> 513 return self.sslsocket_class._create( [2024-05-20T04:03:37.508Z] 514 sock=sock, [2024-05-20T04:03:37.508Z] 515 server_side=server_side, [2024-05-20T04:03:37.508Z] 516 do_handshake_on_connect=do_handshake_on_connect, [2024-05-20T04:03:37.508Z] 517 suppress_ragged_eofs=suppress_ragged_eofs, [2024-05-20T04:03:37.508Z] 518 server_hostname=server_hostname, [2024-05-20T04:03:37.508Z] 519 context=self, [2024-05-20T04:03:37.508Z] 520 session=session [2024-05-20T04:03:37.508Z] 521 ) [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /usr/lib/python3.10/ssl.py:1100, in SSLSocket._create(cls, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, context, session) [2024-05-20T04:03:37.508Z] 1099 raise ValueError("do_handshake_on_connect should not be specified for non-blocking sockets") [2024-05-20T04:03:37.508Z] -> 1100 self.do_handshake() [2024-05-20T04:03:37.508Z] 1101 except (OSError, ValueError): [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /usr/lib/python3.10/ssl.py:1371, in SSLSocket.do_handshake(self, block) [2024-05-20T04:03:37.508Z] 1370 self.settimeout(None) [2024-05-20T04:03:37.508Z] -> 1371 self._sslobj.do_handshake() [2024-05-20T04:03:37.508Z] 1372 finally: [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007) [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] During handling of the above exception, another exception occurred: [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] URLError Traceback (most recent call last) [2024-05-20T04:03:37.508Z] Cell In[4], line 7 [2024-05-20T04:03:37.508Z] 5 if not os.path.exists(data_dir): [2024-05-20T04:03:37.508Z] 6 os.mkdir(data_dir) [2024-05-20T04:03:37.508Z] ----> 7 download_and_extract(resource, compressed_file, data_dir) [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:335, in download_and_extract(url, filepath, output_dir, hash_val, hash_type, file_type, has_base, progress) [2024-05-20T04:03:37.508Z] 333 with tempfile.TemporaryDirectory() as tmp_dir: [2024-05-20T04:03:37.508Z] 334 filename = filepath or Path(tmp_dir, _basename(url)).resolve() [2024-05-20T04:03:37.508Z] --> 335 download_url(url=url, filepath=filename, hash_val=hash_val, hash_type=hash_type, progress=progress) [2024-05-20T04:03:37.508Z] 336 extractall(filepath=filename, output_dir=output_dir, file_type=file_type, has_base=has_base) [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:226, in download_url(url, filepath, hash_val, hash_type, progress, **gdown_kwargs) [2024-05-20T04:03:37.508Z] 221 raise RuntimeError( [2024-05-20T04:03:37.508Z] 222 f"Download of file from {download_url}, received from {url} " [2024-05-20T04:03:37.508Z] 223 + f" to {filepath} failed due to network issue or denied permission." [2024-05-20T04:03:37.508Z] 224 ) [2024-05-20T04:03:37.508Z] 225 else: [2024-05-20T04:03:37.508Z] --> 226 _download_with_progress(url, tmp_name, progress=progress) [2024-05-20T04:03:37.508Z] 227 if not tmp_name.exists(): [2024-05-20T04:03:37.508Z] 228 raise RuntimeError( [2024-05-20T04:03:37.508Z] 229 f"Download of file from {url} to {filepath} failed due to network issue or denied permission." [2024-05-20T04:03:37.508Z] 230 ) [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:119, in _download_with_progress(url, filepath, progress) [2024-05-20T04:03:37.508Z] 117 except (URLError, HTTPError, ContentTooShortError, OSError) as e: [2024-05-20T04:03:37.508Z] 118 logger.error(f"Download failed from {url} to {filepath}.") [2024-05-20T04:03:37.508Z] --> 119 raise e [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/apps/utils.py:112, in _download_with_progress(url, filepath, progress) [2024-05-20T04:03:37.508Z] 109 self.update(b * bsize - self.n) # will also set self.n = b * bsize [2024-05-20T04:03:37.508Z] 111 with TqdmUpTo(unit="B", unit_scale=True, unit_divisor=1024, miniters=1, desc=_basename(filepath)) as t: [2024-05-20T04:03:37.508Z] --> 112 urlretrieve(url, filepath, reporthook=t.update_to) [2024-05-20T04:03:37.508Z] 113 else: [2024-05-20T04:03:37.508Z] 114 if not has_tqdm and progress: [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /usr/lib/python3.10/urllib/request.py:241, in urlretrieve(url, filename, reporthook, data) [2024-05-20T04:03:37.508Z] 224 """ [2024-05-20T04:03:37.508Z] 225 Retrieve a URL into a temporary location on disk. [2024-05-20T04:03:37.508Z] 226 [2024-05-20T04:03:37.508Z] (...) [2024-05-20T04:03:37.508Z] 237 data file as well as the resulting HTTPMessage object. [2024-05-20T04:03:37.508Z] 238 """ [2024-05-20T04:03:37.508Z] 239 url_type, path = _splittype(url) [2024-05-20T04:03:37.508Z] --> 241 with contextlib.closing(urlopen(url, data)) as fp: [2024-05-20T04:03:37.508Z] 242 headers = fp.info() [2024-05-20T04:03:37.508Z] 244 # Just return the local path and the "headers" for file:// [2024-05-20T04:03:37.508Z] 245 # URLs. No sense in performing a copy unless requested. [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /usr/lib/python3.10/urllib/request.py:216, in urlopen(url, data, timeout, cafile, capath, cadefault, context) [2024-05-20T04:03:37.508Z] 214 else: [2024-05-20T04:03:37.508Z] 215 opener = _opener [2024-05-20T04:03:37.508Z] --> 216 return opener.open(url, data, timeout) [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /usr/lib/python3.10/urllib/request.py:519, in OpenerDirector.open(self, fullurl, data, timeout) [2024-05-20T04:03:37.508Z] 516 req = meth(req) [2024-05-20T04:03:37.508Z] 518 sys.audit('urllib.Request', req.full_url, req.data, req.headers, req.get_method()) [2024-05-20T04:03:37.508Z] --> 519 response = self._open(req, data) [2024-05-20T04:03:37.508Z] 521 # post-process response [2024-05-20T04:03:37.508Z] 522 meth_name = protocol+"_response" [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /usr/lib/python3.10/urllib/request.py:536, in OpenerDirector._open(self, req, data) [2024-05-20T04:03:37.508Z] 533 return result [2024-05-20T04:03:37.508Z] 535 protocol = req.type [2024-05-20T04:03:37.508Z] --> 536 result = self._call_chain(self.handle_open, protocol, protocol + [2024-05-20T04:03:37.508Z] 537 '_open', req) [2024-05-20T04:03:37.508Z] 538 if result: [2024-05-20T04:03:37.508Z] 539 return result [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /usr/lib/python3.10/urllib/request.py:496, in OpenerDirector._call_chain(self, chain, kind, meth_name, *args) [2024-05-20T04:03:37.508Z] 494 for handler in handlers: [2024-05-20T04:03:37.508Z] 495 func = getattr(handler, meth_name) [2024-05-20T04:03:37.508Z] --> 496 result = func(*args) [2024-05-20T04:03:37.508Z] 497 if result is not None: [2024-05-20T04:03:37.508Z] 498 return result [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /usr/lib/python3.10/urllib/request.py:1391, in HTTPSHandler.https_open(self, req) [2024-05-20T04:03:37.508Z] 1390 def https_open(self, req): [2024-05-20T04:03:37.508Z] -> 1391 return self.do_open(http.client.HTTPSConnection, req, [2024-05-20T04:03:37.508Z] 1392 context=self._context, check_hostname=self._check_hostname) [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] File /usr/lib/python3.10/urllib/request.py:1351, in AbstractHTTPHandler.do_open(self, http_class, req, **http_conn_args) [2024-05-20T04:03:37.508Z] 1348 h.request(req.get_method(), req.selector, req.data, headers, [2024-05-20T04:03:37.508Z] 1349 encode_chunked=req.has_header('Transfer-encoding')) [2024-05-20T04:03:37.508Z] 1350 except OSError as err: # timeout error [2024-05-20T04:03:37.508Z] -> 1351 raise URLError(err) [2024-05-20T04:03:37.508Z] 1352 r = h.getresponse() [2024-05-20T04:03:37.508Z] 1353 except: [2024-05-20T04:03:37.508Z] [2024-05-20T04:03:37.508Z] URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)>