Ask developers to help answer this question

[fengzfcany]

[2022-05-31 02:44:25] [316241c6] [30] urllib3.connectionpool: http://192.168.2.109:8080 "GET /api/v1/allocations/7.ee72787d-c20e-4824-b0da-ef3e3287e708.1/signals/preemption?timeout_seconds=60 HTTP/1.1" 200 17

[2022-05-31 02:44:25] [316241c6] [30] determined.core: (_PreemptionWatcher thread) _get_preemption(60) [2022-05-31 02:44:25] [316241c6] [30] urllib3.connectionpool: Starting new HTTP connection (1): 192.168.2.109:8080 [2022-05-31 02:44:37] [316241c6] Timeout (0:00:30)! [2022-05-31 02:44:37] [316241c6] Thread 0x00007f4763bfe700 (most recent call first): [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31 02:44:37] [316241c6] [2022-05-31 02:44:37] [316241c6] Thread 0x00007f476bcab700 (most recent call first): [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request

[kelvinwang139]

[2022-05-31 02:44:25] [316241c6] [30] urllib3.connectionpool: http://192.168.2.109:8080 "GET /api/v1/allocations/7.ee72787d-c20e-4824-b0da-ef3e3287e708.1/signals/preemption?timeout_seconds=60 HTTP/1.1" 200 17 [2022-05-31 02:44:25] [316241c6] [30] determined.core: (_PreemptionWatcher thread) _get_preemption(60) [2022-05-31 02:44:25] [316241c6] [30] urllib3.connectionpool: Starting new HTTP connection (1): 192.168.2.109:8080 [2022-05-31 02:44:37] [316241c6] Timeout (0:00:30)! [2022-05-31 02:44:37] [316241c6] Thread 0x00007f4763bfe700 (most recent call first): [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31 02:44:37] [316241c6] [2022-05-31 02:44:37] [316241c6] Thread 0x00007f476bcab700 (most recent call first): [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31 02:44:37] [316241c6] File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request

Full logs added: [2022-05-31T14:24:06.142412Z] || INFO: Scheduling Trial (Experiment 34) (id: 29c616a1-c0bf-418c-a8bd-b855ef8ac23d) [2022-05-31T14:24:06.254675Z] || INFO: Trial (Experiment 34) was assigned to an agent [2022-05-31T14:24:06.271551Z] dc5b74e6 || INFO: image already found, skipping pull phase: docker.io/determinedai/model-hub-mmdetection:0.18.1-dev0 [2022-05-31T14:24:06.356988Z] dc5b74e6 || INFO: copying files to container: / [2022-05-31T14:24:06.444792Z] dc5b74e6 || INFO: copying files to container: /run/determined [2022-05-31T14:24:06.530716Z] dc5b74e6 || INFO: copying files to container: / [2022-05-31T14:24:06.623276Z] dc5b74e6 || INFO: copying files to container: / [2022-05-31T14:24:06.711453Z] dc5b74e6 || INFO: copying files to container: / [2022-05-31T14:24:06.787100Z] dc5b74e6 || INFO: copying files to container: / [2022-05-31T14:24:06.876627Z] dc5b74e6 || INFO: copying files to container: / [2022-05-31T14:24:07.560224Z] dc5b74e6 || INFO: Container for Trial (Experiment 34) has started [2022-05-31T14:24:09.966560Z] dc5b74e6 || WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [2022-05-31T14:24:11.459361Z] dc5b74e6 || INFO: [30] root: New trial runner in (container dc5b74e6-9d7c-4d36-b11d-6e064e58ccbe) on agent determined-agent-0: {"bind_mounts": [{"host_path": "/root/fengzf/data", "container_path": "/tmp", "read_only": false, "propagation": "rprivate"}], "checkpoint_policy": "best", "checkpoint_storage": {"host_path": "/root/.local/share/determined", "propagation": "rprivate", "save_experiment_best": 0, "save_trial_best": 1, "save_trial_latest": 5, "storage_path": null, "type": "shared_fs"}, "data_layer": {"container_storage_path": null, "host_storage_path": null, "type": "shared_fs"}, "data": {"file_client_args": {"backend": "disk"}}, "debug": true, "description": null, "entrypoint": "python3 -m determined.launch.torch_distributed --trial model_hub.mmdetection:MMDetTrial", "environment": {"image": {"cpu": "determinedai/environments:py-3.8-pytorch-1.10-lightning-1.5-tf-2.8-cpu-6e45071", "cuda": "determinedai/model-hub-mmdetection:0.18.1-dev0", "rocm": "determinedai/environments:rocm-4.2-pytorch-1.9-tf-2.5-rocm-6e45071"}, "environment_variables": {"cpu": ["OMP_NUM_THREADS=1"], "cuda": ["OMP_NUM_THREADS=1"], "rocm": ["OMP_NUM_THREADS=1"]}, "ports": {"trial": 1734}, "registry_auth": null, "force_pull_image": false, "pod_spec": null, "slurm": [], "add_capabilities": [], "drop_capabilities": []}, "hyperparameters": {"config_file": {"type": "const", "val": "faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py"}, "global_batch_size": {"type": "const", "val": 16}, "merge_config": {"type": "const", "val": null}, "override_mmdet_config": {"data.train.ann_file": {"type": "const", "val": "/tmp/coco/annotations/instances_train2017.json"}, "data.val.ann_file": {"type": "const", "val": "/tmp/coco/annotations/instances_val2017.json"}}, "use_pretrained": {"type": "const", "val": false}}, "labels": [], "max_restarts": 5, "min_checkpoint_period": {"batches": 0}, "min_validation_period": {"batches": 7320}, "name": "model_hub_mmdet_experiment", "optimizations": {"aggregation_frequency": 1, "average_aggregated_gradients": true, "average_training_metrics": false, "gradient_compression": false, "grad_updates_size_file": null, "mixed_precision": "O0", "tensor_fusion_threshold": 64, "tensor_fusion_cycle_time": 5, "auto_tune_tensor_fusion": false}, "perform_initial_validation": false, "profiling": {"enabled": false, "begin_on_batch": 0, "end_after_batch": null, "sync_timings": true}, "records_per_epoch": 0, "reproducibility": {"experiment_seed": 1654007046}, "resources": {"max_slots": null, "slots_per_trial": 2, "weight": 1, "native_parallel": false, "shm_size": 200000000000, "agent_label": "", "resource_pool": "default", "priority": null, "devices": []}, "scheduling_unit": 100, "searcher": {"max_length": {"batches": 87850}, "metric": "bbox_mAP", "name": "single", "smaller_is_better": false, "source_checkpoint_uuid": null, "source_trial_id": null}} [2022-05-31T14:24:11.459374Z] dc5b74e6 || INFO: [30] root: Validating checkpoint storage ... [2022-05-31T14:24:11.459908Z] dc5b74e6 || INFO: [30] root: Launching: ['sh', '-c', 'python3 -m determined.launch.torch_distributed --trial model_hub.mmdetection:MMDetTrial'] [2022-05-31T14:24:14.115103Z] dc5b74e6 [rank=0] || DEBUG: [49] root: Starting harness. [2022-05-31T14:24:14.115111Z] dc5b74e6 [rank=0] || INFO: [49] root: Loading Trial implementation with entrypoint model_hub.mmdetection:MMDetTrial. [2022-05-31T14:24:14.121974Z] dc5b74e6 [rank=1] || DEBUG: [52] root: Starting harness. [2022-05-31T14:24:14.121978Z] dc5b74e6 [rank=1] || INFO: [52] root: Loading Trial implementation with entrypoint model_hub.mmdetection:MMDetTrial. [2022-05-31T14:24:23.466961Z] dc5b74e6 [rank=1] || DEBUG: [52] matplotlib: matplotlib data path: /opt/conda/lib/python3.8/site-packages/matplotlib/mpl-data [2022-05-31T14:24:23.467284Z] dc5b74e6 [rank=0] || DEBUG: [49] matplotlib: matplotlib data path: /opt/conda/lib/python3.8/site-packages/matplotlib/mpl-data [2022-05-31T14:24:23.471102Z] dc5b74e6 [rank=1] || DEBUG: [52] matplotlib: CONFIGDIR=/root/.config/matplotlib [2022-05-31T14:24:23.471289Z] dc5b74e6 [rank=0] || DEBUG: [49] matplotlib: CONFIGDIR=/root/.config/matplotlib [2022-05-31T14:24:23.472636Z] dc5b74e6 [rank=1] || DEBUG: [52] matplotlib: interactive is False [2022-05-31T14:24:23.472641Z] dc5b74e6 [rank=1] || DEBUG: [52] matplotlib: platform is linux [2022-05-31T14:24:23.472958Z] dc5b74e6 [rank=0] || DEBUG: [49] matplotlib: interactive is False [2022-05-31T14:24:23.472960Z] dc5b74e6 [rank=0] || DEBUG: [49] matplotlib: platform is linux [2022-05-31T14:24:23.474220Z] dc5b74e6 [rank=1] || DEBUG: [52] matplotlib: loaded modules: ['sys', 'builtins', '_frozen_importlib', '_imp', '_warnings', '_io', 'marshal', 'posix', '_frozen_importlib_external', '_thread', '_weakref', 'time', 'zipimport', 'faulthandler', '_codecs', 'codecs', 'encodings.aliases', 'encodings', 'encodings.utf_8', '_signal', 'main', 'encodings.latin_1', '_abc', 'abc', 'io', '_stat', 'stat', '_collections_abc', 'genericpath', 'posixpath', 'os.path', 'os', '_sitebuiltins', '_locale', '_bootlocale', 'types', 'importlib._bootstrap', 'importlib._bootstrap_external', 'warnings', 'importlib', 'importlib.machinery', 'importlib.abc', '_operator', 'operator', 'keyword', '_heapq', 'heapq', 'itertools', 'reprlib', '_collections', 'collections', '_functools', 'functools', 'contextlib', 'importlib.util', 'google', 'google.cloud', 'google.logging', 'mpl_toolkits', 'ruamel', 'site', '_weakrefset', 'weakref', 'pkgutil', 'runpy', 'determined.version', 'collections.abc', 'enum', '_sre', 'sre_constants', 'sre_parse', 'sre_compile', 'copyreg', 're', 'typing.io', 'typing.re', 'typing', 'determined._experiment_config', '_json', 'json.scanner', 'json.decoder', 'json.encoder', 'json', '_csv', 'csv', 'token', 'tokenize', 'linecache', 'traceback', '_string', 'string', 'threading', 'atexit', 'logging', 'errno', 'signal', '_posixsubprocess', 'select', 'math', 'selectors', 'subprocess', 'determined.gpu', 'determined._info', 'fnmatch', 'zlib', '_compression', '_bz2', 'bz2', '_lzma', 'lzma', 'pwd', 'grp', 'shutil', '_socket', 'socket', '_bisect', 'bisect', '_sha512', '_random', 'random', 'tempfile', 'determined.constants', 'future', '_datetime', 'datetime', 'psutil._common', 'psutil._compat', '_struct', 'struct', 'binascii', 'base64', 'glob', 'psutil._psposix', 'psutil._psutil_linux', 'psutil._psutil_posix', 'resource', 'psutil._pslinux', 'psutil', 'cython_runtime', 'textwrap', 'ruamel.yaml.compat', 'ruamel.yaml.error', 'ruamel.yaml.util', 'ruamel.yaml.reader', '_opcode', 'opcode', 'dis', 'inspect', 'ruamel.yaml.tokens', 'ruamel.yaml.scanner', 'ruamel.yaml.events', 'copy', 'ruamel.yaml.anchor', 'ruamel.yaml.scalarstring', 'ruamel.yaml.comments', 'ruamel.yaml.parser', 'ruamel.yaml.nodes', 'ruamel.yaml.composer', 'ruamel.yaml.scalarint', 'ruamel.yaml.scalarfloat', 'ruamel.yaml.scalarbool', 'ruamel.yaml.timestamp', 'ruamel.yaml.constructor', 'ruamel.yaml.emitter', 'ruamel.yaml.serializer', 'ruamel.yaml.representer', '_ruamel_yaml', 'ruamel.yaml.resolver', 'ruamel.yaml.cyaml', 'ruamel.yaml.loader', 'ruamel.yaml.dumper', 'ruamel.yaml.main', 'ruamel.yaml', 'locale', 'gettext', 'argparse', 'termios', 'getpass', '_hashlib', '_blake2', '_sha3', 'hashlib', 'ntpath', 'urllib', 'urllib.parse', 'pathlib', 'filelock._error', 'filelock._api', 'filelock._util', 'filelock._soft', 'fcntl', 'filelock._unix', 'filelock._windows', 'filelock.version', 'filelock', 'determined.common.constants', 'platform', 'determined.common.util', 'importlib.resources', 'certifi.core', 'certifi', 'determined.common.api.certs', 'determined.common.api.authentication', 'urllib3.packages', 'urllib3.packages.six', 'urllib3.packages.six.moves', 'http', 'email', 'email.errors', 'email.quoprimime', 'email.base64mime', 'quopri', 'email.encoders', 'email.charset', 'email.header', 'calendar', 'email._parseaddr', 'email.utils', 'email._policybase', 'email.feedparser', 'email.parser', 'uu', 'email._encoded_words', 'email.iterators', 'email.message', '_ssl', 'ssl', 'http.client', 'urllib3.packages.six.moves.http_client', 'urllib3.exceptions', 'urllib3._version', 'urllib3.contrib', 'urllib3.contrib._appengine_environ', 'urllib3.util.wait', 'urllib3.util.connection', '_cffi_backend', '_brotli.lib', '_brotli', 'brotli.brotli', 'brotli.brotli', 'brotli', 'urllib3.util.request', 'urllib3.util.response', 'urllib3.util.retry', 'hmac', 'urllib3.util.url', 'urllib3.util.ssltransport', 'urllib3.util.ssl', 'urllib3.util.timeout', 'urllib3.util', 'urllib3.util.proxy', 'urllib3._collections', 'ipaddress', 'urllib3.util.ssl_match_hostname', 'urllib3.connection', '_queue', 'queue', 'mimetypes', 'urllib3.fields', 'urllib3.filepost', 'urllib3.packages.six.moves.urllib', 'urllib3.packages.six.moves.urllib.parse', 'urllib3.request', 'urllib3.response', 'urllib3.util.queue', 'urllib3.connectionpool', 'urllib3.poolmanager', 'urllib3', 'charset_normalizer.constant', 'unicodedata', '_multibytecodec', 'charset_normalizer.utils', 'charset_normalizer.md', 'charset_normalizer.models', 'charset_normalizer.assets', 'charset_normalizer.cd', 'charset_normalizer.api', 'charset_normalizer.legacy', 'charset_normalizer.version', 'charset_normalizer', 'urllib.response', 'urllib.error', 'urllib.request', 'http.cookiejar', 'http.cookies', 'requests.compat', 'requests.exceptions', 'requests.version', 'zipfile', 'requests.certs', 'requests._internal_utils', 'requests.cookies', 'requests.structures', 'requests.utils', 'requests.packages.urllib3.packages', 'requests.packages.urllib3.packages.six', 'requests.packages.urllib3.packages.six.moves', 'requests.packages.urllib3.packages.six.moves.http_client', 'requests.packages.urllib3.exceptions', 'requests.packages.urllib3._version', 'requests.packages.urllib3.contrib', 'requests.packages.urllib3.contrib._appengineenviron', 'requests.packages.urllib3.util.wait', 'requests.packages.urllib3.util.connection', 'requests.packages.urllib3.util.request', 'requests.packages.urllib3.util.response', 'requests.packages.urllib3.util.retry', 'requests.packages.urllib3.util.url', 'requests.packages.urllib3.util.ssltransport', 'requests.packages.urllib3.util.ssl', 'requests.packages.urllib3.util.timeout', 'requests.packages.urllib3.util', 'requests.packages.urllib3.util.proxy', 'requests.packages.urllib3._collections', 'requests.packages.urllib3.util.ssl_match_hostname', 'requests.packages.urllib3.connection', 'requests.packages.urllib3.fields', 'requests.packages.urllib3.filepost', 'requests.packages.urllib3.packages.six.moves.urllib', 'requests.packages.urllib3.packages.six.moves.urllib.parse', 'requests.packages.urllib3.request', 'requests.packages.urllib3.response', 'requests.packages.urllib3.util.queue', 'requests.packages.urllib3.connectionpool', 'requests.packages.urllib3.poolmanager', 'requests.packages.urllib3', 'idna.package_data', 'idna.idnadata', 'idna.intranges', 'idna.core', 'idna', 'requests.packages.idna.package_data', 'requests.packages.idna.idnadata', 'requests.packages.idna.intranges', 'requests.packages.idna.core', 'requests.packages.idna', 'requests.packages.chardet', 'requests.packages', 'stringprep', 'encodings.idna', 'requests.hooks', 'requests.auth', 'requests.status_codes', 'requests.models', 'socks', 'urllib3.contrib.socks', 'requests.adapters', 'requests.sessions', 'requests.api', 'requests', 'determined.common.api.errors', 'determined.common.api.metric', 'shlex', 'webbrowser', 'lomond._version', 'six', 'six.moves', 'six.moves.urllib', 'six.moves.urllib.parse', 'lomond.constants', 'lomond.errors', 'lomond.events', 'lomond.compression', 'lomond.mask', 'lomond.opcode', 'lomond.frame', 'lomond.extension', 'lomond.response', 'lomond.parser', 'lomond.utf8validator', 'lomond.frame_parser', 'lomond.message', 'lomond.stream', 'lomond.proxy', 'lomond.selectors', 'lomond.session', 'lomond.status', 'lomond.websocket', 'lomond', 'determined.common.requests', 'determined.common.api.request', 'uuid', 'termcolor', 'tarfile', 'pathspec.compat', 'pathspec.pattern', 'pathspec.util', 'pathspec.pathspec', 'pathspec._meta', 'pathspec.patterns.gitwildmatch', 'pathspec.patterns', 'pathspec', 'determined.common.check', 'determined.common.context', 'determined.common.api.bindings', 'determined.common.api.logs', 'dataclasses', 'determined.common.storage.base', 'determined.common.storage.cloud', 'determined.errors', 'determined.common.storage.azure', 'determined.common.storage.gcs', 'six.moves.queue', 'hdfs.util', 'multiprocessing.process', '_compat_pickle', '_pickle', 'pickle', 'array', 'multiprocessing.reduction', 'multiprocessing.context', 'mp_main', 'multiprocessing', 'multiprocessing.util', '_multiprocessing', 'multiprocessing.connection', 'multiprocessing.pool', 'hdfs.client', 'imp', 'logging.handlers', 'configparser', 'six.moves.configparser', 'hdfs.config', 'hdfs', 'determined.common.storage.hdfs', 'determined.common.storage.s3', 'determined.common.storage.shared', 'determined.common.storage', 'determined.common.experimental.session', 'determined.common.experimental.checkpoint._checkpoint', 'determined.common.experimental.checkpoint', 'determined.common.experimental.trial', 'determined.common.experimental.experiment', 'determined.common.experimental.model', 'determined.common.experimental.determined', 'determined.common.experimental', 'determined.common.api.experiment', 'backoff._common', 'backoff._jitter', 'backoff._sync', 'backoff._decorator', 'backoff._wait_gen', 'backoff', 'concurrent', 'concurrent.futures._base', 'concurrent.futures', 'asyncio.constants', 'asyncio.format_helpers', 'asyncio.base_futures', 'asyncio.log', 'asyncio.coroutines', '_contextvars', 'contextvars', 'asyncio.exceptions', 'asyncio.base_tasks', '_asyncio', 'asyncio.events', 'asyncio.futures', 'asyncio.protocols', 'asyncio.transports', 'asyncio.sslproto', 'asyncio.locks', 'asyncio.tasks', 'asyncio.staggered', 'asyncio.trsock', 'asyncio.base_events', 'asyncio.runners', 'asyncio.queues', 'asyncio.streams', 'asyncio.subprocess', 'asyncio.base_subprocess', 'asyncio.selector_events', 'asyncio.unix_events', 'asyncio', 'determined.common.api.profiler', 'determined.common.api', 'determined.common._logging', 'determined.common', 'determined.ipc', 'determined.core._distributed', 'determined.tensorboard.metric_writers.util', 'determined.tensorboard.metric_writers.callback', 'determined.tensorboard.metric_writers', 'determined.tensorboard.base', 'determined.tensorboard.azure', 'determined.tensorboard.gcs', 'determined.tensorboard.hdfs', 'determined.tensorboard.s3', 'determined.tensorboard.shared', 'determined.tensorboard.build', 'determined.tensorboard.util', 'determined.tensorboard', 'determined.core._checkpoint', 'determined.core._train', 'determined.core._searcher', 'determined.core._preempt', 'appdirs', 'determined.core._context', 'determined.core', 'determined._env_context', 'determined._trial_context', 'determined._trial', 'determined.profiler', 'determined.workload', 'determined._trial_controller', 'determined.load', 'determined._execution', 'determined._hparam', 'numbers', 'determined.util', 'determined', 'determined.exec', 'determined.horovod', 'analytics.version', 'dateutil._version', 'dateutil', 'dateutil.tz._common', 'dateutil.tz._factories', 'dateutil.tz.tz', 'dateutil.tz', '_decimal', 'decimal', 'analytics.utils', 'monotonic', 'gzip', 'analytics.request', 'analytics.consumer', 'analytics.client', 'analytics', 'determined.common.api.analytics', 'model_hub', 'model_hub.mmdetection', 'model_hub.mmdetection._data', 'numpy._globals', 'numpy.config__', 'numpy._distributor_init', 'numpy._version', 'numpy.version', 'numpy.core._multiarray_umath', 'numpy.compat._inspect', 'numpy.compat.py3k', 'numpy.compat', 'numpy.core.overrides', 'numpy.core.multiarray', 'numpy.core.umath', 'numpy.core._string_helpers', 'numpy.core._dtype', 'numpy.core._type_aliases', 'numpy.core.numerictypes', 'numpy.core._exceptions', 'numpy.core._methods', 'numpy.core.fromnumeric', 'numpy.core.shape_base', 'numpy.core._ufunc_config', 'numpy.core.arrayprint', 'numpy.core._asarray', 'numpy.core.numeric', 'numpy.core.defchararray', 'numpy.core.records', 'numpy.core.memmap', 'numpy.core.function_base', 'numpy.core._machar', 'numpy.core.getlimits', 'numpy.core.einsumfunc', 'numpy.core._multiarray_tests', 'numpy.core._add_newdocs', 'numpy.core._add_newdocs_scalars', 'numpy.core._dtype_ctypes', '_ast', 'ast', '_ctypes', 'ctypes._endian', 'ctypes', 'numpy.core._internal', 'numpy._pytesttester', 'numpy.core', 'numpy.lib.mixins', 'numpy.lib.ufunclike', 'numpy.lib.type_check', 'numpy.lib.scimath', 'numpy.lib.stride_tricks', 'numpy.lib.twodim_base', 'numpy.linalg._umath_linalg', 'numpy.linalg.linalg', 'numpy.linalg', 'numpy.matrixlib.defmatrix', 'numpy.matrixlib', 'numpy.lib.histograms', 'numpy.lib.function_base', 'numpy.lib.index_tricks', 'numpy.lib.nanfunctions', 'numpy.lib.shape_base', 'numpy.lib.polynomial', 'numpy.lib.utils', 'numpy.lib.arraysetops', 'numpy.lib.format', 'numpy.lib._datasource', 'numpy.lib._iotools', 'numpy.lib.npyio', 'numpy.lib.arrayterator', 'numpy.lib.arraypad', 'numpy.lib._version', 'numpy.lib', 'numpy.fft._pocketfft_internal', 'numpy.fft._pocketfft', 'numpy.fft.helper', 'numpy.fft', 'numpy.polynomial.polyutils', 'numpy.polynomial._polybase', 'numpy.polynomial.polynomial', 'numpy.polynomial.chebyshev', 'numpy.polynomial.legendre', 'numpy.polynomial.hermite', 'numpy.polynomial.hermite_e', 'numpy.polynomial.laguerre', 'numpy.polynomial', '_cython_0_29_24', 'numpy.random._common', 'secrets', 'numpy.random.bit_generator', 'numpy.random._bounded_integers', 'numpy.random._mt19937', 'numpy.random.mtrand', 'numpy.random._philox', 'numpy.random._pcg64', 'numpy.random._sfc64', 'numpy.random._generator', 'numpy.random._pickle', 'numpy.random', 'numpy.ctypeslib', 'numpy.ma.core', 'numpy.ma.extras', 'numpy.ma', 'numpy', 'mmcv.arraymisc.quantization', 'mmcv.arraymisc', 'addict.addict', 'addict', 'yapf.yapflib', 'lib2to3', 'lib2to3.pgen2', 'lib2to3.pgen2.token', 'lib2to3.pgen2.tokenize', 'yapf.yapflib.errors', 'yapf.yapflib.py3compat', 'yapf.yapflib.style', 'yapf.yapflib.file_resources', 'difflib', 'lib2to3.pgen2.parse', 'lib2to3.pgen2.grammar', 'lib2to3.pgen2.pgen', 'lib2to3.pgen2.driver', 'lib2to3.pytree', 'lib2to3.pygram', 'yapf.yapflib.pytree_utils', 'yapf.yapflib.pytree_visitor', 'yapf.yapflib.blank_line_calculator', 'yapf.yapflib.comment_splicer', 'yapf.yapflib.subtypes', 'yapf.yapflib.format_token', 'yapf.yapflib.continuation_splicer', 'yapf.yapflib.identify_container', 'yapf.yapflib.split_penalty', 'lib2to3.pgen2.literals', 'lib2to3.patcomp', 'lib2to3.fixer_util', 'yapf.yapflib.logical_line', 'yapf.yapflib.object_state', 'yapf.yapflib.pytree_unwrapper', 'yapf.yapflib.format_decision_state', 'yapf.yapflib.line_joiner', 'yapf.yapflib.verifier', 'yapf.yapflib.reformatter', 'yapf.yapflib.subtype_assigner', 'yapf.yapflib.yapf_api', 'yapf', 'mmcv.utils.misc', 'mmcv.utils.path', 'mmcv.utils.config', 'mmcv.utils.timer', 'mmcv.utils.progressbar', 'unittest.util', 'unittest.result', 'pprint', 'unittest.case', 'unittest.async_case', 'unittest.suite', 'unittest.loader', 'unittest.signals', 'unittest.runner', 'unittest.main', 'unittest', 'unittest.mock', 'mmcv.utils.testing', 'packaging.about', 'packaging', 'packaging._structures', 'packaging.version', 'mmcv.utils.version_utils', 'torch.autocast_mode', 'torch._utils', 'torch._utils_internal', 'xml', 'xml.parsers', 'pyexpat.errors', 'pyexpat.model', 'pyexpat', 'xml.parsers.expat.model', 'xml.parsers.expat.errors', 'xml.parsers.expat', 'plistlib', 'pkg_resources.extern', 'pkg_resources._vendor', 'pkg_resources._vendor.appdirs', 'pkg_resources.extern.appdirs', 'pkg_resources._vendor.packaging.about__', 'pkg_resources._vendor.packaging', 'pkg_resources.extern.packaging', 'pkg_resources.extern.packaging._structures', 'pkg_resources.extern.packaging.version', 'sysconfig', 'pkg_resources._vendor.packaging._manylinux', 'pkg_resources._vendor.packaging._musllinux', 'pkg_resources.extern.packaging.tags', 'pkg_resources.extern.packaging.utils', 'pkg_resources.extern.packaging.specifiers', 'pkg_resources._vendor.pyparsing', 'pkg_resources.extern.pyparsing', 'pkg_resources.extern.packaging.markers', 'pkg_resources.extern.packaging.requirements', 'pkg_resources', 'torch.version', 'torch.torch_version', 'torch._six', 'torch._C._onnx', 'torch._C._jit', 'torch._C._jit_tree_views', 'torch._C._te', 'torch._C.cpp', 'torch._C.cpp.nn', 'torch._C._cudart', 'torch._C._nvtx', 'torch._C._cudnn', 'torch._C', 'torch._C._fft', 'torch._C._fx', 'torch._C._linalg', 'torch._C._nn', 'torch._C._special', 'torch._namedtensor_internals', 'torch.overrides', 'torch.utils.throughput_benchmark', 'torch.utils._crash_handler', 'torch.utils', 'torch.utils.hooks', 'torch._tensor', 'torch.storage', 'torch.random', 'torch._sources', 'torch.types', 'torch.serialization', 'torch._tensor_str', 'torch.cuda._utils', 'gc', 'torch.cuda.graphs', 'torch.cuda.streams', 'torch.cuda.memory', 'torch.cuda.random [2022-05-31T14:24:23.474220Z] dc5b74e6 || ', 'torch.cuda.sparse', 'torch.cuda.profiler', 'torch.cuda.nvtx', 'torch.cuda.amp.autocast_mode', 'torch.cuda.amp.common', 'torch.cuda.amp.grad_scaler', 'torch.cuda.amp', 'torch.cuda', 'torch.sparse', 'torch.nn.parameter', 'torch.nn.modules.module', 'torch._VF', 'torch._torch_docs', 'torch._C._distributed_c10d', 'torch.distributed.constants', 'torch.distributed.rendezvous', 'torch.distributed.distributed_c10d', 'torch.distributed.remote_device', 'torch.distributed', 'torch._C._distributed_rpc', 'torch.futures', 'torch.distributed.rpc.internal', 'torch.distributed.rpc.constants', 'torch.distributed.rpc.api', 'torch.distributed.rpc.backend_registry', 'torch.distributed.rpc.functions', 'torch._C._distributed_autograd', 'torch.distributed.autograd', 'torch.distributed.rpc.options', 'torch.autograd.variable', 'torch.autograd.function', 'cmath', 'torch.testing._core', 'torch.testing._asserts', 'torch.testing._creation', 'torch.testing._check_kernel_launches', 'torch.testing._dtype_getters', 'torch.testing._deprecated', 'torch.testing', 'torch.utils._pytree', 'torch._vmap_internals', 'torch.autograd.gradcheck', 'torch.autograd.grad_mode', 'torch.autograd.anomaly_mode', 'torch.autograd.functional', 'torch.autograd.forward_ad', 'torch.autograd.graph', 'torch._C._autograd', 'torch.autograd.profiler_util', 'torch.autograd.profiler', 'torch.autograd', 'torch.autograd.profiler_legacy', 'torch.distributed.rpc.server_process_global_profiler', 'torch.distributed.rpc', 'torch.package.analyze.trace_dependencies', 'torch.package.analyze', 'torch.package._mangling', 'torch.package.analyze.is_from_package', 'torch.package.glob_group', 'torch.package.file_structure_representation', 'torch.package.importer', 'pickletools', 'torch.package._digraph', 'torch.package._importlib', 'torch.package._package_pickler', 'torch.package._stdlib', 'torch.package.find_file_dependencies', 'torch.package.package_exporter', 'torch.package._directory_reader', 'torch.package._package_unpickler', 'torch.package.package_importer', 'torch.package', 'torch._jit_internal', 'torch.nn._reduction', 'torch.nn.modules.utils', 'torch.nn.grad', 'torch.nn.functional', 'torch.nn.init', 'typing_extensions', 'torch.nn.modules.lazy', 'torch.nn.modules.linear', 'torch.nn.common_types', 'torch.nn.modules.conv', 'torch.nn.modules.activation', 'torch.nn.modules.distance', 'torch.nn.modules.loss', 'torch.nn.modules.container', 'torch.nn.modules.pooling', 'torch.nn.modules._functions', 'torch.nn.modules.batchnorm', 'torch.nn.modules.instancenorm', 'torch.nn.modules.normalization', 'torch.nn.modules.dropout', 'torch.nn.modules.padding', 'torch.nn.modules.sparse', 'torch.nn.utils.rnn', 'torch.nn.utils.clip_grad', 'torch.nn.utils.weight_norm', 'torch.nn.utils.convert_parameters', 'torch.nn.utils.spectral_norm', 'torch.nn.utils.fusion', 'torch.nn.utils.memory_format', 'torch.nn.utils.parametrize', 'torch.nn.utils.parametrizations', 'torch.nn.utils.init', 'torch.nn.utils', 'torch.nn.modules.rnn', 'torch.nn.modules.pixelshuffle', 'torch.nn.modules.upsampling', 'torch.nn.modules.fold', 'torch.nn.modules.adaptive', 'torch.nn.modules.transformer', 'torch.nn.modules.flatten', 'torch.nn.modules.channelshuffle', 'torch.nn.modules', 'torch.nn.parallel.parallel_apply', 'torch.cuda.nccl', 'torch.nn.parallel.comm', 'torch.nn.parallel.replicate', 'torch.nn.parallel._functions', 'torch.nn.parallel.scatter_gather', 'torch.nn.parallel.data_parallel', 'torch.distributed.algorithms.join', 'torch.distributed.algorithms', 'torch.nn.parallel.distributed', 'torch.nn.parallel', 'torch.nn', 'torch._linalg_utils', 'torch._lowrank', 'torch.functional', 'torch.cpu.amp.autocast_mode', 'torch.cpu.amp', 'torch.cpu', 'torch.fft', 'torch.nn.intrinsic.modules.fused', 'torch.nn.intrinsic.modules', 'torch.nn.intrinsic', 'torch.nn.quantized.modules.utils', 'torch.jit._monkeytype_config', 'torch.jit._state', 'torch.jit.annotations', 'astunparse.unparser', 'astunparse.printer', 'astunparse', 'torch.jit.frontend', 'torch.backends', 'torch.backends.cudnn', 'torch.jit._builtins', 'torch.jit._check', 'torch.jit._recursive', 'torch.jit._fuser', 'torch.jit._serialization', 'torch._classes', 'torch.jit._script', 'torch.jit._trace', 'torch.jit._async', 'torch.jit._freeze', 'torch.jit', 'torch.nn.quantized.functional', 'torch.nn.quantized.modules.activation', 'torch.nn.quantized.modules.batchnorm', 'torch.nn.quantized.modules.normalization', 'torch.nn.qat.modules.linear', 'torch.nn.qat.modules.conv', 'torch.nn.qat.modules', 'torch.nn.qat', 'torch.nn.intrinsic.qat.modules.linear_relu', 'torch.nn.intrinsic.qat.modules.conv_fused', 'torch.nn.intrinsic.qat.modules', 'torch.nn.intrinsic.qat', 'torch._ops', 'torch.nn.quantized.modules.conv', 'torch.nn.quantized.modules.linear', 'torch.nn.quantized.modules.embedding_ops', 'torch.nn.quantized.modules.functional_modules', 'torch.nn.quantized.modules', 'torch.nn.quantized', 'torch.nn.quantizable.modules.activation', 'torch.nn.quantizable.modules.rnn', 'torch.nn.quantizable.modules', 'torch.nn.quantizable', 'torch.ao.nn.sparse.quantized.utils', 'torch.ao.nn.sparse.quantized.linear', 'torch.ao.nn.sparse.quantized.dynamic.linear', 'torch.ao.nn.sparse.quantized.dynamic', 'torch.ao.nn.sparse.quantized', 'torch.ao.nn.sparse', 'torch.ao.nn', 'torch.ao.sparsity._mappings', 'torch.ao.sparsity.sparsifier', 'torch.ao.sparsity.sparsifier.utils', 'torch.ao.sparsity.sparsifier.base_sparsifier', 'torch.ao.sparsity.sparsifier.weight_norm_sparsifier', 'torch.ao.sparsity.scheduler', 'torch.ao.sparsity.scheduler.base_scheduler', 'torch.ao.sparsity.scheduler.lambda_scheduler', 'torch.ao.sparsity.experimental', 'torch.ao.sparsity.experimental.pruner', 'torch.ao.sparsity.experimental.pruner.parametrization', 'torch.ao.sparsity.experimental.pruner.base_pruner', 'torch.ao.sparsity', 'torch.ao', 'torch.optim._functional', 'torch.optim.optimizer', 'torch.optim.adadelta', 'torch.optim.adagrad', 'torch.optim.adam', 'torch.optim.adamw', 'torch.optim.sparse_adam', 'torch.optim.adamax', 'torch.optim.asgd', 'torch.optim.sgd', 'torch.optim.radam', 'torch.optim.rprop', 'torch.optim.rmsprop', 'torch.optim.nadam', 'torch.optim.lbfgs', 'torch.optim.lr_scheduler', 'torch.optim.swa_utils', 'torch.optim', 'torch.optim._multi_tensor.adam', 'torch.optim._multi_tensor.adamw', 'torch.optim._multi_tensor._functional', 'torch.optim._multi_tensor.nadam', 'torch.optim._multi_tensor.sgd', 'torch.optim._multi_tensor.radam', 'torch.optim._multi_tensor.rmsprop', 'torch.optim._multi_tensor.rprop', 'torch.optim._multi_tensor.asgd', 'torch.optim._multi_tensor.adamax', 'torch.optim._multi_tensor.adadelta', 'torch.optim._multi_tensor.adagrad', 'torch.optim._multi_tensor', 'multiprocessing.resource_sharer', 'torch.multiprocessing.reductions', 'torch.multiprocessing.spawn', 'torch.multiprocessing', 'torch.special', 'torch.utils.backcompat', 'torch.onnx', 'torch.linalg', 'tqdm._monitor', 'tqdm._tqdm_pandas', 'tqdm.utils', 'tqdm.std', 'tqdm._dist_ver', 'tqdm.version', 'tqdm.cli', 'tqdm.gui', 'tqdm', 'tqdm.autonotebook', 'tqdm.asyncio', 'tqdm.auto', 'torch.hub', 'torch.distributions.constraints', 'torch.distributions.utils', 'torch.distributions.distribution', 'torch.distributions.exp_family', 'torch.distributions.bernoulli', 'torch.distributions.dirichlet', 'torch.distributions.beta', 'torch.distributions.binomial', 'torch.distributions.categorical', 'torch.distributions.cauchy', 'torch.distributions.gamma', 'torch.distributions.chi2', 'torch.distributions.transforms', 'torch.distributions.constraint_registry', 'torch.distributions.continuous_bernoulli', 'torch.distributions.exponential', 'torch.distributions.fishersnedecor', 'torch.distributions.geometric', 'torch.distributions.uniform', 'torch.distributions.independent', 'torch.distributions.transformed_distribution', 'torch.distributions.gumbel', 'torch.distributions.half_cauchy', 'torch.distributions.normal', 'torch.distributions.half_normal', 'torch.distributions.laplace', 'torch.distributions.multivariate_normal', 'torch.distributions.lowrank_multivariate_normal', 'torch.distributions.one_hot_categorical', 'torch.distributions.pareto', 'torch.distributions.poisson', 'torch.distributions.kl', 'torch.distributions.kumaraswamy', 'torch.distributions.lkj_cholesky', 'torch.distributions.log_normal', 'torch.distributions.logistic_normal', 'torch.distributions.mixture_same_family', 'torch.distributions.multinomial', 'torch.distributions.negative_binomial', 'torch.distributions.relaxed_bernoulli', 'torch.distributions.relaxed_categorical', 'torch.distributions.studentT', 'torch.distributions.von_mises', 'torch.distributions.weibull', 'torch.distributions', 'torch.backends.cuda', 'torch.backends.mkl', 'torch.backends.mkldnn', 'torch.backends.openmp', 'torch.backends.quantized', 'torch.ao.quantization.quant_type', 'torch.ao.quantization.utils', 'torch.ao.quantization.observer', 'torch.quantization.observer', 'torch.ao.quantization.fake_quantize', 'torch.ao.quantization.fuser_method_mappings', 'torch.quantization.fuser_method_mappings', 'torch.ao.quantization.fuse_modules', 'torch.nn.intrinsic.quantized.modules.linear_relu', 'torch.nn.intrinsic.quantized.modules.conv_relu', 'torch.nn.intrinsic.quantized.modules.bn_relu', 'torch.nn.intrinsic.quantized.modules', 'torch.nn.intrinsic.quantized', 'torch.nn.quantized.dynamic.modules.linear', 'torch.nn.quantized.dynamic.modules.rnn', 'torch.nn.quantized.dynamic.modules', 'torch.nn.quantized.dynamic', 'torch.nn.intrinsic.quantized.dynamic.modules.linear_relu', 'torch.nn.intrinsic.quantized.dynamic.modules', 'torch.nn.intrinsic.quantized.dynamic', 'torch.nn.quantized._reference.modules.utils', 'torch.nn.quantized._reference.modules.linear', 'torch.nn.quantized._reference.modules.conv', 'torch.nn.quantized._reference.modules', 'torch.nn.quantized._reference', 'torch.ao.quantization.stubs', 'torch.ao.quantization.quantization_mappings', 'torch.quantization.quantization_mappings', 'torch.ao.quantization.qconfig', 'torch.quantization.qconfig', 'torch.ao.quantization.quantize', 'torch.ao.quantization', 'torch.quantization.quantize', 'torch.quantization.fake_quantize', 'torch.quantization.fuse_modules', 'torch.quantization.stubs', 'torch.quantization.quant_type', 'torch.ao.quantization.quantize_jit', 'torch.quantization.quantize_jit', 'torch.quantization', 'torch.utils.data.sampler', 'torch.utils.data._typing', 'torch.utils.data.dataset', 'torch.utils.data._utils.signal_handling', 'torch.utils.data._utils.worker', 'torch.utils.data._utils.pin_memory', 'torch.utils.data._utils.collate', 'torch.utils.data._utils.fetch', 'torch.utils.data._utils', 'torch.utils.data.dataloader', 'torch.utils.data.distributed', 'torch.utils.data._decorator', 'torch.utils.data.graph', 'torch.utils.data.sharding', 'torch.utils.data.backward_compatibility', 'torch.utils.data.communication.eventloop', 'torch.utils.data.communication.iter', 'torch.utils.data.communication.messages', 'torch.utils.data.communication.protocol', 'torch.utils.data.communication.queue', 'torch.utils.data.communication', 'dill.info', '_pyio', 'dill.settings', 'dill._dill', 'dill.source', 'dill.temp', 'dill.pointers', 'dill.detect', 'dill.objtypes', 'dill', 'torch.utils.data.datapipes.iter.callable', 'torch.utils.data.datapipes.iter.combinatorics', 'torch.utils.data.datapipes.iter.combining', 'torch.utils.data.datapipes.utils', 'torch.utils.data.datapipes.utils.common', 'torch.utils.data.datapipes.iter.filelister', 'torch.utils.data.datapipes.iter.fileloader', 'torch.utils.data.datapipes.iter.grouping', 'torch.utils.data.datapipes.iter.httpreader', 'torch.utils.data.datapipes.iter.linereader', 'torch.utils.data.datapipes.utils.decoder', 'torch.utils.data.datapipes.iter.routeddecoder', 'pytz.exceptions', 'pytz.lazy', 'pytz.tzinfo', 'pytz.tzfile', 'pytz', 'pandas._typing', 'pandas._libs.tslibs.dtypes', 'pandas._libs.tslibs.base', 'pandas._libs.tslibs.np_datetime', 'pandas._libs.tslibs.nattype', 'dateutil.zoneinfo', 'pandas._libs.tslibs.timezones', 'pandas._libs.tslibs.ccalendar', 'pandas._libs.tslibs.tzconversion', 'pandas._config.config', 'pandas._config.dates', 'pandas._config.display', 'pandas._config', 'pandas._config.localization', 'pandas._libs.tslibs.strptime', 'pandas._libs.tslibs.fields', 'pandas._libs.tslibs.timedeltas', 'pandas._libs.tslibs.timestamps', 'dateutil.easter', 'dateutil._common', 'dateutil.relativedelta', 'pandas._libs.properties', 'pandas._libs.tslibs.offsets', 'dateutil.parser._parser', 'dateutil.parser.isoparser', 'dateutil.parser', 'pandas._libs.tslibs.parsing', 'pandas._libs.tslibs.conversion', 'pandas._libs.tslibs.period', 'pandas._libs.tslibs.vectorized', 'pandas._libs.tslibs', 'pandas._libs.ops_dispatch', 'pandas._libs.missing', 'pandas._libs.hashtable', 'pandas._libs.algos', 'pandas._libs.interval', 'pandas._libs', 'pandas.util._decorators', 'pandas.core', 'pandas.core.util', 'pandas._libs.tslib', 'pandas._libs.lib', 'pandas._libs.hashing', 'pandas.core.dtypes', 'pandas.util._exceptions', 'pandas.errors', 'pandas.core.dtypes.generic', 'pandas.core.dtypes.base', 'pandas.core.dtypes.inference', 'pandas.core.dtypes.dtypes', 'pandas.core.dtypes.common', 'pandas.core.util.hashing', 'pandas.util', 'pandas.util.version', 'pandas.compat.numpy', 'pandas.compat.pyarrow', 'pandas.compat', 'pandas.core.config_init', 'pandas.core.dtypes.missing', 'pandas.util._validators', 'pandas.core.dtypes.cast', 'pandas.core.dtypes.concat', 'pandas.core.array_algos', 'pandas.core.common', 'pandas.core.construction', 'pandas.core.array_algos.take', 'pandas.core.indexers.utils', 'pandas.core.indexers', 'pandas.core.algorithms', 'pandas.compat.numpy.function', 'pandas.core.roperator', 'pandas._libs.ops', 'pandas.core.computation', 'pandas.compat._optional', 'pandas.core.computation.check', 'pandas.core.computation.expressions', 'pandas.core.ops.missing', 'pandas.core.ops.dispatch', 'pandas.core.ops.invalid', 'pandas.core.ops.array_ops', 'pandas.core.ops.common', 'pandas.core.ops.docstrings', 'pandas.core.ops.mask_ops', 'pandas.core.ops.methods', 'pandas.core.ops', 'pandas.core.arraylike', 'pandas.core.missing', 'pandas.core.array_algos.quantile', 'pandas.core.sorting', 'pandas.core.arrays.base', 'pandas.core.nanops', 'pandas.core.array_algos.masked_reductions', 'pandas.core.arrays.masked', 'pandas.core.arrays.boolean', 'pandas._libs.arrays', 'pandas.core.accessor', 'pandas.core.array_algos.transforms', 'pandas.core.arrays._mixins', 'pandas.core.base', 'pandas.core.strings.accessor', 'pandas.core.strings.base', 'pandas.core.strings', 'pandas.core.strings.object_array', 'pandas.io', 'pandas.io.formats', 'pandas.io.formats.console', 'pandas.core.arrays.categorical', 'pandas.tseries', 'pandas.tseries.frequencies', 'pandas.core.arrays.datetimelike', 'pandas.core.arrays._ranges', 'pandas.core.arrays.numeric', 'pandas.core.tools', 'pandas.core.tools.numeric', 'pandas.core.arrays.integer', 'pandas.tseries.offsets', 'pandas.core.arrays.datetimes', 'pandas.core.arrays.floating', 'pandas.core.indexes', 'pandas._libs.index', 'pandas._libs.join', 'pandas.core.array_algos.putmask', 'pandas.libs.sparse', 'pandas.core.arrays.sparse.dtype', 'pandas.io.formats.printing', 'pandas.core.arrays.sparse.array', 'pandas.core.arrays.sparse.accessor', 'pandas.core.arrays.sparse', 'pandas.core.indexes.frozen', 'pandas.core.indexes.base', 'pandas.core.arrays.interval', 'pandas.core.arrays.numpy', 'pandas.core.arrays.period', 'pandas.core.arrays.string_', 'pandas.core.arrays.string_arrow', 'pandas.core.arrays.timedeltas', 'pandas.core.arrays', 'pandas.core.flags', 'pandas._libs.reduction', 'pandas.core.apply', 'pandas._libs.indexing', 'pandas.core.indexes.extension', 'pandas.core.indexes.category', 'pandas.core.indexes.numeric', 'pandas.core.indexes.range', 'pandas.core.tools.timedeltas', 'pandas.core.indexes.datetimelike', 'pandas.core.tools.times', 'pandas.core.indexes.datetimes', 'pandas.core.indexes.multi', 'pandas.core.indexes.timedeltas', 'pandas.core.indexes.interval', 'pandas.core.indexes.period', 'pandas.core.indexes.api', 'pandas.core.indexing', 'pandas.core.array_algos.replace', 'pandas.core.reshape', 'pandas._libs.internals', 'pandas._libs.writers', 'pandas.core.internals.blocks', 'pandas.core.internals.api', 'pandas.core.internals.base', 'pandas.core.internals.array_manager', 'pandas.core.internals.ops', 'pandas.core.internals.managers', 'pandas.core.internals.concat', 'pandas.core.inte [2022-05-31T14:24:23.474220Z] dc5b74e6 || rnals', 'pandas.core.reshape.concat', 'mmap', 'pandas.core.shared_docs', 'pandas.io.common', 'pandas.io.formats.format', 'pandas.core.describe', 'pandas.core.internals.construction', 'pandas.core.sample', 'pandas._libs.window', 'pandas._libs.window.aggregations', 'pandas.libs.window.indexers', 'pandas.core.indexers.objects', 'pandas.core.util.numba', 'pandas.core.window.common', 'pandas.core.window.doc', 'pandas.core.window.numba_', 'pandas.core.window.online', 'pandas.core._numba', 'pandas.core._numba.executor', 'pandas.core.window.rolling', 'pandas.core.window.ewm', 'pandas.core.window.expanding', 'pandas.core.window', 'pandas.core.generic', 'pandas.core.reshape.util', 'pandas.core.reshape.melt', 'pandas._libs.reshape', 'pandas.core.indexes.accessors', 'pandas.arrays', 'pandas.core.tools.datetimes', 'pandas.io.formats.info', 'pandas.plotting._core', 'pandas.plotting._misc', 'pandas.plotting', 'pandas.core.series', 'pandas.core.frame', 'pandas.core.groupby.base', 'pandas.libs.groupby', 'pandas.core.groupby.numba', 'pandas.core.groupby.categorical', 'pandas.core.groupby.grouper', 'pandas.core.groupby.ops', 'pandas.core.groupby.indexing', 'pandas.core.groupby.groupby', 'pandas.core.groupby.generic', 'pandas.core.groupby', 'pandas.core.api', 'pandas.tseries.api', 'pandas.core.computation.common', 'pandas.core.computation.align', 'pandas.compat.chainmap', 'pandas.core.computation.scope', 'pandas.core.computation.ops', 'pandas.core.computation.parsing', 'pandas.core.computation.expr', 'pandas.core.computation.engines', 'pandas.core.computation.eval', 'pandas.core.computation.api', 'pandas.core.reshape.merge', 'pandas.core.reshape.pivot', 'pandas.core.reshape.reshape', 'pandas.core.reshape.tile', 'pandas.core.reshape.api', 'pandas.api.extensions', 'pandas.api.indexers', 'pandas.core.dtypes.api', 'pandas.api.types', 'pandas.api', 'pandas._testing._random', 'pandas._testing.contexts', 'pandas._testing._io', 'pandas._testing._warnings', 'pandas._libs.testing', 'pandas._testing.asserters', 'pandas._testing.compat', 'pandas._testing', 'pandas.testing', 'pandas.util._print_versions', 'pandas.io.clipboards', 'pandas._libs.parsers', 'pandas.io.excel._util', 'pandas.io.date_converters', 'pandas.io.parsers.base_parser', 'pandas.io.parsers.arrow_parser_wrapper', 'pandas.io.parsers.c_parser_wrapper', 'pandas.io.parsers.python_parser', 'pandas.io.parsers.readers', 'pandas.io.parsers', 'pandas.io.excel._odfreader', 'pandas.io.excel._openpyxl', 'pandas.io.excel._pyxlsb', 'pandas.io.excel._xlrd', 'pandas.io.excel._base', 'pandas._libs.json', 'pandas.io.formats._color_data', 'pandas.io.formats.css', 'pandas.io.formats.excel', 'pandas.io.excel._odswriter', 'pandas.io.excel._xlsxwriter', 'pandas.io.excel._xlwt', 'pandas.io.excel', 'pandas.io.feather_format', 'pandas.io.gbq', 'pandas.io.html', 'pandas.io.json._normalize', 'pandas.io.json._table_schema', 'pandas.io.json._json', 'pandas.io.json', 'pandas.io.orc', 'pandas.io.parquet', 'pandas.compat.pickle_compat', 'pandas.io.pickle', 'pandas.core.computation.pytables', 'pandas.io.pytables', 'pandas.io.sas.sasreader', 'pandas.io.sas', 'pandas.io.spss', 'pandas.io.sql', 'pandas.io.stata', 'pandas.io.xml', 'pandas.io.api', 'pandas.util._tester', 'pandas._version', 'pandas', 'torch.utils.data.datapipes.iter.selecting', 'torch.utils.data.datapipes.iter.streamreader', 'torch.utils.data.datapipes.iter.tararchivereader', 'torch.utils.data.datapipes.iter.ziparchivereader', 'torch.utils.data.datapipes.iter.utils', 'torch.utils.data.datapipes.iter', 'torch.utils.data.datapipes.map.callable', 'torch.utils.data.datapipes.map.combining', 'torch.utils.data.datapipes.map', 'torch.utils.data.datapipes.dataframe.structures', 'torch.utils.data.datapipes.dataframe.dataframes', 'torch.utils.data.datapipes.dataframe.datapipes', 'torch.utils.data.datapipes.dataframe', 'torch.utils.data.datapipes', 'torch.utils.data.dataloader_experimental', 'torch.utils.data', 'torch.config', 'torch.future', 'torch.profiler.profiler', 'torch.profiler', 'torch._tensor_docs', 'torch._storage_docs', 'torch.quasirandom', 'torch.multiprocessing._atfork', 'torch._lobpcg', 'torch.utils.dlpack', 'torch', 'cv2.Error', 'cv2.cuda', 'cv2.detail', 'cv2.dnn', 'cv2.fisheye', 'cv2.flann', 'cv2.gapi.core', 'cv2.gapi.core.cpu', 'cv2.gapi.core.fluid', 'cv2.gapi.core.ocl', 'cv2.gapi.ie', 'cv2.gapi.ie.detail', 'cv2.gapi.onnx', 'cv2.gapi.own', 'cv2.gapi.own.detail', 'cv2.gapi.render', 'cv2.gapi.render.ocv', 'cv2.gapi.streaming', 'cv2.gapi.video', 'cv2.gapi.wip', 'cv2.gapi.wip.draw', 'cv2.gapi.wip.gst', 'cv2.gapi.wip.onevpl', 'cv2.ipp', 'cv2.ml', 'cv2.ocl', 'cv2.ogl', 'cv2.parallel', 'cv2.samples', 'cv2.segmentation', 'cv2.utils.fs', 'cv2.videoio_registry', 'cv2.cv2', 'cv2.misc.version', 'cv2.version', 'cv2.load_config_py3', 'cv2.qt', 'cv2.gapi', 'cv2.utils', 'cv2.misc', 'cv2.data', 'cv2.mat_wrapper', 'cv2', '_distutils_hack', '_distutils_hack.override', 'distutils', 'distutils.debug', 'distutils.errors', 'getopt', 'distutils.fancy_getopt', 'distutils.dep_util', 'distutils.log', 'distutils.spawn', 'distutils.util', 'distutils.dist', 'distutils.dir_util', 'distutils.file_util', 'distutils.archive_util', 'distutils.cmd', 'distutils.config', 'distutils.extension', 'distutils.core', 'setuptools._deprecation_warning', 'setuptools.version', 'distutils.filelist', 'setuptools.monkey', 'setuptools.extension', 'distutils.command', 'distutils.version', 'setuptools.extern', 'setuptools._vendor', 'setuptools._vendor.packaging.about', 'setuptools._vendor.packaging', 'setuptools.extern.packaging', 'setuptools._vendor.ordered_set', 'setuptools.extern.ordered_set', 'concurrent.futures.thread', 'setuptools._vendor.more_itertools.recipes', 'setuptools._vendor.more_itertools.more', 'setuptools._vendor.more_itertools', 'setuptools.extern.more_itertools', 'distutils.command.bdist', 'setuptools.command', 'setuptools.windows_support', 'setuptools.extern.packaging._structures', 'setuptools.extern.packaging.version', 'setuptools._vendor.packaging._manylinux', 'setuptools._vendor.packaging._musllinux', 'setuptools.extern.packaging.tags', 'setuptools.extern.packaging.utils', 'setuptools.extern.packaging.specifiers', 'setuptools.config', 'setuptools.dist', 'setuptools.py34compat', 'setuptools._imp', 'setuptools.depends', 'distutils.ccompiler', 'setuptools.msvc', 'setuptools', 'torch._appdirs', 'torch.utils.file_baton', 'torch.utils._cpp_extension_versioner', 'torch.utils.hipify.version', 'torch.utils.hipify', 'torch.utils.hipify.constants', 'torch.utils.hipify.cuda_to_hip_mappings', 'torch.utils.hipify.hipify_python', 'distutils.sysconfig', 'distutils.command.build_ext', 'cython.parallel', 'Cython.Shadow', 'Cython', '_sysconfigdatalinux_x86_64-linux-gnu', 'Cython.Distutils.old_build_ext', 'Cython.Distutils.build_ext', 'Cython.Distutils.extension', 'Cython.Distutils', 'Cython.Compiler', 'Cython.Utils', 'Cython.Compiler.DebugFlags', 'Cython.Compiler.Options', 'Cython.Compiler.Errors', 'Cython.Compiler.StringEncoding', '_cython_0_29_30', 'Cython.Plex.Actions', 'Cython.Plex.Transitions', 'Cython.Plex.Machines', 'Cython.Plex.DFA', 'Cython.Plex.Errors', 'Cython.Plex.Regexps', 'Cython.Plex.Lexicons', 'Cython.Plex.Scanners', 'Cython.Plex', 'Cython.Compiler.Lexicon', 'Cython.Compiler.Future', 'Cython.Compiler.Scanning', 'Cython.Compiler.Naming', 'cython', 'Cython.Compiler.Version', 'Cython.StringIOTree', 'Cython.Compiler.Code', 'Cython.Compiler.PyrexTypes', 'Cython.Compiler.TypeSlots', 'Cython.Compiler.Symtab', 'Cython.Compiler.Main', 'setuptools.command.build_ext', 'torch.utils.cpp_extension', 'mmcv.utils.parrots_wrapper', 'mmcv.utils.env', 'torch.utils.model_zoo', 'mmcv.utils.hub', 'mmcv.utils.logging', 'mmcv.utils.parrots_jit', 'mmcv.utils.registry', 'mmcv.utils.trace', 'mmcv.utils', 'mmcv.fileio.file_client', 'mmcv.fileio.handlers.base', 'mmcv.fileio.handlers.json_handler', 'mmcv.fileio.handlers.pickle_handler', 'yaml.error', 'yaml.tokens', 'yaml.events', 'yaml.nodes', 'yaml.reader', 'yaml.scanner', 'yaml.parser', 'yaml.composer', 'yaml.constructor', 'yaml.resolver', 'yaml.loader', 'yaml.emitter', 'yaml.serializer', 'yaml.representer', 'yaml.dumper', 'yaml._yaml', 'yaml.cyaml', 'yaml', 'mmcv.fileio.handlers.yaml_handler', 'mmcv.fileio.handlers', 'mmcv.fileio.io', 'mmcv.fileio.parse', 'mmcv.fileio', 'mmcv.image.colorspace', 'PIL._version', 'PIL', 'defusedxml.common', 'defusedxml', 'xml.etree', 'xml.etree.ElementPath', '_elementtree', 'xml.etree.ElementTree', 'defusedxml.ElementTree', 'PIL.ImageMode', 'PIL.TiffTags', 'PIL._binary', 'PIL._util', 'PIL._imaging', 'cffi.lock', 'cffi.error', 'cffi.model', 'cffi.api', 'cffi', 'PIL.Image', 'PIL.ImageOps', 'mmcv.image.io', 'mmcv.image.geometric', 'mmcv.image.misc', 'mmcv.image.photometric', 'mmcv.image', 'mmcv.version', 'mmcv.video.io', 'mmcv.video.optflow', 'mmcv.video.processing', 'mmcv.video', 'mmcv.visualization.color', 'mmcv.visualization.image', 'mmcv.visualization.optflow', 'mmcv.visualization', 'mmcv', 'mmcv.parallel.data_container', 'mmcv.parallel.collate', 'mmcv.parallel._functions', 'mmcv.parallel.scatter_gather', 'mmcv.parallel.data_parallel', 'mmcv.parallel.distributed', 'mmcv.parallel.registry', 'mmcv.parallel.utils', 'mmcv.parallel', 'mmdet.version', 'mmdet', 'mmdet.datasets', 'mmcv.runner.dist_utils', 'mmcv.runner.base_module', 'torchvision._internally_replaced_utils', 'torchvision.extension', 'torchvision.models.alexnet', 'torchvision.models.resnet', 'torchvision.models.vgg', 'torchvision.models.squeezenet', 'torchvision.models.inception', 'torch.utils.checkpoint', 'torchvision.models.densenet', 'torchvision.models.googlenet', 'torchvision.ops._box_convert', 'torchvision.ops.boxes', 'torchvision.ops.deform_conv', 'torchvision.ops._utils', 'torchvision.ops.roi_align', 'torchvision.ops.roi_pool', 'torchvision.ops.ps_roi_align', 'torchvision.ops.ps_roi_pool', 'torchvision.ops.poolers', 'torchvision.ops.feature_pyramid_network', 'torchvision.ops.focal_loss', 'torch.fx._compatibility', 'torch.fx.immutable_collections', 'torch.fx.operator_schemas', 'torch.fx.node', 'torch.fx._pytree', 'torch.fx.graph', 'torch.fx.graph_module', 'torch.fx.proxy', 'torch.fx._symbolic_trace', 'torch.fx.interpreter', 'torch.fx.subgraph_rewriter', 'torch.fx', 'torchvision.ops.stochastic_depth', 'torchvision.ops._register_onnx_ops', 'torch.onnx.utils', 'torch.onnx.symbolic_helper', 'torch.onnx.symbolic_opset9', 'torch.onnx.symbolic_opset11', 'torch.onnx.symbolic_opset7', 'torch.onnx.symbolic_opset8', 'torch.onnx.symbolic_opset10', 'torch.onnx.symbolic_opset12', 'torch.onnx.symbolic_opset13', 'torch.onnx.symbolic_opset14', 'torch.onnx.symbolic_registry', 'torchvision.ops', 'torchvision.ops.misc', 'torchvision.models._utils', 'torchvision.models.mobilenetv2', 'torchvision.models.mobilenetv3', 'torchvision.models.mobilenet', 'torchvision.models.mnasnet', 'torchvision.models.shufflenetv2', 'torchvision.models.efficientnet', 'torchvision.models.regnet', 'torchvision.models.segmentation._utils', 'torchvision.models.segmentation.deeplabv3', 'torchvision.models.segmentation.fcn', 'torchvision.models.segmentation.lraspp', 'torchvision.models.segmentation.segmentation', 'torchvision.models.segmentation', 'torchvision.models.detection._utils', 'torchvision.models.detection.image_list', 'torchvision.models.detection.anchor_utils', 'torchvision.models.detection.generalized_rcnn', 'torchvision.models.detection.rpn', 'torchvision.models.detection.roi_heads', 'torchvision.models.detection.transform', 'torchvision.models.detection.backbone_utils', 'torchvision.models.detection.faster_rcnn', 'torchvision.models.detection.mask_rcnn', 'torchvision.models.detection.keypoint_rcnn', 'torchvision.models.detection.retinanet', 'torchvision.models.detection.ssd', 'torchvision.models.detection.ssdlite', 'torchvision.models.detection', 'torchvision.models.video.resnet', 'torchvision.models.video', 'torchvision.models.quantization.utils', 'torchvision.models.quantization.mobilenetv2', 'torchvision.models.quantization.mobilenetv3', 'torchvision.models.quantization.mobilenet', 'torchvision.models.quantization.resnet', 'torchvision.models.quantization.googlenet', 'torchvision.models.quantization.inception', 'torchvision.models.quantization.shufflenetv2', 'torchvision.models.quantization', 'torchvision.models.feature_extraction', 'torchvision.models', 'torchvision.datasets.vision', 'torchvision.datasets.utils', 'torchvision.datasets.lsun', 'torchvision.datasets.folder', 'torchvision.datasets.coco', 'torchvision.datasets.cifar', 'torchvision.datasets.stl10', 'torchvision.datasets.mnist', 'torchvision.datasets.svhn', 'torchvision.datasets.phototour', 'PIL.ImageFilter', 'PIL.ImageStat', 'PIL.ImageEnhance', 'torchvision.transforms.functional_pil', 'torchvision.transforms.functional_tensor', 'torchvision.transforms.functional', 'torchvision.transforms.transforms', 'torchvision.transforms.autoaugment', 'torchvision.transforms', 'torchvision.datasets.fakedata', 'torchvision.datasets.semeion', 'torchvision.datasets.omniglot', 'torchvision.datasets.sbu', 'html.entities', 'html', '_markupbase', 'html.parser', 'torchvision.datasets.flickr', 'torchvision.datasets.voc', 'torchvision.datasets.cityscapes', 'torchvision.datasets.imagenet', 'torchvision.datasets.caltech', 'torchvision.datasets.celeba', 'torchvision.datasets.widerface', 'torchvision.datasets.sbd', 'torchvision.datasets.usps', 'fractions', 'torchvision.io._video_opt', 'torchvision.io.video', 'torchvision.io.image', 'torchvision.io', 'torchvision.datasets.video_utils', 'torchvision.datasets.kinetics', 'torchvision.datasets.hmdb51', 'torchvision.datasets.ucf101', 'torchvision.datasets.places365', 'torchvision.datasets.kitti', 'torchvision.datasets.inaturalist', 'torchvision.datasets.lfw', 'torchvision.datasets', 'PIL.ImageColor', 'PIL._imagingft', 'PIL.ImageFont', 'PIL.ImageDraw', 'torchvision.utils', 'torchvision.version', 'torchvision', 'mmcv.runner.checkpoint', 'mmcv.runner.hooks.hook', 'mmcv.runner.hooks.checkpoint', 'mmcv.runner.hooks.closure', 'mmcv.runner.hooks.ema', 'mmcv.runner.hooks.logger.base', 'mmcv.runner.hooks.logger.dvclive', 'mmcv.runner.hooks.logger.mlflow', 'mmcv.runner.hooks.logger.neptune', 'mmcv.runner.hooks.logger.pavi', 'mmcv.runner.hooks.logger.tensorboard', 'mmcv.runner.hooks.logger.text', 'mmcv.runner.hooks.logger.wandb', 'mmcv.runner.hooks.logger', 'mmcv.runner.hooks.evaluation', 'mmcv.runner.hooks.iter_timer', 'mmcv.runner.hooks.lr_updater', 'mmcv.runner.hooks.memory', 'mmcv.runner.hooks.momentum_updater', 'mmcv.runner.fp16_utils', 'mmcv.runner.hooks.optimizer', 'mmcv.runner.hooks.profiler', 'mmcv.runner.hooks.sampler_seed', 'mmcv.runner.hooks.sync_buffer', 'mmcv.runner.hooks', 'mmcv.runner.log_buffer', 'mmcv.runner.priority', 'mmcv.runner.utils', 'mmcv.runner.base_runner', 'mmcv.runner.builder', 'mmcv.runner.default_constructor', 'mmcv.runner.epoch_based_runner', 'mmcv.runner.iter_based_runner', 'mmcv.runner.optimizer.builder', 'mmcv.utils.ext_loader', 'mmcv.runner.optimizer.default_constructor', 'mmcv.runner.optimizer', 'mmcv.runner', 'mmdet.datasets.samplers.distributed_sampler', 'mmdet.datasets.samplers.group_sampler', 'mmdet.datasets.samplers.infinite_sampler', 'mmdet.datasets.samplers', 'mmdet.datasets.builder', 'pycocotools', '_cython_0_29_21', 'pycocotools._mask', 'pycocotools.mask', 'terminaltables.terminal_io', 'terminaltables.width_and_alignment', 'terminaltables.build', 'terminaltables.base_table', 'terminaltables.ascii_table', 'terminaltables.github_table', 'encodings.cp437', 'terminaltables.other_tables', 'terminaltables', 'mmdet.core.anchor.builder', 'mmdet.core.anchor.anchor_generator', 'mmdet.core.anchor.point_generator', 'mmdet.core.anchor.utils', 'mmdet.core.anchor', 'mmdet.core.bbox.builder', 'mmdet.core.bbox.iou_calculators.builder', 'mmdet.core.bbox.iou_calculators.iou2d_calculator', 'mmdet.core.bbox.iou_calculators', 'mmdet.utils.collect_env', 'mmdet.utils.logger', 'mmdet.utils.misc', 'mmdet.utils.setup_env', 'mmdet.utils', 'mmdet.utils.util_mixins', 'mmdet.core.bbox.assigners.assign_result', 'mmdet.core.bbox.assigners.base_assigner', 'mmdet.core.bbox.assigners.max_iou_assigner', 'mmdet.core.bbox.assigners.approx_max_iou_assigner', 'mmdet.core.bbox.assigners.atss_assigner', 'mmdet.core.bbox.assigners.center_region_assigner', 'mmdet.core.bbox.assigners.grid_assigner', 'mmdet.core.bbox.match_costs.builder', 'mmdet.core.bbox.transfor [2022-05-31T14:24:23.474220Z] dc5b74e6 || ms', 'mmdet.core.bbox.match_costs.match_cost', 'mmdet.core.bbox.match_costs', 'scipy._lib._testutils', 'scipy._lib', 'scipy._lib.deprecation', 'scipy.config', 'scipy.version', 'scipy._distributor_init', 'scipy._lib._pep440', 'scipy._lib._ccallback_c', 'scipy._lib._ccallback', 'scipy', 'scipy._lib._util', 'scipy.sparse._sputils', 'scipy.sparse._base', 'scipy.sparse._sparsetools', 'scipy.sparse._data', 'scipy.sparse._index', 'scipy.sparse._compressed', 'scipy.sparse._csr', 'scipy.sparse._csc', 'scipy.sparse._csparsetools', 'scipy.sparse._lil', 'scipy.sparse._dok', 'scipy.sparse._coo', 'scipy.sparse._dia', 'scipy.sparse._bsr', 'scipy.sparse._construct', 'scipy.sparse._extract', 'scipy.sparse._matrix_io', 'scipy.sparse._arrays', 'scipy.sparse.csgraph._laplacian', '_cython_0_29_27', 'scipy.sparse.csgraph._tools', 'scipy.sparse.csgraph._validation', 'scipy.sparse.csgraph._shortest_path', 'scipy.sparse.csgraph._traversal', 'scipy.sparse.csgraph._min_spanning_tree', 'scipy.sparse.csgraph._flow', 'scipy.sparse.csgraph._matching', 'scipy.sparse.csgraph._reordering', 'scipy.sparse.csgraph', 'scipy.sparse.base', 'scipy.sparse.bsr', 'scipy.sparse.compressed', 'scipy.sparse.construct', 'scipy.sparse.coo', 'scipy.sparse.csc', 'scipy.sparse.csr', 'scipy.sparse.data', 'scipy.sparse.dia', 'scipy.sparse.dok', 'scipy.sparse.extract', 'scipy.sparse.lil', 'scipy.sparse.sparsetools', 'scipy.sparse.sputils', 'scipy.sparse', 'scipy.sparse.linalg._isolve._iterative', 'scipy.sparse.linalg._interface', 'scipy.sparse.linalg._isolve.utils', 'scipy._lib.decorator', 'scipy._lib._threadsafety', 'scipy.sparse.linalg._isolve.iterative', 'scipy.sparse.linalg._isolve.minres', 'scipy.linalg._fblas', 'scipy.linalg.blas', 'scipy.linalg._flapack', 'scipy.linalg.lapack', 'scipy.linalg._misc', 'scipy.linalg._cythonized_array_utils', 'scipy.linalg._flinalg', 'scipy.linalg._flinalg_py', 'scipy.linalg._decomp', 'scipy.linalg._decomp_svd', 'scipy.linalg._solve_toeplitz', 'scipy.linalg._basic', 'scipy.linalg._decomp_lu', 'scipy.linalg._decomp_ldl', 'scipy.linalg._decomp_cholesky', 'scipy.linalg._decomp_qr', 'scipy.linalg._decomp_qz', 'scipy.linalg._decomp_schur', 'scipy.linalg._decomp_polar', 'scipy.linalg._special_matrices', 'scipy.linalg._expm_frechet', 'scipy.linalg._matfuncs_sqrtm_triu', 'scipy.linalg._matfuncs_sqrtm', 'scipy.linalg._matfuncs', 'scipy.linalg._solvers', 'scipy.linalg._procrustes', 'scipy.linalg.cython_blas', 'scipy.linalg.cython_lapack', 'scipy.linalg._decomp_update', 'scipy.linalg._sketches', 'scipy.linalg._decomp_cossin', 'scipy.linalg.decomp', 'scipy.linalg.decomp_cholesky', 'scipy.linalg.decomp_lu', 'scipy.linalg.decomp_qr', 'scipy.linalg.decomp_svd', 'scipy.linalg.decomp_schur', 'scipy.linalg.basic', 'scipy.linalg.misc', 'scipy.linalg.special_matrices', 'scipy.linalg.flinalg', 'scipy.linalg.matfuncs', 'scipy.linalg', 'scipy.sparse.linalg._isolve._gcrotmk', 'scipy.sparse.linalg._isolve.lgmres', 'scipy.sparse.linalg._isolve.lsqr', 'scipy.sparse.linalg._isolve.lsmr', 'scipy.sparse.linalg._isolve.tfqmr', 'scipy.sparse.linalg._isolve', 'scipy.sparse.linalg._dsolve._superlu', 'scipy.sparse.linalg._dsolve.linsolve', 'scipy.sparse.linalg._dsolve._add_newdocs', 'scipy.sparse.linalg._dsolve', 'scipy.sparse.linalg._eigen.arpack._arpack', 'scipy.sparse.linalg._eigen.arpack.arpack', 'scipy.sparse.linalg._eigen.arpack', 'scipy.sparse.linalg._eigen.lobpcg.lobpcg', 'scipy.sparse.linalg._eigen.lobpcg', 'scipy.sparse.linalg._eigen._svds', 'scipy.sparse.linalg._eigen', 'scipy.sparse.linalg._expm_multiply', 'scipy.sparse.linalg._matfuncs', 'scipy.sparse.linalg._onenormest', 'scipy.sparse.linalg._norm', 'scipy.sparse.linalg.isolve', 'scipy.sparse.linalg.dsolve', 'scipy.sparse.linalg.interface', 'scipy.sparse.linalg.eigen', 'scipy.sparse.linalg.matfuncs', 'scipy.sparse.linalg', 'scipy.optimize._minpack2', 'scipy.optimize._linesearch', 'scipy.optimize._group_columns', 'scipy.optimize._numdiff', 'scipy.optimize._hessian_update_strategy', 'scipy.optimize._differentiable_functions', 'scipy.optimize._optimize', 'scipy.optimize._trustregion', 'scipy.optimize._trustregion_dogleg', 'scipy.optimize._trustregion_ncg', 'scipy._lib.messagestream', 'scipy.optimize._trlib._trlib', 'scipy.optimize._trlib', 'scipy.optimize._trustregion_krylov', 'scipy.optimize._trustregion_exact', 'numpy.testing._private', 'numpy.linalg.lapack_lite', 'numpy.testing._private.utils', 'numpy.testing._private.extbuild', 'numpy.testing._private.decorators', 'numpy.testing._private.nosetester', 'numpy.testing', 'scipy.optimize._constraints', 'scipy.optimize._trustregion_constr.projections', 'scipy.optimize._trustregion_constr.qp_subproblem', 'scipy.optimize._trustregion_constr.equality_constrained_sqp', 'scipy.optimize._trustregion_constr.canonical_constraint', 'scipy.optimize._trustregion_constr.tr_interior_point', 'scipy.optimize._trustregion_constr.report', 'scipy.optimize._trustregion_constr.minimize_trustregion_constr', 'scipy.optimize._trustregion_constr', 'scipy.optimize._lbfgsb', 'scipy.optimize._lbfgsb_py', '_moduleTNC', 'scipy.optimize._moduleTNC', 'scipy.optimize._tnc', 'scipy.optimize._cobyla', 'scipy.optimize._cobyla_py', 'scipy.optimize._slsqp', 'scipy.optimize._slsqp_py', 'scipy.optimize._minimize', 'scipy.optimize._minpack', 'scipy.optimize._lsq.common', 'scipy.optimize._lsq.trf', 'scipy.optimize._lsq.dogbox', 'scipy.optimize._lsq.least_squares', 'scipy.optimize._lsq.givens_elimination', 'scipy.optimize._lsq.trf_linear', 'scipy.optimize._lsq.bvls', 'scipy.optimize._lsq.lsq_linear', 'scipy.optimize._lsq', 'scipy.optimize._minpack_py', 'scipy.optimize._spectral', 'scipy.optimize._nonlin', 'scipy.optimize._root', 'scipy.optimize._zeros', 'scipy.optimize._zeros_py', 'scipy.optimize._root_scalar', 'scipy.optimize.nnls', 'scipy.optimize._nnls', 'scipy.optimize._basinhopping', 'scipy.optimize._highs', 'scipy.optimize._highs.cython.src._highs_wrapper', 'scipy.optimize._highs._highs_wrapper', 'scipy.optimize._highs.cython.src._highs_constants', 'scipy.optimize._highs._highs_constants', 'scipy.optimize._linprog_highs', 'scipy.linalg._interpolative', 'scipy.linalg._interpolative_backend', 'scipy.linalg.interpolative', 'scipy.optimize._remove_redundancy', 'scipy.optimize._linprog_util', 'scipy.optimize._linprog_ip', 'scipy.optimize._linprog_simplex', 'scipy.optimize._bglu_dense', 'scipy.optimize._linprog_rs', 'scipy.optimize._linprog_doc', 'scipy.optimize._linprog', 'scipy.optimize._lsap_module', 'scipy.optimize._lsap', 'scipy.optimize._differentialevolution', 'scipy.spatial._ckdtree', 'scipy.spatial._kdtree', 'scipy.spatial._qhull', 'scipy.spatial._voronoi', 'scipy.spatial._spherical_voronoi', 'scipy.spatial._plotutils', 'scipy.spatial._procrustes', 'scipy.spatial._distance_wrap', 'scipy.spatial._hausdorff', 'scipy.special._sf_error', 'scipy.special._ufuncs_cxx', 'scipy.special._ufuncs', 'scipy.special._specfun', 'scipy.special._orthogonal', 'scipy.special._comb', 'scipy.special._basic', 'scipy.special._logsumexp', 'scipy.special._spfun_stats', 'scipy.special._ellip_harm_2', 'scipy.special._ellip_harm', 'scipy.special._lambertw', 'scipy.special._spherical_bessel', 'scipy.special._add_newdocs', 'scipy.special.add_newdocs', 'scipy.special.basic', 'scipy.special.orthogonal', 'scipy.special.specfun', 'scipy.special.sf_error', 'scipy.special.spfun_stats', 'scipy.special', 'scipy.spatial._distance_pybind', 'scipy.spatial.distance', 'scipy.spatial._geometric_slerp', 'scipy.spatial.ckdtree', 'scipy.spatial.kdtree', 'scipy.spatial.qhull', 'scipy.constants._codata', 'scipy.constants._constants', 'scipy.constants.codata', 'scipy.constants.constants', 'scipy.constants', 'scipy.spatial.transform._rotation_groups', 'scipy.spatial.transform._rotation', 'scipy.spatial.transform._rotation_spline', 'scipy.spatial.transform.rotation', 'scipy.spatial.transform', 'scipy.spatial', 'scipy.optimize._shgo_lib', 'scipy.optimize._shgo_lib.triangulation', 'scipy.optimize._shgo', 'scipy.optimize._dual_annealing', 'scipy.optimize._qap', 'scipy.optimize.cobyla', 'scipy.optimize.lbfgsb', 'scipy.optimize.linesearch', 'scipy.optimize.minpack', 'scipy.optimize.minpack2', 'scipy.optimize.moduleTNC', 'scipy.optimize.nonlin', 'scipy.optimize.optimize', 'scipy.optimize.slsqp', 'scipy.optimize.tnc', 'scipy.optimize.zeros', 'scipy.optimize', 'mmdet.core.bbox.assigners.hungarian_assigner', 'mmdet.core.bbox.assigners.point_assigner', 'mmdet.core.bbox.assigners.region_assigner', 'mmdet.core.bbox.assigners.sim_ota_assigner', 'mmdet.core.bbox.assigners.task_aligned_assigner', 'mmdet.core.bbox.assigners.uniform_assigner', 'mmdet.core.bbox.assigners', 'mmdet.core.bbox.coder.base_bbox_coder', 'mmdet.core.bbox.coder.bucketing_bbox_coder', 'mmdet.core.bbox.coder.delta_xywh_bbox_coder', 'mmdet.core.bbox.coder.distance_point_bbox_coder', 'mmdet.core.bbox.coder.legacy_delta_xywh_bbox_coder', 'mmdet.core.bbox.coder.pseudo_bbox_coder', 'mmdet.core.bbox.coder.tblr_bbox_coder', 'mmdet.core.bbox.coder.yolo_bbox_coder', 'mmdet.core.bbox.coder', 'mmdet.core.bbox.samplers.sampling_result', 'mmdet.core.bbox.samplers.base_sampler', 'mmdet.core.bbox.samplers.combined_sampler', 'mmdet.core.bbox.samplers.random_sampler', 'mmdet.core.bbox.samplers.instance_balanced_pos_sampler', 'mmdet.core.bbox.samplers.iou_balanced_neg_sampler', 'mmdet.core.bbox.samplers.ohem_sampler', 'mmdet.core.bbox.samplers.pseudo_sampler', 'mmcv._ext', 'mmcv.ops.active_rotated_filter', 'mmcv.ops.assign_score_withk', 'mmcv.ops.ball_query', 'mmcv.ops.bbox', 'mmcv.ops.border_align', 'mmcv.ops.box_iou_rotated', 'mmcv.cnn.alexnet', 'mmcv.cnn.bricks.registry', 'mmcv.cnn.bricks.activation', 'mmcv.cnn.utils.flops_counter', 'mmcv.cnn.utils.fuse_conv_bn', 'mmcv.cnn.utils.sync_bn', 'mmcv.cnn.utils.weight_init', 'mmcv.cnn.utils', 'mmcv.cnn.bricks.context_block', 'mmcv.cnn.bricks.conv', 'mmcv.cnn.bricks.conv2d_adaptive_padding', 'mmcv.cnn.bricks.norm', 'mmcv.cnn.bricks.padding', 'mmcv.cnn.bricks.conv_module', 'mmcv.cnn.bricks.conv_ws', 'mmcv.cnn.bricks.depthwise_separable_conv_module', 'mmcv.cnn.bricks.drop', 'mmcv.cnn.bricks.generalized_attention', 'mmcv.cnn.bricks.hsigmoid', 'mmcv.cnn.bricks.hswish', 'mmcv.cnn.bricks.non_local', 'mmcv.cnn.bricks.plugin', 'mmcv.cnn.bricks.scale', 'mmcv.cnn.bricks.swish', 'mmcv.cnn.bricks.upsample', 'mmcv.cnn.bricks.wrappers', 'mmcv.cnn.bricks', 'mmcv.cnn.builder', 'mmcv.cnn.resnet', 'mmcv.cnn.vgg', 'mmcv.cnn', 'mmcv.ops.carafe', 'mmcv.ops.cc_attention', 'mmcv.ops.contour_expand', 'mmcv.ops.convex_iou', 'mmcv.ops.corner_pool', 'mmcv.ops.correlation', 'mmcv.ops.deform_conv', 'mmcv.ops.deform_roi_pool', 'mmcv.ops.deprecated_wrappers', 'mmcv.ops.focal_loss', 'mmcv.ops.furthest_point_sample', 'mmcv.ops.fused_bias_leakyrelu', 'mmcv.ops.gather_points', 'mmcv.ops.knn', 'mmcv.ops.group_points', 'mmcv.ops.info', 'mmcv.ops.iou3d', 'mmcv.ops.masked_conv', 'mmcv.ops.min_area_polygons', 'mmcv.ops.modulated_deform_conv', 'mmcv.ops.multi_scale_deform_attn', 'mmcv.ops.nms', 'mmcv.ops.pixel_group', 'torch.onnx.operators', 'mmcv.ops.point_sample', 'mmcv.ops.points_in_boxes', 'mmcv.ops.points_in_polygons', 'mmcv.ops.points_sampler', 'mmcv.ops.psa_mask', 'mmcv.ops.riroi_align_rotated', 'mmcv.ops.roi_align', 'mmcv.ops.roi_align_rotated', 'mmcv.ops.roi_pool', 'mmcv.ops.roiaware_pool3d', 'mmcv.ops.roipoint_pool3d', 'mmcv.ops.rotated_feature_align', 'mmcv.ops.saconv', 'mmcv.ops.scatter_points', 'mmcv.ops.sync_bn', 'mmcv.ops.three_interpolate', 'mmcv.ops.three_nn', 'mmcv.ops.tin_shift', 'mmcv.ops.upfirdn2d', 'mmcv.ops.voxelize', 'mmcv.ops', 'mmdet.core.bbox.samplers.score_hlr_sampler', 'mmdet.core.bbox.samplers', 'mmdet.core.bbox', 'mmdet.core.data_structures.general_data', 'mmdet.core.data_structures.instance_data', 'mmdet.core.data_structures', 'mmdet.core.evaluation.class_names', 'mmdet.core.evaluation.eval_hooks', 'mmdet.core.evaluation.bbox_overlaps', 'mmdet.core.evaluation.mean_ap', 'mmdet.core.evaluation.recall', 'mmdet.core.evaluation', 'mmdet.core.hook.checkloss_hook', 'mmdet.core.hook.ema', 'mmdet.core.hook.set_epoch_info_hook', 'mmdet.core.utils.dist_utils', 'mmdet.core.mask.mask_target', 'mmdet.core.mask.structures', 'mmdet.core.mask.utils', 'mmdet.core.mask', 'mmdet.core.utils.misc', 'mmdet.core.utils', 'mmdet.core.hook.sync_norm_hook', 'mmdet.core.hook.sync_random_size_hook', 'mmdet.core.hook.yolox_lrupdater_hook', 'mmdet.core.hook.yolox_mode_switch_hook', 'mmdet.core.hook', 'mmdet.core.post_processing.bbox_nms', 'mmdet.core.post_processing.matrix_nms', 'mmdet.core.post_processing.merge_augs', 'mmdet.core.post_processing', 'mmdet.core', 'pycocotools.coco', 'pycocotools.cocoeval', 'mmdet.datasets.api_wrappers.coco_api', 'panopticapi', 'panopticapi.utils', 'panopticapi.evaluation', 'mmdet.datasets.api_wrappers.panoptic_evaluation', 'mmdet.datasets.api_wrappers', 'mmdet.datasets.pipelines.compose', 'mmdet.datasets.pipelines.auto_augment', 'mmdet.datasets.pipelines.formatting', 'mmdet.datasets.pipelines.instaboost', 'mmdet.datasets.pipelines.loading', 'mmdet.datasets.pipelines.test_time_aug', 'mmdet.datasets.pipelines.transforms', 'mmdet.datasets.pipelines', 'mmdet.datasets.custom', 'mmdet.datasets.coco', 'mmdet.datasets.cityscapes', 'mmdet.datasets.coco_panoptic', 'mmdet.datasets.dataset_wrappers', 'mmdet.datasets.deepfashion', 'mmdet.datasets.lvis', 'mmdet.datasets.utils', 'mmdet.models', 'mmdet.models.builder', 'mmdet.models.utils.brick_wrappers', 'mmdet.models.utils.builder', 'mmdet.models.utils.ckpt_convert', 'mmdet.models.utils.conv_upsample', 'mmdet.models.utils.csp_layer', 'mmdet.models.utils.gaussian_target', 'mmdet.models.utils.se_layer', 'mmdet.models.utils.inverted_residual', 'mmdet.models.utils.make_divisible', 'mmdet.models.utils.misc', 'mmdet.models.utils.normed_predictor', 'mmcv.cnn.bricks.transformer', 'mmdet.models.utils.positional_encoding', 'mmdet.models.utils.res_layer', 'mmdet.models.utils.transformer', 'mmdet.models.utils', 'mmdet.models.backbones.csp_darknet', 'mmdet.models.backbones.darknet', 'mmdet.models.backbones.resnet', 'mmdet.models.backbones.detectors_resnet', 'mmdet.models.backbones.detectors_resnext', 'mmdet.models.backbones.hourglass', 'mmdet.models.backbones.hrnet', 'mmdet.models.backbones.mobilenet_v2', 'mmdet.models.backbones.pvt', 'mmdet.models.backbones.resnext', 'mmdet.models.backbones.regnet', 'mmdet.models.backbones.res2net', 'mmdet.models.backbones.resnest', 'mmdet.models.necks.bfp', 'mmdet.models.necks.channel_mapper', 'mmdet.models.necks.ct_resnet_neck', 'mmdet.models.necks.dilated_encoder', 'mmdet.models.necks.fpg', 'mmdet.models.necks.fpn', 'mmdet.models.necks.fpn_carafe', 'mmdet.models.necks.hrfpn', 'mmcv.ops.merge_cells', 'mmdet.models.necks.nas_fpn', 'mmdet.models.necks.nasfcos_fpn', 'mmdet.models.necks.pafpn', 'mmdet.models.necks.rfp', 'mmdet.models.necks.ssd_neck', 'mmdet.models.necks.yolo_neck', 'mmdet.models.necks.yolox_pafpn', 'mmdet.models.necks', 'mmdet.models.backbones.ssd_vgg', 'mmdet.models.backbones.swin', 'mmdet.models.backbones.trident_resnet', 'mmdet.models.backbones', 'mmdet.models.dense_heads.base_dense_head', 'mmdet.utils.contextmanagers', 'mmdet.models.dense_heads.dense_test_mixins', 'mmdet.models.dense_heads.anchor_free_head', 'mmdet.models.dense_heads.anchor_head', 'mmdet.models.dense_heads.atss_head', 'mmdet.models.dense_heads.fcos_head', 'mmdet.models.dense_heads.paa_head', 'mmdet.models.dense_heads.autoassign_head', 'mmdet.models.dense_heads.rpn_head', 'mmdet.models.dense_heads.cascade_rpn_head', 'mmdet.models.dense_heads.centernet_head', 'mmdet.models.dense_heads.corner_head', 'mmdet.models.dense_heads.centripetal_head', 'mmdet.models.dense_heads.detr_head', 'mmdet.models.dense_heads.deformable_detr_head', 'mmdet.models.dense_heads.embedding_rpn_head', 'mmdet.models.dense_heads.fovea_head', 'mmdet.models.dense_heads.retina_head', 'mmdet.models.dense_heads.free_anchor_retina_head', 'mmdet.models.losses.accuracy', 'mmdet.models.losses.ae_loss', 'mmdet.models.losses.utils', 'mmdet.models.losses.balanced_l1_loss', 'mmdet.models.losses.cross_entropy_loss', 'mmdet.models.losses.dice_loss', 'mmdet.models.losses.focal_loss', 'mmdet.models.losses.gaussian_focal_loss', 'mmdet.models.losses.gfocal_loss', 'mmdet.models.losses.ghm_loss', 'mmdet.models.losses.iou_loss', 'mmdet.models.losses.kd_loss', 'mmdet.models.losses.mse_loss', 'mmdet.models.losses.pisa_loss', 'mmdet.models.losses.seesaw_loss', 'mmdet.models.losses.smooth_l1loss', 'mmdet.models.losses.varifocal [2022-05-31T14:24:23.474220Z] dc5b74e6 || loss', 'mmdet.models.losses', 'mmdet.models.dense_heads.fsaf_head', 'mmdet.models.dense_heads.guided_anchor_head', 'mmdet.models.dense_heads.ga_retina_head', 'mmdet.models.dense_heads.ga_rpn_head', 'mmdet.models.dense_heads.gfl_head', 'mmdet.models.dense_heads.lad_head', 'mmdet.models.dense_heads.ld_head', 'mmdet.models.dense_heads.nasfcos_head', 'mmdet.models.dense_heads.pisa_retinanet_head', 'mmdet.models.dense_heads.ssd_head', 'mmdet.models.dense_heads.pisa_ssd_head', 'mmdet.models.dense_heads.reppoints_head', 'mmdet.models.dense_heads.retina_sepbn_head', 'mmdet.models.dense_heads.sabl_retina_head', 'mmdet.models.dense_heads.base_mask_head', 'mmdet.models.dense_heads.solo_head', 'mmdet.models.dense_heads.tood_head', 'mmdet.models.dense_heads.vfnet_head', 'mmdet.models.dense_heads.yolact_head', 'mmdet.models.dense_heads.yolo_head', 'mmdet.models.dense_heads.yolof_head', 'mmdet.models.dense_heads.yolox_head', 'mmdet.models.dense_heads', 'mmdet.models.detectors', 'mmdet.models.detectors.atss', 'mmdet.models.detectors.single_stage', 'mmdet.models.detectors.base', 'mmdet.core.visualization', 'mmdet.core.visualization.image', 'matplotlib', 'matplotlib._api.deprecation', 'matplotlib._api', 'matplotlib._version', 'matplotlib._c_internal_utils', 'matplotlib.cbook', 'matplotlib.docstring', 'PIL.ImageChops', 'PIL.ImageFile', 'PIL.GimpGradientFile', 'PIL.GimpPaletteFile', 'PIL.PaletteFile', 'PIL.ImagePalette', 'PIL.ImageSequence', 'PIL.PngImagePlugin', 'matplotlib._path', 'matplotlib.bezier', 'matplotlib.path', 'matplotlib.transforms', 'matplotlib.ticker', 'matplotlib.scale', 'matplotlib._color_data', 'matplotlib.colors', 'pyparsing.util', 'pyparsing.unicode', 'pyparsing.exceptions', 'pyparsing.actions', 'pyparsing.results', 'pyparsing.core', 'pyparsing.helpers', 'pyparsing.testing', 'pyparsing.common', 'pyparsing', 'matplotlib.fontconfig_pattern', 'matplotlib._enums', 'cycler', 'matplotlib.rcsetup', 'matplotlib.ft2font', 'kiwisolver._cext', 'kiwisolver'] [2022-05-31T14:24:23.474628Z] dc5b74e6 [rank=0] || DEBUG: [49] matplotlib: loaded modules: ['sys', 'builtins', '_frozen_importlib', '_imp', '_warnings', '_io', 'marshal', 'posix', '_frozen_importlib_external', '_thread', '_weakref', 'time', 'zipimport', 'faulthandler', '_codecs', 'codecs', 'encodings.aliases', 'encodings', 'encodings.utf_8', '_signal', 'main', 'encodings.latin_1', '_abc', 'abc', 'io', '_stat', 'stat', '_collections_abc', 'genericpath', 'posixpath', 'os.path', 'os', '_sitebuiltins', '_locale', '_bootlocale', 'types', 'importlib._bootstrap', 'importlib._bootstrap_external', 'warnings', 'importlib', 'importlib.machinery', 'importlib.abc', '_operator', 'operator', 'keyword', '_heapq', 'heapq', 'itertools', 'reprlib', '_collections', 'collections', '_functools', 'functools', 'contextlib', 'importlib.util', 'google', 'google.cloud', 'google.logging', 'mpl_toolkits', 'ruamel', 'site', '_weakrefset', 'weakref', 'pkgutil', 'runpy', 'determined.version', 'collections.abc', 'enum', '_sre', 'sre_constants', 'sre_parse', 'sre_compile', 'copyreg', 're', 'typing.io', 'typing.re', 'typing', 'determined._experiment_config', '_json', 'json.scanner', 'json.decoder', 'json.encoder', 'json', '_csv', 'csv', 'token', 'tokenize', 'linecache', 'traceback', '_string', 'string', 'threading', 'atexit', 'logging', 'errno', 'signal', '_posixsubprocess', 'select', 'math', 'selectors', 'subprocess', 'determined.gpu', 'determined._info', 'fnmatch', 'zlib', '_compression', '_bz2', 'bz2', '_lzma', 'lzma', 'pwd', 'grp', 'shutil', '_socket', 'socket', '_bisect', 'bisect', '_sha512', '_random', 'random', 'tempfile', 'determined.constants', 'future', '_datetime', 'datetime', 'psutil._common', 'psutil._compat', '_struct', 'struct', 'binascii', 'base64', 'glob', 'psutil._psposix', 'psutil._psutil_linux', 'psutil._psutil_posix', 'resource', 'psutil._pslinux', 'psutil', 'cython_runtime', 'textwrap', 'ruamel.yaml.compat', 'ruamel.yaml.error', 'ruamel.yaml.util', 'ruamel.yaml.reader', '_opcode', 'opcode', 'dis', 'inspect', 'ruamel.yaml.tokens', 'ruamel.yaml.scanner', 'ruamel.yaml.events', 'copy', 'ruamel.yaml.anchor', 'ruamel.yaml.scalarstring', 'ruamel.yaml.comments', 'ruamel.yaml.parser', 'ruamel.yaml.nodes', 'ruamel.yaml.composer', 'ruamel.yaml.scalarint', 'ruamel.yaml.scalarfloat', 'ruamel.yaml.scalarbool', 'ruamel.yaml.timestamp', 'ruamel.yaml.constructor', 'ruamel.yaml.emitter', 'ruamel.yaml.serializer', 'ruamel.yaml.representer', '_ruamel_yaml', 'ruamel.yaml.resolver', 'ruamel.yaml.cyaml', 'ruamel.yaml.loader', 'ruamel.yaml.dumper', 'ruamel.yaml.main', 'ruamel.yaml', 'locale', 'gettext', 'argparse', 'termios', 'getpass', '_hashlib', '_blake2', '_sha3', 'hashlib', 'ntpath', 'urllib', 'urllib.parse', 'pathlib', 'filelock._error', 'filelock._api', 'filelock._util', 'filelock._soft', 'fcntl', 'filelock._unix', 'filelock._windows', 'filelock.version', 'filelock', 'determined.common.constants', 'platform', 'determined.common.util', 'importlib.resources', 'certifi.core', 'certifi', 'determined.common.api.certs', 'determined.common.api.authentication', 'urllib3.packages', 'urllib3.packages.six', 'urllib3.packages.six.moves', 'http', 'email', 'email.errors', 'email.quoprimime', 'email.base64mime', 'quopri', 'email.encoders', 'email.charset', 'email.header', 'calendar', 'email._parseaddr', 'email.utils', 'email._policybase', 'email.feedparser', 'email.parser', 'uu', 'email._encoded_words', 'email.iterators', 'email.message', '_ssl', 'ssl', 'http.client', 'urllib3.packages.six.moves.http_client', 'urllib3.exceptions', 'urllib3._version', 'urllib3.contrib', 'urllib3.contrib._appengine_environ', 'urllib3.util.wait', 'urllib3.util.connection', '_cffi_backend', '_brotli.lib', '_brotli', 'brotli.brotli', 'brotli.brotli', 'brotli', 'urllib3.util.request', 'urllib3.util.response', 'urllib3.util.retry', 'hmac', 'urllib3.util.url', 'urllib3.util.ssltransport', 'urllib3.util.ssl', 'urllib3.util.timeout', 'urllib3.util', 'urllib3.util.proxy', 'urllib3._collections', 'ipaddress', 'urllib3.util.ssl_match_hostname', 'urllib3.connection', '_queue', 'queue', 'mimetypes', 'urllib3.fields', 'urllib3.filepost', 'urllib3.packages.six.moves.urllib', 'urllib3.packages.six.moves.urllib.parse', 'urllib3.request', 'urllib3.response', 'urllib3.util.queue', 'urllib3.connectionpool', 'urllib3.poolmanager', 'urllib3', 'charset_normalizer.constant', 'unicodedata', '_multibytecodec', 'charset_normalizer.utils', 'charset_normalizer.md', 'charset_normalizer.models', 'charset_normalizer.assets', 'charset_normalizer.cd', 'charset_normalizer.api', 'charset_normalizer.legacy', 'charset_normalizer.version', 'charset_normalizer', 'urllib.response', 'urllib.error', 'urllib.request', 'http.cookiejar', 'http.cookies', 'requests.compat', 'requests.exceptions', 'requests.version', 'zipfile', 'requests.certs', 'requests._internal_utils', 'requests.cookies', 'requests.structures', 'requests.utils', 'requests.packages.urllib3.packages', 'requests.packages.urllib3.packages.six', 'requests.packages.urllib3.packages.six.moves', 'requests.packages.urllib3.packages.six.moves.http_client', 'requests.packages.urllib3.exceptions', 'requests.packages.urllib3._version', 'requests.packages.urllib3.contrib', 'requests.packages.urllib3.contrib._appengineenviron', 'requests.packages.urllib3.util.wait', 'requests.packages.urllib3.util.connection', 'requests.packages.urllib3.util.request', 'requests.packages.urllib3.util.response', 'requests.packages.urllib3.util.retry', 'requests.packages.urllib3.util.url', 'requests.packages.urllib3.util.ssltransport', 'requests.packages.urllib3.util.ssl', 'requests.packages.urllib3.util.timeout', 'requests.packages.urllib3.util', 'requests.packages.urllib3.util.proxy', 'requests.packages.urllib3._collections', 'requests.packages.urllib3.util.ssl_match_hostname', 'requests.packages.urllib3.connection', 'requests.packages.urllib3.fields', 'requests.packages.urllib3.filepost', 'requests.packages.urllib3.packages.six.moves.urllib', 'requests.packages.urllib3.packages.six.moves.urllib.parse', 'requests.packages.urllib3.request', 'requests.packages.urllib3.response', 'requests.packages.urllib3.util.queue', 'requests.packages.urllib3.connectionpool', 'requests.packages.urllib3.poolmanager', 'requests.packages.urllib3', 'idna.package_data', 'idna.idnadata', 'idna.intranges', 'idna.core', 'idna', 'requests.packages.idna.package_data', 'requests.packages.idna.idnadata', 'requests.packages.idna.intranges', 'requests.packages.idna.core', 'requests.packages.idna', 'requests.packages.chardet', 'requests.packages', 'stringprep', 'encodings.idna', 'requests.hooks', 'requests.auth', 'requests.status_codes', 'requests.models', 'socks', 'urllib3.contrib.socks', 'requests.adapters', 'requests.sessions', 'requests.api', 'requests', 'determined.common.api.errors', 'determined.common.api.metric', 'shlex', 'webbrowser', 'lomond._version', 'six', 'six.moves', 'six.moves.urllib', 'six.moves.urllib.parse', 'lomond.constants', 'lomond.errors', 'lomond.events', 'lomond.compression', 'lomond.mask', 'lomond.opcode', 'lomond.frame', 'lomond.extension', 'lomond.response', 'lomond.parser', 'lomond.utf8validator', 'lomond.frame_parser', 'lomond.message', 'lomond.stream', 'lomond.proxy', 'lomond.selectors', 'lomond.session', 'lomond.status', 'lomond.websocket', 'lomond', 'determined.common.requests', 'determined.common.api.request', 'uuid', 'termcolor', 'tarfile', 'pathspec.compat', 'pathspec.pattern', 'pathspec.util', 'pathspec.pathspec', 'pathspec._meta', 'pathspec.patterns.gitwildmatch', 'pathspec.patterns', 'pathspec', 'determined.common.check', 'determined.common.context', 'determined.common.api.bindings', 'determined.common.api.logs', 'dataclasses', 'determined.common.storage.base', 'determined.common.storage.cloud', 'determined.errors', 'determined.common.storage.azure', 'determined.common.storage.gcs', 'six.moves.queue', 'hdfs.util', 'multiprocessing.process', '_compat_pickle', '_pickle', 'pickle', 'array', 'multiprocessing.reduction', 'multiprocessing.context', 'mp_main', 'multiprocessing', 'multiprocessing.util', '_multiprocessing', 'multiprocessing.connection', 'multiprocessing.pool', 'hdfs.client', 'imp', 'logging.handlers', 'configparser', 'six.moves.configparser', 'hdfs.config', 'hdfs', 'determined.common.storage.hdfs', 'determined.common.storage.s3', 'determined.common.storage.shared', 'determined.common.storage', 'determined.common.experimental.session', 'determined.common.experimental.checkpoint._checkpoint', 'determined.common.experimental.checkpoint', 'determined.common.experimental.trial', 'determined.common.experimental.experiment', 'determined.common.experimental.model', 'determined.common.experimental.determined', 'determined.common.experimental', 'determined.common.api.experiment', 'backoff._common', 'backoff._jitter', 'backoff._sync', 'backoff._decorator', 'backoff._wait_gen', 'backoff', 'concurrent', 'concurrent.futures._base', 'concurrent.futures', 'asyncio.constants', 'asyncio.format_helpers', 'asyncio.base_futures', 'asyncio.log', 'asyncio.coroutines', '_contextvars', 'contextvars', 'asyncio.exceptions', 'asyncio.base_tasks', '_asyncio', 'asyncio.events', 'asyncio.futures', 'asyncio.protocols', 'asyncio.transports', 'asyncio.sslproto', 'asyncio.locks', 'asyncio.tasks', 'asyncio.staggered', 'asyncio.trsock', 'asyncio.base_events', 'asyncio.runners', 'asyncio.queues', 'asyncio.streams', 'asyncio.subprocess', 'asyncio.base_subprocess', 'asyncio.selector_events', 'asyncio.unix_events', 'asyncio', 'determined.common.api.profiler', 'determined.common.api', 'determined.common._logging', 'determined.common', 'determined.ipc', 'determined.core._distributed', 'determined.tensorboard.metric_writers.util', 'determined.tensorboard.metric_writers.callback', 'determined.tensorboard.metric_writers', 'determined.tensorboard.base', 'determined.tensorboard.azure', 'determined.tensorboard.gcs', 'determined.tensorboard.hdfs', 'determined.tensorboard.s3', 'determined.tensorboard.shared', 'determined.tensorboard.build', 'determined.tensorboard.util', 'determined.tensorboard', 'determined.core._checkpoint', 'determined.core._train', 'determined.core._searcher', 'determined.core._preempt', 'appdirs', 'determined.core._context', 'determined.core', 'determined._env_context', 'determined._trial_context', 'determined._trial', 'determined.profiler', 'determined.workload', 'determined._trial_controller', 'determined.load', 'determined._execution', 'determined._hparam', 'numbers', 'determined.util', 'determined', 'determined.exec', 'determined.horovod', 'analytics.version', 'dateutil._version', 'dateutil', 'dateutil.tz._common', 'dateutil.tz._factories', 'dateutil.tz.tz', 'dateutil.tz', '_decimal', 'decimal', 'analytics.utils', 'monotonic', 'gzip', 'analytics.request', 'analytics.consumer', 'analytics.client', 'analytics', 'determined.common.api.analytics', 'model_hub', 'model_hub.mmdetection', 'model_hub.mmdetection._data', 'numpy._globals', 'numpy.config__', 'numpy._distributor_init', 'numpy._version', 'numpy.version', 'numpy.core._multiarray_umath', 'numpy.compat._inspect', 'numpy.compat.py3k', 'numpy.compat', 'numpy.core.overrides', 'numpy.core.multiarray', 'numpy.core.umath', 'numpy.core._string_helpers', 'numpy.core._dtype', 'numpy.core._type_aliases', 'numpy.core.numerictypes', 'numpy.core._exceptions', 'numpy.core._methods', 'numpy.core.fromnumeric', 'numpy.core.shape_base', 'numpy.core._ufunc_config', 'numpy.core.arrayprint', 'numpy.core._asarray', 'numpy.core.numeric', 'numpy.core.defchararray', 'numpy.core.records', 'numpy.core.memmap', 'numpy.core.function_base', 'numpy.core._machar', 'numpy.core.getlimits', 'numpy.core.einsumfunc', 'numpy.core._multiarray_tests', 'numpy.core._add_newdocs', 'numpy.core._add_newdocs_scalars', 'numpy.core._dtype_ctypes', '_ast', 'ast', '_ctypes', 'ctypes._endian', 'ctypes', 'numpy.core._internal', 'numpy._pytesttester', 'numpy.core', 'numpy.lib.mixins', 'numpy.lib.ufunclike', 'numpy.lib.type_check', 'numpy.lib.scimath', 'numpy.lib.stride_tricks', 'numpy.lib.twodim_base', 'numpy.linalg._umath_linalg', 'numpy.linalg.linalg', 'numpy.linalg', 'numpy.matrixlib.defmatrix', 'numpy.matrixlib', 'numpy.lib.histograms', 'numpy.lib.function_base', 'numpy.lib.index_tricks', 'numpy.lib.nanfunctions', 'numpy.lib.shape_base', 'numpy.lib.polynomial', 'numpy.lib.utils', 'numpy.lib.arraysetops', 'numpy.lib.format', 'numpy.lib._datasource', 'numpy.lib._iotools', 'numpy.lib.npyio', 'numpy.lib.arrayterator', 'numpy.lib.arraypad', 'numpy.lib._version', 'numpy.lib', 'numpy.fft._pocketfft_internal', 'numpy.fft._pocketfft', 'numpy.fft.helper', 'numpy.fft', 'numpy.polynomial.polyutils', 'numpy.polynomial._polybase', 'numpy.polynomial.polynomial', 'numpy.polynomial.chebyshev', 'numpy.polynomial.legendre', 'numpy.polynomial.hermite', 'numpy.polynomial.hermite_e', 'numpy.polynomial.laguerre', 'numpy.polynomial', '_cython_0_29_24', 'numpy.random._common', 'secrets', 'numpy.random.bit_generator', 'numpy.random._bounded_integers', 'numpy.random._mt19937', 'numpy.random.mtrand', 'numpy.random._philox', 'numpy.random._pcg64', 'numpy.random._sfc64', 'numpy.random._generator', 'numpy.random._pickle', 'numpy.random', 'numpy.ctypeslib', 'numpy.ma.core', 'numpy.ma.extras', 'numpy.ma', 'numpy', 'mmcv.arraymisc.quantization', 'mmcv.arraymisc', 'addict.addict', 'addict', 'yapf.yapflib', 'lib2to3', 'lib2to3.pgen2', 'lib2to3.pgen2.token', 'lib2to3.pgen2.tokenize', 'yapf.yapflib.errors', 'yapf.yapflib.py3compat', 'yapf.yapflib.style', 'yapf.yapflib.file_resources', 'difflib', 'lib2to3.pgen2.parse', 'lib2to3.pgen2.grammar', 'lib2to3.pgen2.pgen', 'lib2to3.pgen2.driver', 'lib2to3.pytree', 'lib2to3.pygram', 'yapf.yapflib.pytree_utils', 'yapf.yapflib.pytree_visitor', 'yapf.yapflib.blank_line_calculator', 'yapf.yapflib.comment_splicer', 'yapf.yapflib.subtypes', 'yapf.yapflib.format_token', 'yapf.yapflib.continuation_splicer', 'yapf.yapflib.identify_container', 'yapf.yapflib.split_penalty', 'lib2to3.pgen2.literals', 'lib2to3.patcomp', 'lib2to3.fixer_util', 'yapf.yapflib.logical_line', 'yapf.yapflib.object_state', 'yapf.yapflib.pytree_unwrapper', 'yapf.yapflib.format_decision_state', 'yapf.yapflib.line_joiner', 'yapf.yapflib.verifier', 'yapf.yapflib.reformatter', 'yapf.yapflib.subtype_assigner', 'yapf.yapflib.yapf_api', 'yapf', 'mmcv.utils.misc', 'mmcv.utils.path', 'mmcv.utils.config', 'mmcv.utils.timer', 'mmcv.utils.progressbar', 'unittest.util', 'unittest.result', 'pprint', 'unittest.case', 'unittest.async_case', 'unittest.suite', 'unittest.loader', 'unittest.signals', 'unittest.runner', 'unittest.main', 'unittest', 'unittest.mock', 'mmcv.utils.testing', 'packaging.about', 'packaging', 'packaging._structures', 'packaging.version', 'mmcv.utils.version_utils', 'torch.autocast_mode', 'torch._utils', 'torch._utils_internal', 'xml', 'xml.parsers', 'pyexpat.errors', 'pyexpat.model', 'pyexpat', 'xml.parsers.expat.model', 'xml.parsers.expat.errors', 'xml.parsers.expat', 'plistlib', 'pkg_resources.extern', 'pkg_resources._vendor', 'pkg_resources._vendor.appdirs', 'pkg_resources.extern.appdirs', 'pkg_resources._vendor.packaging.about__', 'pkg_resources._vendor.packaging', 'pkg_resources.extern.packaging', 'pkg_resources.extern.packaging._structures', 'pkg_resources.extern.packaging.version', 'sysconfig', 'pkg_resources._vendor.packaging._manylinux', 'pkg_resources._vendor.packaging._musllinux', 'pkg_resources.extern.packaging.tags', 'pkg_resources.extern.packaging.utils', 'pkg_resources.extern.packaging.specifiers', 'pkg_resources._vendor.pyparsing', 'pkg_resources.extern.pyparsing', 'pkg_resources.extern.packaging.markers', 'pkg_resources.extern.packaging.requirements', 'pkg_resources', 'torch.version', 'torch.torch_version', 'torch._six', 'torch._C._onnx', 'torch._C._jit', 'torch._C._jit_tree_views', 'torch._C._te', 'torch._C.cpp', 'torch._C.cpp.nn', 'torch._C._cudart', 'torch._C._nvtx', 'torch._C._cudnn', 'torch._C', 'torch._C._fft', 'torch._C._fx', 'torch._C._linalg', 'torch._C._nn', 'torch._C._special', 'torch._namedtensor_internals', 'torch.overrides', 'torch.utils.throughput_benchmark', 'torch.utils._crash_handler', 'torch.utils', 'torch.utils.hooks', 'torch._tensor', 'torch.storage', 'torch.random', 'torch._sources', 'torch.types', 'torch.serialization', 'torch._tensor_str', 'torch.cuda._utils', 'gc', 'torch.cuda.graphs', 'torch.cuda.streams', 'torch.cuda.memory', 'torch.cuda.random [2022-05-31T14:24:23.474628Z] dc5b74e6 || ', 'torch.cuda.sparse', 'torch.cuda.profiler', 'torch.cuda.nvtx', 'torch.cuda.amp.autocast_mode', 'torch.cuda.amp.common', 'torch.cuda.amp.grad_scaler', 'torch.cuda.amp', 'torch.cuda', 'torch.sparse', 'torch.nn.parameter', 'torch.nn.modules.module', 'torch._VF', 'torch._torch_docs', 'torch._C._distributed_c10d', 'torch.distributed.constants', 'torch.distributed.rendezvous', 'torch.distributed.distributed_c10d', 'torch.distributed.remote_device', 'torch.distributed', 'torch._C._distributed_rpc', 'torch.futures', 'torch.distributed.rpc.internal', 'torch.distributed.rpc.constants', 'torch.distributed.rpc.api', 'torch.distributed.rpc.backend_registry', 'torch.distributed.rpc.functions', 'torch._C._distributed_autograd', 'torch.distributed.autograd', 'torch.distributed.rpc.options', 'torch.autograd.variable', 'torch.autograd.function', 'cmath', 'torch.testing._core', 'torch.testing._asserts', 'torch.testing._creation', 'torch.testing._check_kernel_launches', 'torch.testing._dtype_getters', 'torch.testing._deprecated', 'torch.testing', 'torch.utils._pytree', 'torch._vmap_internals', 'torch.autograd.gradcheck', 'torch.autograd.grad_mode', 'torch.autograd.anomaly_mode', 'torch.autograd.functional', 'torch.autograd.forward_ad', 'torch.autograd.graph', 'torch._C._autograd', 'torch.autograd.profiler_util', 'torch.autograd.profiler', 'torch.autograd', 'torch.autograd.profiler_legacy', 'torch.distributed.rpc.server_process_global_profiler', 'torch.distributed.rpc', 'torch.package.analyze.trace_dependencies', 'torch.package.analyze', 'torch.package._mangling', 'torch.package.analyze.is_from_package', 'torch.package.glob_group', 'torch.package.file_structure_representation', 'torch.package.importer', 'pickletools', 'torch.package._digraph', 'torch.package._importlib', 'torch.package._package_pickler', 'torch.package._stdlib', 'torch.package.find_file_dependencies', 'torch.package.package_exporter', 'torch.package._directory_reader', 'torch.package._package_unpickler', 'torch.package.package_importer', 'torch.package', 'torch._jit_internal', 'torch.nn._reduction', 'torch.nn.modules.utils', 'torch.nn.grad', 'torch.nn.functional', 'torch.nn.init', 'typing_extensions', 'torch.nn.modules.lazy', 'torch.nn.modules.linear', 'torch.nn.common_types', 'torch.nn.modules.conv', 'torch.nn.modules.activation', 'torch.nn.modules.distance', 'torch.nn.modules.loss', 'torch.nn.modules.container', 'torch.nn.modules.pooling', 'torch.nn.modules._functions', 'torch.nn.modules.batchnorm', 'torch.nn.modules.instancenorm', 'torch.nn.modules.normalization', 'torch.nn.modules.dropout', 'torch.nn.modules.padding', 'torch.nn.modules.sparse', 'torch.nn.utils.rnn', 'torch.nn.utils.clip_grad', 'torch.nn.utils.weight_norm', 'torch.nn.utils.convert_parameters', 'torch.nn.utils.spectral_norm', 'torch.nn.utils.fusion', 'torch.nn.utils.memory_format', 'torch.nn.utils.parametrize', 'torch.nn.utils.parametrizations', 'torch.nn.utils.init', 'torch.nn.utils', 'torch.nn.modules.rnn', 'torch.nn.modules.pixelshuffle', 'torch.nn.modules.upsampling', 'torch.nn.modules.fold', 'torch.nn.modules.adaptive', 'torch.nn.modules.transformer', 'torch.nn.modules.flatten', 'torch.nn.modules.channelshuffle', 'torch.nn.modules', 'torch.nn.parallel.parallel_apply', 'torch.cuda.nccl', 'torch.nn.parallel.comm', 'torch.nn.parallel.replicate', 'torch.nn.parallel._functions', 'torch.nn.parallel.scatter_gather', 'torch.nn.parallel.data_parallel', 'torch.distributed.algorithms.join', 'torch.distributed.algorithms', 'torch.nn.parallel.distributed', 'torch.nn.parallel', 'torch.nn', 'torch._linalg_utils', 'torch._lowrank', 'torch.functional', 'torch.cpu.amp.autocast_mode', 'torch.cpu.amp', 'torch.cpu', 'torch.fft', 'torch.nn.intrinsic.modules.fused', 'torch.nn.intrinsic.modules', 'torch.nn.intrinsic', 'torch.nn.quantized.modules.utils', 'torch.jit._monkeytype_config', 'torch.jit._state', 'torch.jit.annotations', 'astunparse.unparser', 'astunparse.printer', 'astunparse', 'torch.jit.frontend', 'torch.backends', 'torch.backends.cudnn', 'torch.jit._builtins', 'torch.jit._check', 'torch.jit._recursive', 'torch.jit._fuser', 'torch.jit._serialization', 'torch._classes', 'torch.jit._script', 'torch.jit._trace', 'torch.jit._async', 'torch.jit._freeze', 'torch.jit', 'torch.nn.quantized.functional', 'torch.nn.quantized.modules.activation', 'torch.nn.quantized.modules.batchnorm', 'torch.nn.quantized.modules.normalization', 'torch.nn.qat.modules.linear', 'torch.nn.qat.modules.conv', 'torch.nn.qat.modules', 'torch.nn.qat', 'torch.nn.intrinsic.qat.modules.linear_relu', 'torch.nn.intrinsic.qat.modules.conv_fused', 'torch.nn.intrinsic.qat.modules', 'torch.nn.intrinsic.qat', 'torch._ops', 'torch.nn.quantized.modules.conv', 'torch.nn.quantized.modules.linear', 'torch.nn.quantized.modules.embedding_ops', 'torch.nn.quantized.modules.functional_modules', 'torch.nn.quantized.modules', 'torch.nn.quantized', 'torch.nn.quantizable.modules.activation', 'torch.nn.quantizable.modules.rnn', 'torch.nn.quantizable.modules', 'torch.nn.quantizable', 'torch.ao.nn.sparse.quantized.utils', 'torch.ao.nn.sparse.quantized.linear', 'torch.ao.nn.sparse.quantized.dynamic.linear', 'torch.ao.nn.sparse.quantized.dynamic', 'torch.ao.nn.sparse.quantized', 'torch.ao.nn.sparse', 'torch.ao.nn', 'torch.ao.sparsity._mappings', 'torch.ao.sparsity.sparsifier', 'torch.ao.sparsity.sparsifier.utils', 'torch.ao.sparsity.sparsifier.base_sparsifier', 'torch.ao.sparsity.sparsifier.weight_norm_sparsifier', 'torch.ao.sparsity.scheduler', 'torch.ao.sparsity.scheduler.base_scheduler', 'torch.ao.sparsity.scheduler.lambda_scheduler', 'torch.ao.sparsity.experimental', 'torch.ao.sparsity.experimental.pruner', 'torch.ao.sparsity.experimental.pruner.parametrization', 'torch.ao.sparsity.experimental.pruner.base_pruner', 'torch.ao.sparsity', 'torch.ao', 'torch.optim._functional', 'torch.optim.optimizer', 'torch.optim.adadelta', 'torch.optim.adagrad', 'torch.optim.adam', 'torch.optim.adamw', 'torch.optim.sparse_adam', 'torch.optim.adamax', 'torch.optim.asgd', 'torch.optim.sgd', 'torch.optim.radam', 'torch.optim.rprop', 'torch.optim.rmsprop', 'torch.optim.nadam', 'torch.optim.lbfgs', 'torch.optim.lr_scheduler', 'torch.optim.swa_utils', 'torch.optim', 'torch.optim._multi_tensor.adam', 'torch.optim._multi_tensor.adamw', 'torch.optim._multi_tensor._functional', 'torch.optim._multi_tensor.nadam', 'torch.optim._multi_tensor.sgd', 'torch.optim._multi_tensor.radam', 'torch.optim._multi_tensor.rmsprop', 'torch.optim._multi_tensor.rprop', 'torch.optim._multi_tensor.asgd', 'torch.optim._multi_tensor.adamax', 'torch.optim._multi_tensor.adadelta', 'torch.optim._multi_tensor.adagrad', 'torch.optim._multi_tensor', 'multiprocessing.resource_sharer', 'torch.multiprocessing.reductions', 'torch.multiprocessing.spawn', 'torch.multiprocessing', 'torch.special', 'torch.utils.backcompat', 'torch.onnx', 'torch.linalg', 'tqdm._monitor', 'tqdm._tqdm_pandas', 'tqdm.utils', 'tqdm.std', 'tqdm._dist_ver', 'tqdm.version', 'tqdm.cli', 'tqdm.gui', 'tqdm', 'tqdm.autonotebook', 'tqdm.asyncio', 'tqdm.auto', 'torch.hub', 'torch.distributions.constraints', 'torch.distributions.utils', 'torch.distributions.distribution', 'torch.distributions.exp_family', 'torch.distributions.bernoulli', 'torch.distributions.dirichlet', 'torch.distributions.beta', 'torch.distributions.binomial', 'torch.distributions.categorical', 'torch.distributions.cauchy', 'torch.distributions.gamma', 'torch.distributions.chi2', 'torch.distributions.transforms', 'torch.distributions.constraint_registry', 'torch.distributions.continuous_bernoulli', 'torch.distributions.exponential', 'torch.distributions.fishersnedecor', 'torch.distributions.geometric', 'torch.distributions.uniform', 'torch.distributions.independent', 'torch.distributions.transformed_distribution', 'torch.distributions.gumbel', 'torch.distributions.half_cauchy', 'torch.distributions.normal', 'torch.distributions.half_normal', 'torch.distributions.laplace', 'torch.distributions.multivariate_normal', 'torch.distributions.lowrank_multivariate_normal', 'torch.distributions.one_hot_categorical', 'torch.distributions.pareto', 'torch.distributions.poisson', 'torch.distributions.kl', 'torch.distributions.kumaraswamy', 'torch.distributions.lkj_cholesky', 'torch.distributions.log_normal', 'torch.distributions.logistic_normal', 'torch.distributions.mixture_same_family', 'torch.distributions.multinomial', 'torch.distributions.negative_binomial', 'torch.distributions.relaxed_bernoulli', 'torch.distributions.relaxed_categorical', 'torch.distributions.studentT', 'torch.distributions.von_mises', 'torch.distributions.weibull', 'torch.distributions', 'torch.backends.cuda', 'torch.backends.mkl', 'torch.backends.mkldnn', 'torch.backends.openmp', 'torch.backends.quantized', 'torch.ao.quantization.quant_type', 'torch.ao.quantization.utils', 'torch.ao.quantization.observer', 'torch.quantization.observer', 'torch.ao.quantization.fake_quantize', 'torch.ao.quantization.fuser_method_mappings', 'torch.quantization.fuser_method_mappings', 'torch.ao.quantization.fuse_modules', 'torch.nn.intrinsic.quantized.modules.linear_relu', 'torch.nn.intrinsic.quantized.modules.conv_relu', 'torch.nn.intrinsic.quantized.modules.bn_relu', 'torch.nn.intrinsic.quantized.modules', 'torch.nn.intrinsic.quantized', 'torch.nn.quantized.dynamic.modules.linear', 'torch.nn.quantized.dynamic.modules.rnn', 'torch.nn.quantized.dynamic.modules', 'torch.nn.quantized.dynamic', 'torch.nn.intrinsic.quantized.dynamic.modules.linear_relu', 'torch.nn.intrinsic.quantized.dynamic.modules', 'torch.nn.intrinsic.quantized.dynamic', 'torch.nn.quantized._reference.modules.utils', 'torch.nn.quantized._reference.modules.linear', 'torch.nn.quantized._reference.modules.conv', 'torch.nn.quantized._reference.modules', 'torch.nn.quantized._reference', 'torch.ao.quantization.stubs', 'torch.ao.quantization.quantization_mappings', 'torch.quantization.quantization_mappings', 'torch.ao.quantization.qconfig', 'torch.quantization.qconfig', 'torch.ao.quantization.quantize', 'torch.ao.quantization', 'torch.quantization.quantize', 'torch.quantization.fake_quantize', 'torch.quantization.fuse_modules', 'torch.quantization.stubs', 'torch.quantization.quant_type', 'torch.ao.quantization.quantize_jit', 'torch.quantization.quantize_jit', 'torch.quantization', 'torch.utils.data.sampler', 'torch.utils.data._typing', 'torch.utils.data.dataset', 'torch.utils.data._utils.signal_handling', 'torch.utils.data._utils.worker', 'torch.utils.data._utils.pin_memory', 'torch.utils.data._utils.collate', 'torch.utils.data._utils.fetch', 'torch.utils.data._utils', 'torch.utils.data.dataloader', 'torch.utils.data.distributed', 'torch.utils.data._decorator', 'torch.utils.data.graph', 'torch.utils.data.sharding', 'torch.utils.data.backward_compatibility', 'torch.utils.data.communication.eventloop', 'torch.utils.data.communication.iter', 'torch.utils.data.communication.messages', 'torch.utils.data.communication.protocol', 'torch.utils.data.communication.queue', 'torch.utils.data.communication', 'dill.info', '_pyio', 'dill.settings', 'dill._dill', 'dill.source', 'dill.temp', 'dill.pointers', 'dill.detect', 'dill.objtypes', 'dill', 'torch.utils.data.datapipes.iter.callable', 'torch.utils.data.datapipes.iter.combinatorics', 'torch.utils.data.datapipes.iter.combining', 'torch.utils.data.datapipes.utils', 'torch.utils.data.datapipes.utils.common', 'torch.utils.data.datapipes.iter.filelister', 'torch.utils.data.datapipes.iter.fileloader', 'torch.utils.data.datapipes.iter.grouping', 'torch.utils.data.datapipes.iter.httpreader', 'torch.utils.data.datapipes.iter.linereader', 'torch.utils.data.datapipes.utils.decoder', 'torch.utils.data.datapipes.iter.routeddecoder', 'pytz.exceptions', 'pytz.lazy', 'pytz.tzinfo', 'pytz.tzfile', 'pytz', 'pandas._typing', 'pandas._libs.tslibs.dtypes', 'pandas._libs.tslibs.base', 'pandas._libs.tslibs.np_datetime', 'pandas._libs.tslibs.nattype', 'dateutil.zoneinfo', 'pandas._libs.tslibs.timezones', 'pandas._libs.tslibs.ccalendar', 'pandas._libs.tslibs.tzconversion', 'pandas._config.config', 'pandas._config.dates', 'pandas._config.display', 'pandas._config', 'pandas._config.localization', 'pandas._libs.tslibs.strptime', 'pandas._libs.tslibs.fields', 'pandas._libs.tslibs.timedeltas', 'pandas._libs.tslibs.timestamps', 'dateutil.easter', 'dateutil._common', 'dateutil.relativedelta', 'pandas._libs.properties', 'pandas._libs.tslibs.offsets', 'dateutil.parser._parser', 'dateutil.parser.isoparser', 'dateutil.parser', 'pandas._libs.tslibs.parsing', 'pandas._libs.tslibs.conversion', 'pandas._libs.tslibs.period', 'pandas._libs.tslibs.vectorized', 'pandas._libs.tslibs', 'pandas._libs.ops_dispatch', 'pandas._libs.missing', 'pandas._libs.hashtable', 'pandas._libs.algos', 'pandas._libs.interval', 'pandas._libs', 'pandas.util._decorators', 'pandas.core', 'pandas.core.util', 'pandas._libs.tslib', 'pandas._libs.lib', 'pandas._libs.hashing', 'pandas.core.dtypes', 'pandas.util._exceptions', 'pandas.errors', 'pandas.core.dtypes.generic', 'pandas.core.dtypes.base', 'pandas.core.dtypes.inference', 'pandas.core.dtypes.dtypes', 'pandas.core.dtypes.common', 'pandas.core.util.hashing', 'pandas.util', 'pandas.util.version', 'pandas.compat.numpy', 'pandas.compat.pyarrow', 'pandas.compat', 'pandas.core.config_init', 'pandas.core.dtypes.missing', 'pandas.util._validators', 'pandas.core.dtypes.cast', 'pandas.core.dtypes.concat', 'pandas.core.array_algos', 'pandas.core.common', 'pandas.core.construction', 'pandas.core.array_algos.take', 'pandas.core.indexers.utils', 'pandas.core.indexers', 'pandas.core.algorithms', 'pandas.compat.numpy.function', 'pandas.core.roperator', 'pandas._libs.ops', 'pandas.core.computation', 'pandas.compat._optional', 'pandas.core.computation.check', 'pandas.core.computation.expressions', 'pandas.core.ops.missing', 'pandas.core.ops.dispatch', 'pandas.core.ops.invalid', 'pandas.core.ops.array_ops', 'pandas.core.ops.common', 'pandas.core.ops.docstrings', 'pandas.core.ops.mask_ops', 'pandas.core.ops.methods', 'pandas.core.ops', 'pandas.core.arraylike', 'pandas.core.missing', 'pandas.core.array_algos.quantile', 'pandas.core.sorting', 'pandas.core.arrays.base', 'pandas.core.nanops', 'pandas.core.array_algos.masked_reductions', 'pandas.core.arrays.masked', 'pandas.core.arrays.boolean', 'pandas._libs.arrays', 'pandas.core.accessor', 'pandas.core.array_algos.transforms', 'pandas.core.arrays._mixins', 'pandas.core.base', 'pandas.core.strings.accessor', 'pandas.core.strings.base', 'pandas.core.strings', 'pandas.core.strings.object_array', 'pandas.io', 'pandas.io.formats', 'pandas.io.formats.console', 'pandas.core.arrays.categorical', 'pandas.tseries', 'pandas.tseries.frequencies', 'pandas.core.arrays.datetimelike', 'pandas.core.arrays._ranges', 'pandas.core.arrays.numeric', 'pandas.core.tools', 'pandas.core.tools.numeric', 'pandas.core.arrays.integer', 'pandas.tseries.offsets', 'pandas.core.arrays.datetimes', 'pandas.core.arrays.floating', 'pandas.core.indexes', 'pandas._libs.index', 'pandas._libs.join', 'pandas.core.array_algos.putmask', 'pandas.libs.sparse', 'pandas.core.arrays.sparse.dtype', 'pandas.io.formats.printing', 'pandas.core.arrays.sparse.array', 'pandas.core.arrays.sparse.accessor', 'pandas.core.arrays.sparse', 'pandas.core.indexes.frozen', 'pandas.core.indexes.base', 'pandas.core.arrays.interval', 'pandas.core.arrays.numpy', 'pandas.core.arrays.period', 'pandas.core.arrays.string_', 'pandas.core.arrays.string_arrow', 'pandas.core.arrays.timedeltas', 'pandas.core.arrays', 'pandas.core.flags', 'pandas._libs.reduction', 'pandas.core.apply', 'pandas._libs.indexing', 'pandas.core.indexes.extension', 'pandas.core.indexes.category', 'pandas.core.indexes.numeric', 'pandas.core.indexes.range', 'pandas.core.tools.timedeltas', 'pandas.core.indexes.datetimelike', 'pandas.core.tools.times', 'pandas.core.indexes.datetimes', 'pandas.core.indexes.multi', 'pandas.core.indexes.timedeltas', 'pandas.core.indexes.interval', 'pandas.core.indexes.period', 'pandas.core.indexes.api', 'pandas.core.indexing', 'pandas.core.array_algos.replace', 'pandas.core.reshape', 'pandas._libs.internals', 'pandas._libs.writers', 'pandas.core.internals.blocks', 'pandas.core.internals.api', 'pandas.core.internals.base', 'pandas.core.internals.array_manager', 'pandas.core.internals.ops', 'pandas.core.internals.managers', 'pandas.core.internals.concat', 'pandas.core.inte [2022-05-31T14:24:23.474628Z] dc5b74e6 || rnals', 'pandas.core.reshape.concat', 'mmap', 'pandas.core.shared_docs', 'pandas.io.common', 'pandas.io.formats.format', 'pandas.core.describe', 'pandas.core.internals.construction', 'pandas.core.sample', 'pandas._libs.window', 'pandas._libs.window.aggregations', 'pandas.libs.window.indexers', 'pandas.core.indexers.objects', 'pandas.core.util.numba', 'pandas.core.window.common', 'pandas.core.window.doc', 'pandas.core.window.numba_', 'pandas.core.window.online', 'pandas.core._numba', 'pandas.core._numba.executor', 'pandas.core.window.rolling', 'pandas.core.window.ewm', 'pandas.core.window.expanding', 'pandas.core.window', 'pandas.core.generic', 'pandas.core.reshape.util', 'pandas.core.reshape.melt', 'pandas._libs.reshape', 'pandas.core.indexes.accessors', 'pandas.arrays', 'pandas.core.tools.datetimes', 'pandas.io.formats.info', 'pandas.plotting._core', 'pandas.plotting._misc', 'pandas.plotting', 'pandas.core.series', 'pandas.core.frame', 'pandas.core.groupby.base', 'pandas.libs.groupby', 'pandas.core.groupby.numba', 'pandas.core.groupby.categorical', 'pandas.core.groupby.grouper', 'pandas.core.groupby.ops', 'pandas.core.groupby.indexing', 'pandas.core.groupby.groupby', 'pandas.core.groupby.generic', 'pandas.core.groupby', 'pandas.core.api', 'pandas.tseries.api', 'pandas.core.computation.common', 'pandas.core.computation.align', 'pandas.compat.chainmap', 'pandas.core.computation.scope', 'pandas.core.computation.ops', 'pandas.core.computation.parsing', 'pandas.core.computation.expr', 'pandas.core.computation.engines', 'pandas.core.computation.eval', 'pandas.core.computation.api', 'pandas.core.reshape.merge', 'pandas.core.reshape.pivot', 'pandas.core.reshape.reshape', 'pandas.core.reshape.tile', 'pandas.core.reshape.api', 'pandas.api.extensions', 'pandas.api.indexers', 'pandas.core.dtypes.api', 'pandas.api.types', 'pandas.api', 'pandas._testing._random', 'pandas._testing.contexts', 'pandas._testing._io', 'pandas._testing._warnings', 'pandas._libs.testing', 'pandas._testing.asserters', 'pandas._testing.compat', 'pandas._testing', 'pandas.testing', 'pandas.util._print_versions', 'pandas.io.clipboards', 'pandas._libs.parsers', 'pandas.io.excel._util', 'pandas.io.date_converters', 'pandas.io.parsers.base_parser', 'pandas.io.parsers.arrow_parser_wrapper', 'pandas.io.parsers.c_parser_wrapper', 'pandas.io.parsers.python_parser', 'pandas.io.parsers.readers', 'pandas.io.parsers', 'pandas.io.excel._odfreader', 'pandas.io.excel._openpyxl', 'pandas.io.excel._pyxlsb', 'pandas.io.excel._xlrd', 'pandas.io.excel._base', 'pandas._libs.json', 'pandas.io.formats._color_data', 'pandas.io.formats.css', 'pandas.io.formats.excel', 'pandas.io.excel._odswriter', 'pandas.io.excel._xlsxwriter', 'pandas.io.excel._xlwt', 'pandas.io.excel', 'pandas.io.feather_format', 'pandas.io.gbq', 'pandas.io.html', 'pandas.io.json._normalize', 'pandas.io.json._table_schema', 'pandas.io.json._json', 'pandas.io.json', 'pandas.io.orc', 'pandas.io.parquet', 'pandas.compat.pickle_compat', 'pandas.io.pickle', 'pandas.core.computation.pytables', 'pandas.io.pytables', 'pandas.io.sas.sasreader', 'pandas.io.sas', 'pandas.io.spss', 'pandas.io.sql', 'pandas.io.stata', 'pandas.io.xml', 'pandas.io.api', 'pandas.util._tester', 'pandas._version', 'pandas', 'torch.utils.data.datapipes.iter.selecting', 'torch.utils.data.datapipes.iter.streamreader', 'torch.utils.data.datapipes.iter.tararchivereader', 'torch.utils.data.datapipes.iter.ziparchivereader', 'torch.utils.data.datapipes.iter.utils', 'torch.utils.data.datapipes.iter', 'torch.utils.data.datapipes.map.callable', 'torch.utils.data.datapipes.map.combining', 'torch.utils.data.datapipes.map', 'torch.utils.data.datapipes.dataframe.structures', 'torch.utils.data.datapipes.dataframe.dataframes', 'torch.utils.data.datapipes.dataframe.datapipes', 'torch.utils.data.datapipes.dataframe', 'torch.utils.data.datapipes', 'torch.utils.data.dataloader_experimental', 'torch.utils.data', 'torch.config', 'torch.future', 'torch.profiler.profiler', 'torch.profiler', 'torch._tensor_docs', 'torch._storage_docs', 'torch.quasirandom', 'torch.multiprocessing._atfork', 'torch._lobpcg', 'torch.utils.dlpack', 'torch', 'cv2.Error', 'cv2.cuda', 'cv2.detail', 'cv2.dnn', 'cv2.fisheye', 'cv2.flann', 'cv2.gapi.core', 'cv2.gapi.core.cpu', 'cv2.gapi.core.fluid', 'cv2.gapi.core.ocl', 'cv2.gapi.ie', 'cv2.gapi.ie.detail', 'cv2.gapi.onnx', 'cv2.gapi.own', 'cv2.gapi.own.detail', 'cv2.gapi.render', 'cv2.gapi.render.ocv', 'cv2.gapi.streaming', 'cv2.gapi.video', 'cv2.gapi.wip', 'cv2.gapi.wip.draw', 'cv2.gapi.wip.gst', 'cv2.gapi.wip.onevpl', 'cv2.ipp', 'cv2.ml', 'cv2.ocl', 'cv2.ogl', 'cv2.parallel', 'cv2.samples', 'cv2.segmentation', 'cv2.utils.fs', 'cv2.videoio_registry', 'cv2.cv2', 'cv2.misc.version', 'cv2.version', 'cv2.load_config_py3', 'cv2.qt', 'cv2.gapi', 'cv2.utils', 'cv2.misc', 'cv2.data', 'cv2.mat_wrapper', 'cv2', '_distutils_hack', '_distutils_hack.override', 'distutils', 'distutils.debug', 'distutils.errors', 'getopt', 'distutils.fancy_getopt', 'distutils.dep_util', 'distutils.log', 'distutils.spawn', 'distutils.util', 'distutils.dist', 'distutils.dir_util', 'distutils.file_util', 'distutils.archive_util', 'distutils.cmd', 'distutils.config', 'distutils.extension', 'distutils.core', 'setuptools._deprecation_warning', 'setuptools.version', 'distutils.filelist', 'setuptools.monkey', 'setuptools.extension', 'distutils.command', 'distutils.version', 'setuptools.extern', 'setuptools._vendor', 'setuptools._vendor.packaging.about', 'setuptools._vendor.packaging', 'setuptools.extern.packaging', 'setuptools._vendor.ordered_set', 'setuptools.extern.ordered_set', 'concurrent.futures.thread', 'setuptools._vendor.more_itertools.recipes', 'setuptools._vendor.more_itertools.more', 'setuptools._vendor.more_itertools', 'setuptools.extern.more_itertools', 'distutils.command.bdist', 'setuptools.command', 'setuptools.windows_support', 'setuptools.extern.packaging._structures', 'setuptools.extern.packaging.version', 'setuptools._vendor.packaging._manylinux', 'setuptools._vendor.packaging._musllinux', 'setuptools.extern.packaging.tags', 'setuptools.extern.packaging.utils', 'setuptools.extern.packaging.specifiers', 'setuptools.config', 'setuptools.dist', 'setuptools.py34compat', 'setuptools._imp', 'setuptools.depends', 'distutils.ccompiler', 'setuptools.msvc', 'setuptools', 'torch._appdirs', 'torch.utils.file_baton', 'torch.utils._cpp_extension_versioner', 'torch.utils.hipify.version', 'torch.utils.hipify', 'torch.utils.hipify.constants', 'torch.utils.hipify.cuda_to_hip_mappings', 'torch.utils.hipify.hipify_python', 'distutils.sysconfig', 'distutils.command.build_ext', 'cython.parallel', 'Cython.Shadow', 'Cython', '_sysconfigdatalinux_x86_64-linux-gnu', 'Cython.Distutils.old_build_ext', 'Cython.Distutils.build_ext', 'Cython.Distutils.extension', 'Cython.Distutils', 'Cython.Compiler', 'Cython.Utils', 'Cython.Compiler.DebugFlags', 'Cython.Compiler.Options', 'Cython.Compiler.Errors', 'Cython.Compiler.StringEncoding', '_cython_0_29_30', 'Cython.Plex.Actions', 'Cython.Plex.Transitions', 'Cython.Plex.Machines', 'Cython.Plex.DFA', 'Cython.Plex.Errors', 'Cython.Plex.Regexps', 'Cython.Plex.Lexicons', 'Cython.Plex.Scanners', 'Cython.Plex', 'Cython.Compiler.Lexicon', 'Cython.Compiler.Future', 'Cython.Compiler.Scanning', 'Cython.Compiler.Naming', 'cython', 'Cython.Compiler.Version', 'Cython.StringIOTree', 'Cython.Compiler.Code', 'Cython.Compiler.PyrexTypes', 'Cython.Compiler.TypeSlots', 'Cython.Compiler.Symtab', 'Cython.Compiler.Main', 'setuptools.command.build_ext', 'torch.utils.cpp_extension', 'mmcv.utils.parrots_wrapper', 'mmcv.utils.env', 'torch.utils.model_zoo', 'mmcv.utils.hub', 'mmcv.utils.logging', 'mmcv.utils.parrots_jit', 'mmcv.utils.registry', 'mmcv.utils.trace', 'mmcv.utils', 'mmcv.fileio.file_client', 'mmcv.fileio.handlers.base', 'mmcv.fileio.handlers.json_handler', 'mmcv.fileio.handlers.pickle_handler', 'yaml.error', 'yaml.tokens', 'yaml.events', 'yaml.nodes', 'yaml.reader', 'yaml.scanner', 'yaml.parser', 'yaml.composer', 'yaml.constructor', 'yaml.resolver', 'yaml.loader', 'yaml.emitter', 'yaml.serializer', 'yaml.representer', 'yaml.dumper', 'yaml._yaml', 'yaml.cyaml', 'yaml', 'mmcv.fileio.handlers.yaml_handler', 'mmcv.fileio.handlers', 'mmcv.fileio.io', 'mmcv.fileio.parse', 'mmcv.fileio', 'mmcv.image.colorspace', 'PIL._version', 'PIL', 'defusedxml.common', 'defusedxml', 'xml.etree', 'xml.etree.ElementPath', '_elementtree', 'xml.etree.ElementTree', 'defusedxml.ElementTree', 'PIL.ImageMode', 'PIL.TiffTags', 'PIL._binary', 'PIL._util', 'PIL._imaging', 'cffi.lock', 'cffi.error', 'cffi.model', 'cffi.api', 'cffi', 'PIL.Image', 'PIL.ImageOps', 'mmcv.image.io', 'mmcv.image.geometric', 'mmcv.image.misc', 'mmcv.image.photometric', 'mmcv.image', 'mmcv.version', 'mmcv.video.io', 'mmcv.video.optflow', 'mmcv.video.processing', 'mmcv.video', 'mmcv.visualization.color', 'mmcv.visualization.image', 'mmcv.visualization.optflow', 'mmcv.visualization', 'mmcv', 'mmcv.parallel.data_container', 'mmcv.parallel.collate', 'mmcv.parallel._functions', 'mmcv.parallel.scatter_gather', 'mmcv.parallel.data_parallel', 'mmcv.parallel.distributed', 'mmcv.parallel.registry', 'mmcv.parallel.utils', 'mmcv.parallel', 'mmdet.version', 'mmdet', 'mmdet.datasets', 'mmcv.runner.dist_utils', 'mmcv.runner.base_module', 'torchvision._internally_replaced_utils', 'torchvision.extension', 'torchvision.models.alexnet', 'torchvision.models.resnet', 'torchvision.models.vgg', 'torchvision.models.squeezenet', 'torchvision.models.inception', 'torch.utils.checkpoint', 'torchvision.models.densenet', 'torchvision.models.googlenet', 'torchvision.ops._box_convert', 'torchvision.ops.boxes', 'torchvision.ops.deform_conv', 'torchvision.ops._utils', 'torchvision.ops.roi_align', 'torchvision.ops.roi_pool', 'torchvision.ops.ps_roi_align', 'torchvision.ops.ps_roi_pool', 'torchvision.ops.poolers', 'torchvision.ops.feature_pyramid_network', 'torchvision.ops.focal_loss', 'torch.fx._compatibility', 'torch.fx.immutable_collections', 'torch.fx.operator_schemas', 'torch.fx.node', 'torch.fx._pytree', 'torch.fx.graph', 'torch.fx.graph_module', 'torch.fx.proxy', 'torch.fx._symbolic_trace', 'torch.fx.interpreter', 'torch.fx.subgraph_rewriter', 'torch.fx', 'torchvision.ops.stochastic_depth', 'torchvision.ops._register_onnx_ops', 'torch.onnx.utils', 'torch.onnx.symbolic_helper', 'torch.onnx.symbolic_opset9', 'torch.onnx.symbolic_opset11', 'torch.onnx.symbolic_opset7', 'torch.onnx.symbolic_opset8', 'torch.onnx.symbolic_opset10', 'torch.onnx.symbolic_opset12', 'torch.onnx.symbolic_opset13', 'torch.onnx.symbolic_opset14', 'torch.onnx.symbolic_registry', 'torchvision.ops', 'torchvision.ops.misc', 'torchvision.models._utils', 'torchvision.models.mobilenetv2', 'torchvision.models.mobilenetv3', 'torchvision.models.mobilenet', 'torchvision.models.mnasnet', 'torchvision.models.shufflenetv2', 'torchvision.models.efficientnet', 'torchvision.models.regnet', 'torchvision.models.segmentation._utils', 'torchvision.models.segmentation.deeplabv3', 'torchvision.models.segmentation.fcn', 'torchvision.models.segmentation.lraspp', 'torchvision.models.segmentation.segmentation', 'torchvision.models.segmentation', 'torchvision.models.detection._utils', 'torchvision.models.detection.image_list', 'torchvision.models.detection.anchor_utils', 'torchvision.models.detection.generalized_rcnn', 'torchvision.models.detection.rpn', 'torchvision.models.detection.roi_heads', 'torchvision.models.detection.transform', 'torchvision.models.detection.backbone_utils', 'torchvision.models.detection.faster_rcnn', 'torchvision.models.detection.mask_rcnn', 'torchvision.models.detection.keypoint_rcnn', 'torchvision.models.detection.retinanet', 'torchvision.models.detection.ssd', 'torchvision.models.detection.ssdlite', 'torchvision.models.detection', 'torchvision.models.video.resnet', 'torchvision.models.video', 'torchvision.models.quantization.utils', 'torchvision.models.quantization.mobilenetv2', 'torchvision.models.quantization.mobilenetv3', 'torchvision.models.quantization.mobilenet', 'torchvision.models.quantization.resnet', 'torchvision.models.quantization.googlenet', 'torchvision.models.quantization.inception', 'torchvision.models.quantization.shufflenetv2', 'torchvision.models.quantization', 'torchvision.models.feature_extraction', 'torchvision.models', 'torchvision.datasets.vision', 'torchvision.datasets.utils', 'torchvision.datasets.lsun', 'torchvision.datasets.folder', 'torchvision.datasets.coco', 'torchvision.datasets.cifar', 'torchvision.datasets.stl10', 'torchvision.datasets.mnist', 'torchvision.datasets.svhn', 'torchvision.datasets.phototour', 'PIL.ImageFilter', 'PIL.ImageStat', 'PIL.ImageEnhance', 'torchvision.transforms.functional_pil', 'torchvision.transforms.functional_tensor', 'torchvision.transforms.functional', 'torchvision.transforms.transforms', 'torchvision.transforms.autoaugment', 'torchvision.transforms', 'torchvision.datasets.fakedata', 'torchvision.datasets.semeion', 'torchvision.datasets.omniglot', 'torchvision.datasets.sbu', 'html.entities', 'html', '_markupbase', 'html.parser', 'torchvision.datasets.flickr', 'torchvision.datasets.voc', 'torchvision.datasets.cityscapes', 'torchvision.datasets.imagenet', 'torchvision.datasets.caltech', 'torchvision.datasets.celeba', 'torchvision.datasets.widerface', 'torchvision.datasets.sbd', 'torchvision.datasets.usps', 'fractions', 'torchvision.io._video_opt', 'torchvision.io.video', 'torchvision.io.image', 'torchvision.io', 'torchvision.datasets.video_utils', 'torchvision.datasets.kinetics', 'torchvision.datasets.hmdb51', 'torchvision.datasets.ucf101', 'torchvision.datasets.places365', 'torchvision.datasets.kitti', 'torchvision.datasets.inaturalist', 'torchvision.datasets.lfw', 'torchvision.datasets', 'PIL.ImageColor', 'PIL._imagingft', 'PIL.ImageFont', 'PIL.ImageDraw', 'torchvision.utils', 'torchvision.version', 'torchvision', 'mmcv.runner.checkpoint', 'mmcv.runner.hooks.hook', 'mmcv.runner.hooks.checkpoint', 'mmcv.runner.hooks.closure', 'mmcv.runner.hooks.ema', 'mmcv.runner.hooks.logger.base', 'mmcv.runner.hooks.logger.dvclive', 'mmcv.runner.hooks.logger.mlflow', 'mmcv.runner.hooks.logger.neptune', 'mmcv.runner.hooks.logger.pavi', 'mmcv.runner.hooks.logger.tensorboard', 'mmcv.runner.hooks.logger.text', 'mmcv.runner.hooks.logger.wandb', 'mmcv.runner.hooks.logger', 'mmcv.runner.hooks.evaluation', 'mmcv.runner.hooks.iter_timer', 'mmcv.runner.hooks.lr_updater', 'mmcv.runner.hooks.memory', 'mmcv.runner.hooks.momentum_updater', 'mmcv.runner.fp16_utils', 'mmcv.runner.hooks.optimizer', 'mmcv.runner.hooks.profiler', 'mmcv.runner.hooks.sampler_seed', 'mmcv.runner.hooks.sync_buffer', 'mmcv.runner.hooks', 'mmcv.runner.log_buffer', 'mmcv.runner.priority', 'mmcv.runner.utils', 'mmcv.runner.base_runner', 'mmcv.runner.builder', 'mmcv.runner.default_constructor', 'mmcv.runner.epoch_based_runner', 'mmcv.runner.iter_based_runner', 'mmcv.runner.optimizer.builder', 'mmcv.utils.ext_loader', 'mmcv.runner.optimizer.default_constructor', 'mmcv.runner.optimizer', 'mmcv.runner', 'mmdet.datasets.samplers.distributed_sampler', 'mmdet.datasets.samplers.group_sampler', 'mmdet.datasets.samplers.infinite_sampler', 'mmdet.datasets.samplers', 'mmdet.datasets.builder', 'pycocotools', '_cython_0_29_21', 'pycocotools._mask', 'pycocotools.mask', 'terminaltables.terminal_io', 'terminaltables.width_and_alignment', 'terminaltables.build', 'terminaltables.base_table', 'terminaltables.ascii_table', 'terminaltables.github_table', 'encodings.cp437', 'terminaltables.other_tables', 'terminaltables', 'mmdet.core.anchor.builder', 'mmdet.core.anchor.anchor_generator', 'mmdet.core.anchor.point_generator', 'mmdet.core.anchor.utils', 'mmdet.core.anchor', 'mmdet.core.bbox.builder', 'mmdet.core.bbox.iou_calculators.builder', 'mmdet.core.bbox.iou_calculators.iou2d_calculator', 'mmdet.core.bbox.iou_calculators', 'mmdet.utils.collect_env', 'mmdet.utils.logger', 'mmdet.utils.misc', 'mmdet.utils.setup_env', 'mmdet.utils', 'mmdet.utils.util_mixins', 'mmdet.core.bbox.assigners.assign_result', 'mmdet.core.bbox.assigners.base_assigner', 'mmdet.core.bbox.assigners.max_iou_assigner', 'mmdet.core.bbox.assigners.approx_max_iou_assigner', 'mmdet.core.bbox.assigners.atss_assigner', 'mmdet.core.bbox.assigners.center_region_assigner', 'mmdet.core.bbox.assigners.grid_assigner', 'mmdet.core.bbox.match_costs.builder', 'mmdet.core.bbox.transfor [2022-05-31T14:24:23.474628Z] dc5b74e6 || ms', 'mmdet.core.bbox.match_costs.match_cost', 'mmdet.core.bbox.match_costs', 'scipy._lib._testutils', 'scipy._lib', 'scipy._lib.deprecation', 'scipy.config', 'scipy.version', 'scipy._distributor_init', 'scipy._lib._pep440', 'scipy._lib._ccallback_c', 'scipy._lib._ccallback', 'scipy', 'scipy._lib._util', 'scipy.sparse._sputils', 'scipy.sparse._base', 'scipy.sparse._sparsetools', 'scipy.sparse._data', 'scipy.sparse._index', 'scipy.sparse._compressed', 'scipy.sparse._csr', 'scipy.sparse._csc', 'scipy.sparse._csparsetools', 'scipy.sparse._lil', 'scipy.sparse._dok', 'scipy.sparse._coo', 'scipy.sparse._dia', 'scipy.sparse._bsr', 'scipy.sparse._construct', 'scipy.sparse._extract', 'scipy.sparse._matrix_io', 'scipy.sparse._arrays', 'scipy.sparse.csgraph._laplacian', '_cython_0_29_27', 'scipy.sparse.csgraph._tools', 'scipy.sparse.csgraph._validation', 'scipy.sparse.csgraph._shortest_path', 'scipy.sparse.csgraph._traversal', 'scipy.sparse.csgraph._min_spanning_tree', 'scipy.sparse.csgraph._flow', 'scipy.sparse.csgraph._matching', 'scipy.sparse.csgraph._reordering', 'scipy.sparse.csgraph', 'scipy.sparse.base', 'scipy.sparse.bsr', 'scipy.sparse.compressed', 'scipy.sparse.construct', 'scipy.sparse.coo', 'scipy.sparse.csc', 'scipy.sparse.csr', 'scipy.sparse.data', 'scipy.sparse.dia', 'scipy.sparse.dok', 'scipy.sparse.extract', 'scipy.sparse.lil', 'scipy.sparse.sparsetools', 'scipy.sparse.sputils', 'scipy.sparse', 'scipy.sparse.linalg._isolve._iterative', 'scipy.sparse.linalg._interface', 'scipy.sparse.linalg._isolve.utils', 'scipy._lib.decorator', 'scipy._lib._threadsafety', 'scipy.sparse.linalg._isolve.iterative', 'scipy.sparse.linalg._isolve.minres', 'scipy.linalg._fblas', 'scipy.linalg.blas', 'scipy.linalg._flapack', 'scipy.linalg.lapack', 'scipy.linalg._misc', 'scipy.linalg._cythonized_array_utils', 'scipy.linalg._flinalg', 'scipy.linalg._flinalg_py', 'scipy.linalg._decomp', 'scipy.linalg._decomp_svd', 'scipy.linalg._solve_toeplitz', 'scipy.linalg._basic', 'scipy.linalg._decomp_lu', 'scipy.linalg._decomp_ldl', 'scipy.linalg._decomp_cholesky', 'scipy.linalg._decomp_qr', 'scipy.linalg._decomp_qz', 'scipy.linalg._decomp_schur', 'scipy.linalg._decomp_polar', 'scipy.linalg._special_matrices', 'scipy.linalg._expm_frechet', 'scipy.linalg._matfuncs_sqrtm_triu', 'scipy.linalg._matfuncs_sqrtm', 'scipy.linalg._matfuncs', 'scipy.linalg._solvers', 'scipy.linalg._procrustes', 'scipy.linalg.cython_blas', 'scipy.linalg.cython_lapack', 'scipy.linalg._decomp_update', 'scipy.linalg._sketches', 'scipy.linalg._decomp_cossin', 'scipy.linalg.decomp', 'scipy.linalg.decomp_cholesky', 'scipy.linalg.decomp_lu', 'scipy.linalg.decomp_qr', 'scipy.linalg.decomp_svd', 'scipy.linalg.decomp_schur', 'scipy.linalg.basic', 'scipy.linalg.misc', 'scipy.linalg.special_matrices', 'scipy.linalg.flinalg', 'scipy.linalg.matfuncs', 'scipy.linalg', 'scipy.sparse.linalg._isolve._gcrotmk', 'scipy.sparse.linalg._isolve.lgmres', 'scipy.sparse.linalg._isolve.lsqr', 'scipy.sparse.linalg._isolve.lsmr', 'scipy.sparse.linalg._isolve.tfqmr', 'scipy.sparse.linalg._isolve', 'scipy.sparse.linalg._dsolve._superlu', 'scipy.sparse.linalg._dsolve.linsolve', 'scipy.sparse.linalg._dsolve._add_newdocs', 'scipy.sparse.linalg._dsolve', 'scipy.sparse.linalg._eigen.arpack._arpack', 'scipy.sparse.linalg._eigen.arpack.arpack', 'scipy.sparse.linalg._eigen.arpack', 'scipy.sparse.linalg._eigen.lobpcg.lobpcg', 'scipy.sparse.linalg._eigen.lobpcg', 'scipy.sparse.linalg._eigen._svds', 'scipy.sparse.linalg._eigen', 'scipy.sparse.linalg._expm_multiply', 'scipy.sparse.linalg._matfuncs', 'scipy.sparse.linalg._onenormest', 'scipy.sparse.linalg._norm', 'scipy.sparse.linalg.isolve', 'scipy.sparse.linalg.dsolve', 'scipy.sparse.linalg.interface', 'scipy.sparse.linalg.eigen', 'scipy.sparse.linalg.matfuncs', 'scipy.sparse.linalg', 'scipy.optimize._minpack2', 'scipy.optimize._linesearch', 'scipy.optimize._group_columns', 'scipy.optimize._numdiff', 'scipy.optimize._hessian_update_strategy', 'scipy.optimize._differentiable_functions', 'scipy.optimize._optimize', 'scipy.optimize._trustregion', 'scipy.optimize._trustregion_dogleg', 'scipy.optimize._trustregion_ncg', 'scipy._lib.messagestream', 'scipy.optimize._trlib._trlib', 'scipy.optimize._trlib', 'scipy.optimize._trustregion_krylov', 'scipy.optimize._trustregion_exact', 'numpy.testing._private', 'numpy.linalg.lapack_lite', 'numpy.testing._private.utils', 'numpy.testing._private.extbuild', 'numpy.testing._private.decorators', 'numpy.testing._private.nosetester', 'numpy.testing', 'scipy.optimize._constraints', 'scipy.optimize._trustregion_constr.projections', 'scipy.optimize._trustregion_constr.qp_subproblem', 'scipy.optimize._trustregion_constr.equality_constrained_sqp', 'scipy.optimize._trustregion_constr.canonical_constraint', 'scipy.optimize._trustregion_constr.tr_interior_point', 'scipy.optimize._trustregion_constr.report', 'scipy.optimize._trustregion_constr.minimize_trustregion_constr', 'scipy.optimize._trustregion_constr', 'scipy.optimize._lbfgsb', 'scipy.optimize._lbfgsb_py', '_moduleTNC', 'scipy.optimize._moduleTNC', 'scipy.optimize._tnc', 'scipy.optimize._cobyla', 'scipy.optimize._cobyla_py', 'scipy.optimize._slsqp', 'scipy.optimize._slsqp_py', 'scipy.optimize._minimize', 'scipy.optimize._minpack', 'scipy.optimize._lsq.common', 'scipy.optimize._lsq.trf', 'scipy.optimize._lsq.dogbox', 'scipy.optimize._lsq.least_squares', 'scipy.optimize._lsq.givens_elimination', 'scipy.optimize._lsq.trf_linear', 'scipy.optimize._lsq.bvls', 'scipy.optimize._lsq.lsq_linear', 'scipy.optimize._lsq', 'scipy.optimize._minpack_py', 'scipy.optimize._spectral', 'scipy.optimize._nonlin', 'scipy.optimize._root', 'scipy.optimize._zeros', 'scipy.optimize._zeros_py', 'scipy.optimize._root_scalar', 'scipy.optimize.nnls', 'scipy.optimize._nnls', 'scipy.optimize._basinhopping', 'scipy.optimize._highs', 'scipy.optimize._highs.cython.src._highs_wrapper', 'scipy.optimize._highs._highs_wrapper', 'scipy.optimize._highs.cython.src._highs_constants', 'scipy.optimize._highs._highs_constants', 'scipy.optimize._linprog_highs', 'scipy.linalg._interpolative', 'scipy.linalg._interpolative_backend', 'scipy.linalg.interpolative', 'scipy.optimize._remove_redundancy', 'scipy.optimize._linprog_util', 'scipy.optimize._linprog_ip', 'scipy.optimize._linprog_simplex', 'scipy.optimize._bglu_dense', 'scipy.optimize._linprog_rs', 'scipy.optimize._linprog_doc', 'scipy.optimize._linprog', 'scipy.optimize._lsap_module', 'scipy.optimize._lsap', 'scipy.optimize._differentialevolution', 'scipy.spatial._ckdtree', 'scipy.spatial._kdtree', 'scipy.spatial._qhull', 'scipy.spatial._voronoi', 'scipy.spatial._spherical_voronoi', 'scipy.spatial._plotutils', 'scipy.spatial._procrustes', 'scipy.spatial._distance_wrap', 'scipy.spatial._hausdorff', 'scipy.special._sf_error', 'scipy.special._ufuncs_cxx', 'scipy.special._ufuncs', 'scipy.special._specfun', 'scipy.special._orthogonal', 'scipy.special._comb', 'scipy.special._basic', 'scipy.special._logsumexp', 'scipy.special._spfun_stats', 'scipy.special._ellip_harm_2', 'scipy.special._ellip_harm', 'scipy.special._lambertw', 'scipy.special._spherical_bessel', 'scipy.special._add_newdocs', 'scipy.special.add_newdocs', 'scipy.special.basic', 'scipy.special.orthogonal', 'scipy.special.specfun', 'scipy.special.sf_error', 'scipy.special.spfun_stats', 'scipy.special', 'scipy.spatial._distance_pybind', 'scipy.spatial.distance', 'scipy.spatial._geometric_slerp', 'scipy.spatial.ckdtree', 'scipy.spatial.kdtree', 'scipy.spatial.qhull', 'scipy.constants._codata', 'scipy.constants._constants', 'scipy.constants.codata', 'scipy.constants.constants', 'scipy.constants', 'scipy.spatial.transform._rotation_groups', 'scipy.spatial.transform._rotation', 'scipy.spatial.transform._rotation_spline', 'scipy.spatial.transform.rotation', 'scipy.spatial.transform', 'scipy.spatial', 'scipy.optimize._shgo_lib', 'scipy.optimize._shgo_lib.triangulation', 'scipy.optimize._shgo', 'scipy.optimize._dual_annealing', 'scipy.optimize._qap', 'scipy.optimize.cobyla', 'scipy.optimize.lbfgsb', 'scipy.optimize.linesearch', 'scipy.optimize.minpack', 'scipy.optimize.minpack2', 'scipy.optimize.moduleTNC', 'scipy.optimize.nonlin', 'scipy.optimize.optimize', 'scipy.optimize.slsqp', 'scipy.optimize.tnc', 'scipy.optimize.zeros', 'scipy.optimize', 'mmdet.core.bbox.assigners.hungarian_assigner', 'mmdet.core.bbox.assigners.point_assigner', 'mmdet.core.bbox.assigners.region_assigner', 'mmdet.core.bbox.assigners.sim_ota_assigner', 'mmdet.core.bbox.assigners.task_aligned_assigner', 'mmdet.core.bbox.assigners.uniform_assigner', 'mmdet.core.bbox.assigners', 'mmdet.core.bbox.coder.base_bbox_coder', 'mmdet.core.bbox.coder.bucketing_bbox_coder', 'mmdet.core.bbox.coder.delta_xywh_bbox_coder', 'mmdet.core.bbox.coder.distance_point_bbox_coder', 'mmdet.core.bbox.coder.legacy_delta_xywh_bbox_coder', 'mmdet.core.bbox.coder.pseudo_bbox_coder', 'mmdet.core.bbox.coder.tblr_bbox_coder', 'mmdet.core.bbox.coder.yolo_bbox_coder', 'mmdet.core.bbox.coder', 'mmdet.core.bbox.samplers.sampling_result', 'mmdet.core.bbox.samplers.base_sampler', 'mmdet.core.bbox.samplers.combined_sampler', 'mmdet.core.bbox.samplers.random_sampler', 'mmdet.core.bbox.samplers.instance_balanced_pos_sampler', 'mmdet.core.bbox.samplers.iou_balanced_neg_sampler', 'mmdet.core.bbox.samplers.ohem_sampler', 'mmdet.core.bbox.samplers.pseudo_sampler', 'mmcv._ext', 'mmcv.ops.active_rotated_filter', 'mmcv.ops.assign_score_withk', 'mmcv.ops.ball_query', 'mmcv.ops.bbox', 'mmcv.ops.border_align', 'mmcv.ops.box_iou_rotated', 'mmcv.cnn.alexnet', 'mmcv.cnn.bricks.registry', 'mmcv.cnn.bricks.activation', 'mmcv.cnn.utils.flops_counter', 'mmcv.cnn.utils.fuse_conv_bn', 'mmcv.cnn.utils.sync_bn', 'mmcv.cnn.utils.weight_init', 'mmcv.cnn.utils', 'mmcv.cnn.bricks.context_block', 'mmcv.cnn.bricks.conv', 'mmcv.cnn.bricks.conv2d_adaptive_padding', 'mmcv.cnn.bricks.norm', 'mmcv.cnn.bricks.padding', 'mmcv.cnn.bricks.conv_module', 'mmcv.cnn.bricks.conv_ws', 'mmcv.cnn.bricks.depthwise_separable_conv_module', 'mmcv.cnn.bricks.drop', 'mmcv.cnn.bricks.generalized_attention', 'mmcv.cnn.bricks.hsigmoid', 'mmcv.cnn.bricks.hswish', 'mmcv.cnn.bricks.non_local', 'mmcv.cnn.bricks.plugin', 'mmcv.cnn.bricks.scale', 'mmcv.cnn.bricks.swish', 'mmcv.cnn.bricks.upsample', 'mmcv.cnn.bricks.wrappers', 'mmcv.cnn.bricks', 'mmcv.cnn.builder', 'mmcv.cnn.resnet', 'mmcv.cnn.vgg', 'mmcv.cnn', 'mmcv.ops.carafe', 'mmcv.ops.cc_attention', 'mmcv.ops.contour_expand', 'mmcv.ops.convex_iou', 'mmcv.ops.corner_pool', 'mmcv.ops.correlation', 'mmcv.ops.deform_conv', 'mmcv.ops.deform_roi_pool', 'mmcv.ops.deprecated_wrappers', 'mmcv.ops.focal_loss', 'mmcv.ops.furthest_point_sample', 'mmcv.ops.fused_bias_leakyrelu', 'mmcv.ops.gather_points', 'mmcv.ops.knn', 'mmcv.ops.group_points', 'mmcv.ops.info', 'mmcv.ops.iou3d', 'mmcv.ops.masked_conv', 'mmcv.ops.min_area_polygons', 'mmcv.ops.modulated_deform_conv', 'mmcv.ops.multi_scale_deform_attn', 'mmcv.ops.nms', 'mmcv.ops.pixel_group', 'torch.onnx.operators', 'mmcv.ops.point_sample', 'mmcv.ops.points_in_boxes', 'mmcv.ops.points_in_polygons', 'mmcv.ops.points_sampler', 'mmcv.ops.psa_mask', 'mmcv.ops.riroi_align_rotated', 'mmcv.ops.roi_align', 'mmcv.ops.roi_align_rotated', 'mmcv.ops.roi_pool', 'mmcv.ops.roiaware_pool3d', 'mmcv.ops.roipoint_pool3d', 'mmcv.ops.rotated_feature_align', 'mmcv.ops.saconv', 'mmcv.ops.scatter_points', 'mmcv.ops.sync_bn', 'mmcv.ops.three_interpolate', 'mmcv.ops.three_nn', 'mmcv.ops.tin_shift', 'mmcv.ops.upfirdn2d', 'mmcv.ops.voxelize', 'mmcv.ops', 'mmdet.core.bbox.samplers.score_hlr_sampler', 'mmdet.core.bbox.samplers', 'mmdet.core.bbox', 'mmdet.core.data_structures.general_data', 'mmdet.core.data_structures.instance_data', 'mmdet.core.data_structures', 'mmdet.core.evaluation.class_names', 'mmdet.core.evaluation.eval_hooks', 'mmdet.core.evaluation.bbox_overlaps', 'mmdet.core.evaluation.mean_ap', 'mmdet.core.evaluation.recall', 'mmdet.core.evaluation', 'mmdet.core.hook.checkloss_hook', 'mmdet.core.hook.ema', 'mmdet.core.hook.set_epoch_info_hook', 'mmdet.core.utils.dist_utils', 'mmdet.core.mask.mask_target', 'mmdet.core.mask.structures', 'mmdet.core.mask.utils', 'mmdet.core.mask', 'mmdet.core.utils.misc', 'mmdet.core.utils', 'mmdet.core.hook.sync_norm_hook', 'mmdet.core.hook.sync_random_size_hook', 'mmdet.core.hook.yolox_lrupdater_hook', 'mmdet.core.hook.yolox_mode_switch_hook', 'mmdet.core.hook', 'mmdet.core.post_processing.bbox_nms', 'mmdet.core.post_processing.matrix_nms', 'mmdet.core.post_processing.merge_augs', 'mmdet.core.post_processing', 'mmdet.core', 'pycocotools.coco', 'pycocotools.cocoeval', 'mmdet.datasets.api_wrappers.coco_api', 'panopticapi', 'panopticapi.utils', 'panopticapi.evaluation', 'mmdet.datasets.api_wrappers.panoptic_evaluation', 'mmdet.datasets.api_wrappers', 'mmdet.datasets.pipelines.compose', 'mmdet.datasets.pipelines.auto_augment', 'mmdet.datasets.pipelines.formatting', 'mmdet.datasets.pipelines.instaboost', 'mmdet.datasets.pipelines.loading', 'mmdet.datasets.pipelines.test_time_aug', 'mmdet.datasets.pipelines.transforms', 'mmdet.datasets.pipelines', 'mmdet.datasets.custom', 'mmdet.datasets.coco', 'mmdet.datasets.cityscapes', 'mmdet.datasets.coco_panoptic', 'mmdet.datasets.dataset_wrappers', 'mmdet.datasets.deepfashion', 'mmdet.datasets.lvis', 'mmdet.datasets.utils', 'mmdet.models', 'mmdet.models.builder', 'mmdet.models.utils.brick_wrappers', 'mmdet.models.utils.builder', 'mmdet.models.utils.ckpt_convert', 'mmdet.models.utils.conv_upsample', 'mmdet.models.utils.csp_layer', 'mmdet.models.utils.gaussian_target', 'mmdet.models.utils.se_layer', 'mmdet.models.utils.inverted_residual', 'mmdet.models.utils.make_divisible', 'mmdet.models.utils.misc', 'mmdet.models.utils.normed_predictor', 'mmcv.cnn.bricks.transformer', 'mmdet.models.utils.positional_encoding', 'mmdet.models.utils.res_layer', 'mmdet.models.utils.transformer', 'mmdet.models.utils', 'mmdet.models.backbones.csp_darknet', 'mmdet.models.backbones.darknet', 'mmdet.models.backbones.resnet', 'mmdet.models.backbones.detectors_resnet', 'mmdet.models.backbones.detectors_resnext', 'mmdet.models.backbones.hourglass', 'mmdet.models.backbones.hrnet', 'mmdet.models.backbones.mobilenet_v2', 'mmdet.models.backbones.pvt', 'mmdet.models.backbones.resnext', 'mmdet.models.backbones.regnet', 'mmdet.models.backbones.res2net', 'mmdet.models.backbones.resnest', 'mmdet.models.necks.bfp', 'mmdet.models.necks.channel_mapper', 'mmdet.models.necks.ct_resnet_neck', 'mmdet.models.necks.dilated_encoder', 'mmdet.models.necks.fpg', 'mmdet.models.necks.fpn', 'mmdet.models.necks.fpn_carafe', 'mmdet.models.necks.hrfpn', 'mmcv.ops.merge_cells', 'mmdet.models.necks.nas_fpn', 'mmdet.models.necks.nasfcos_fpn', 'mmdet.models.necks.pafpn', 'mmdet.models.necks.rfp', 'mmdet.models.necks.ssd_neck', 'mmdet.models.necks.yolo_neck', 'mmdet.models.necks.yolox_pafpn', 'mmdet.models.necks', 'mmdet.models.backbones.ssd_vgg', 'mmdet.models.backbones.swin', 'mmdet.models.backbones.trident_resnet', 'mmdet.models.backbones', 'mmdet.models.dense_heads.base_dense_head', 'mmdet.utils.contextmanagers', 'mmdet.models.dense_heads.dense_test_mixins', 'mmdet.models.dense_heads.anchor_free_head', 'mmdet.models.dense_heads.anchor_head', 'mmdet.models.dense_heads.atss_head', 'mmdet.models.dense_heads.fcos_head', 'mmdet.models.dense_heads.paa_head', 'mmdet.models.dense_heads.autoassign_head', 'mmdet.models.dense_heads.rpn_head', 'mmdet.models.dense_heads.cascade_rpn_head', 'mmdet.models.dense_heads.centernet_head', 'mmdet.models.dense_heads.corner_head', 'mmdet.models.dense_heads.centripetal_head', 'mmdet.models.dense_heads.detr_head', 'mmdet.models.dense_heads.deformable_detr_head', 'mmdet.models.dense_heads.embedding_rpn_head', 'mmdet.models.dense_heads.fovea_head', 'mmdet.models.dense_heads.retina_head', 'mmdet.models.dense_heads.free_anchor_retina_head', 'mmdet.models.losses.accuracy', 'mmdet.models.losses.ae_loss', 'mmdet.models.losses.utils', 'mmdet.models.losses.balanced_l1_loss', 'mmdet.models.losses.cross_entropy_loss', 'mmdet.models.losses.dice_loss', 'mmdet.models.losses.focal_loss', 'mmdet.models.losses.gaussian_focal_loss', 'mmdet.models.losses.gfocal_loss', 'mmdet.models.losses.ghm_loss', 'mmdet.models.losses.iou_loss', 'mmdet.models.losses.kd_loss', 'mmdet.models.losses.mse_loss', 'mmdet.models.losses.pisa_loss', 'mmdet.models.losses.seesaw_loss', 'mmdet.models.losses.smooth_l1loss', 'mmdet.models.losses.varifocal [2022-05-31T14:24:23.474628Z] dc5b74e6 || loss', 'mmdet.models.losses', 'mmdet.models.dense_heads.fsaf_head', 'mmdet.models.dense_heads.guided_anchor_head', 'mmdet.models.dense_heads.ga_retina_head', 'mmdet.models.dense_heads.ga_rpn_head', 'mmdet.models.dense_heads.gfl_head', 'mmdet.models.dense_heads.lad_head', 'mmdet.models.dense_heads.ld_head', 'mmdet.models.dense_heads.nasfcos_head', 'mmdet.models.dense_heads.pisa_retinanet_head', 'mmdet.models.dense_heads.ssd_head', 'mmdet.models.dense_heads.pisa_ssd_head', 'mmdet.models.dense_heads.reppoints_head', 'mmdet.models.dense_heads.retina_sepbn_head', 'mmdet.models.dense_heads.sabl_retina_head', 'mmdet.models.dense_heads.base_mask_head', 'mmdet.models.dense_heads.solo_head', 'mmdet.models.dense_heads.tood_head', 'mmdet.models.dense_heads.vfnet_head', 'mmdet.models.dense_heads.yolact_head', 'mmdet.models.dense_heads.yolo_head', 'mmdet.models.dense_heads.yolof_head', 'mmdet.models.dense_heads.yolox_head', 'mmdet.models.dense_heads', 'mmdet.models.detectors', 'mmdet.models.detectors.atss', 'mmdet.models.detectors.single_stage', 'mmdet.models.detectors.base', 'mmdet.core.visualization', 'mmdet.core.visualization.image', 'matplotlib', 'matplotlib._api.deprecation', 'matplotlib._api', 'matplotlib._version', 'matplotlib._c_internal_utils', 'matplotlib.cbook', 'matplotlib.docstring', 'PIL.ImageChops', 'PIL.ImageFile', 'PIL.GimpGradientFile', 'PIL.GimpPaletteFile', 'PIL.PaletteFile', 'PIL.ImagePalette', 'PIL.ImageSequence', 'PIL.PngImagePlugin', 'matplotlib._path', 'matplotlib.bezier', 'matplotlib.path', 'matplotlib.transforms', 'matplotlib.ticker', 'matplotlib.scale', 'matplotlib._color_data', 'matplotlib.colors', 'pyparsing.util', 'pyparsing.unicode', 'pyparsing.exceptions', 'pyparsing.actions', 'pyparsing.results', 'pyparsing.core', 'pyparsing.helpers', 'pyparsing.testing', 'pyparsing.common', 'pyparsing', 'matplotlib.fontconfig_pattern', 'matplotlib._enums', 'cycler', 'matplotlib.rcsetup', 'matplotlib.ft2font', 'kiwisolver._cext', 'kiwisolver'] [2022-05-31T14:24:23.585434Z] dc5b74e6 [rank=1] || DEBUG: [52] matplotlib: CACHEDIR=/root/.cache/matplotlib [2022-05-31T14:24:23.585439Z] dc5b74e6 [rank=1] || DEBUG: [52] matplotlib.font_manager: font search path [PosixPath('/opt/conda/lib/python3.8/site-packages/matplotlib/mpl-data/fonts/ttf'), PosixPath('/opt/conda/lib/python3.8/site-packages/matplotlib/mpl-data/fonts/afm'), PosixPath('/opt/conda/lib/python3.8/site-packages/matplotlib/mpl-data/fonts/pdfcorefonts')] [2022-05-31T14:24:23.586046Z] dc5b74e6 [rank=0] || DEBUG: [49] matplotlib: CACHEDIR=/root/.cache/matplotlib [2022-05-31T14:24:23.586049Z] dc5b74e6 [rank=0] || DEBUG: [49] matplotlib.font_manager: font search path [PosixPath('/opt/conda/lib/python3.8/site-packages/matplotlib/mpl-data/fonts/ttf'), PosixPath('/opt/conda/lib/python3.8/site-packages/matplotlib/mpl-data/fonts/afm'), PosixPath('/opt/conda/lib/python3.8/site-packages/matplotlib/mpl-data/fonts/pdfcorefonts')] [2022-05-31T14:24:23.869577Z] dc5b74e6 [rank=1] || INFO: [52] matplotlib.font_manager: generated new fontManager [2022-05-31T14:24:23.981619Z] dc5b74e6 [rank=0] || INFO: [49] matplotlib.font_manager: generated new fontManager [2022-05-31T14:24:24.310053Z] dc5b74e6 [rank=1] || DEBUG: [52] matplotlib.pyplot: Loaded backend agg version unknown. [2022-05-31T14:24:24.310057Z] dc5b74e6 [rank=1] || DEBUG: [52] matplotlib.pyplot: Loaded backend agg version unknown. [2022-05-31T14:24:24.428557Z] dc5b74e6 [rank=0] || DEBUG: [49] matplotlib.pyplot: Loaded backend agg version unknown. [2022-05-31T14:24:24.428560Z] dc5b74e6 [rank=0] || DEBUG: [49] matplotlib.pyplot: Loaded backend agg version unknown. [2022-05-31T14:24:24.569899Z] dc5b74e6 [rank=1] || INFO: [52] torch.distributed.distributed_c10d: Added key: store_based_barrier_key:1 to store for rank: 1 [2022-05-31T14:24:24.614535Z] dc5b74e6 [rank=0] || INFO: [49] torch.distributed.distributed_c10d: Added key: store_based_barrier_key:1 to store for rank: 0 [2022-05-31T14:24:24.614539Z] dc5b74e6 [rank=0] || INFO: [49] torch.distributed.distributed_c10d: Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. [2022-05-31T14:24:24.615399Z] dc5b74e6 [rank=0] || DEBUG: [49] root: Chief setting up server with ports 12360/12376. [2022-05-31T14:24:24.621524Z] dc5b74e6 [rank=1] || INFO: [52] torch.distributed.distributed_c10d: Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes. [2022-05-31T14:24:24.623811Z] dc5b74e6 [rank=1] || DEBUG: [52] root: Non-Chief 1 setting up comm to 127.0.0.1 w/ ports 12360/12376. [2022-05-31T14:24:24.676484Z] dc5b74e6 [rank=1] || DEBUG: [52] root: Worker 1 beginning zmq allgather. [2022-05-31T14:24:24.677075Z] dc5b74e6 [rank=0] || DEBUG: [49] root: Local Chief setting up server with urls ipc:///tmp/ipcgxzo8mpd/pub.sock/ipc:///tmp/ipcgxzo8mpd/pull.sock. [2022-05-31T14:24:24.677891Z] dc5b74e6 [rank=0] || DEBUG: [49] root: Worker 0 beginning zmq allgather. [2022-05-31T14:24:24.678112Z] dc5b74e6 [rank=0] || DEBUG: [49] root: Worker 0 finished zmq allgather. [2022-05-31T14:24:24.678488Z] dc5b74e6 [rank=1] || DEBUG: [52] root: Worker 1 finished zmq allgather. [2022-05-31T14:24:24.678493Z] dc5b74e6 [rank=1] || DEBUG: [52] root: Local Worker setting up server with urls ipc:///tmp/ipcgxzo8mpd/pub.sock/ipc:///tmp/ipcgxzo8mpd/pull.sock. [2022-05-31T14:24:25.558917Z] dc5b74e6 [rank=0] || DEBUG: [49] tensorflow: Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client. [2022-05-31T14:24:25.558923Z] dc5b74e6 [rank=1] || DEBUG: [52] tensorflow: Falling back to TensorFlow client; we recommended you install the Cloud TPU client directly with pip install cloud-tpu-client. [2022-05-31T14:24:25.779976Z] dc5b74e6 [rank=1] || DEBUG: [52] h5py._conv: Creating converter from 7 to 5 [2022-05-31T14:24:25.779979Z] dc5b74e6 [rank=1] || DEBUG: [52] h5py._conv: Creating converter from 5 to 7 [2022-05-31T14:24:25.779982Z] dc5b74e6 [rank=0] || DEBUG: [49] h5py._conv: Creating converter from 7 to 5 [2022-05-31T14:24:25.779983Z] dc5b74e6 [rank=1] || DEBUG: [52] h5py._conv: Creating converter from 7 to 5 [2022-05-31T14:24:25.779984Z] dc5b74e6 [rank=1] || DEBUG: [52] h5py._conv: Creating converter from 5 to 7 [2022-05-31T14:24:25.779992Z] dc5b74e6 [rank=0] || DEBUG: [49] h5py._conv: Creating converter from 5 to 7 [2022-05-31T14:24:25.779996Z] dc5b74e6 [rank=0] || DEBUG: [49] h5py._conv: Creating converter from 7 to 5 [2022-05-31T14:24:25.780000Z] dc5b74e6 [rank=0] || DEBUG: [49] h5py._conv: Creating converter from 5 to 7 [2022-05-31T14:24:26.335468Z] dc5b74e6 [rank=0] || DEBUG: [49] determined.core: (_PreemptionWatcher thread) _get_preemption(0) [2022-05-31T14:24:26.345959Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: Starting new HTTP connection (1): 10.6.5.8:8080 [2022-05-31T14:24:26.348192Z] dc5b74e6 [rank=1] || /mmdetection/mmdet/datasets/utils.py:65: UserWarning: "ImageToTensor" pipeline is replaced by "DefaultFormatBundle" for batch inference. It is recommended to manually replace it in the test data pipeline in your config file. [2022-05-31T14:24:26.348195Z] dc5b74e6 [rank=1] || warnings.warn( [2022-05-31T14:24:26.348746Z] dc5b74e6 [rank=1] || INFO: [52] root: Config (path: /mmdetection/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py): {'model': {'type': 'FasterRCNN', 'backbone': {'type': 'ResNet', 'depth': 50, 'num_stages': 4, 'out_indices': (0, 1, 2, 3), 'frozen_stages': 1, 'norm_cfg': {'type': 'BN', 'requires_grad': True}, 'norm_eval': True, 'style': 'pytorch', 'init_cfg': {'type': 'Pretrained', 'checkpoint': 'torchvision://resnet50'}}, 'neck': {'type': 'FPN', 'in_channels': [256, 512, 1024, 2048], 'out_channels': 256, 'num_outs': 5}, 'rpn_head': {'type': 'RPNHead', 'in_channels': 256, 'feat_channels': 256, 'anchor_generator': {'type': 'AnchorGenerator', 'scales': [8], 'ratios': [0.5, 1.0, 2.0], 'strides': [4, 8, 16, 32, 64]}, 'bbox_coder': {'type': 'DeltaXYWHBBoxCoder', 'target_means': [0.0, 0.0, 0.0, 0.0], 'target_stds': [1.0, 1.0, 1.0, 1.0]}, 'loss_cls': {'type': 'CrossEntropyLoss', 'use_sigmoid': True, 'loss_weight': 1.0}, 'loss_bbox': {'type': 'L1Loss', 'loss_weight': 1.0}}, 'roi_head': {'type': 'StandardRoIHead', 'bbox_roi_extractor': {'type': 'SingleRoIExtractor', 'roi_layer': {'type': 'RoIAlign', 'output_size': 7, 'sampling_ratio': 0}, 'out_channels': 256, 'featmap_strides': [4, 8, 16, 32]}, 'bbox_head': {'type': 'Shared2FCBBoxHead', 'in_channels': 256, 'fc_out_channels': 1024, 'roi_feat_size': 7, 'num_classes': 80, 'bbox_coder': {'type': 'DeltaXYWHBBoxCoder', 'target_means': [0.0, 0.0, 0.0, 0.0], 'target_stds': [0.1, 0.1, 0.2, 0.2]}, 'reg_class_agnostic': False, 'loss_cls': {'type': 'CrossEntropyLoss', 'use_sigmoid': False, 'loss_weight': 1.0}, 'loss_bbox': {'type': 'L1Loss', 'loss_weight': 1.0}}}, 'train_cfg': {'rpn': {'assigner': {'type': 'MaxIoUAssigner', 'pos_iou_thr': 0.7, 'neg_iou_thr': 0.3, 'min_pos_iou': 0.3, 'match_low_quality': True, 'ignore_iof_thr': -1}, 'sampler': {'type': 'RandomSampler', 'num': 256, 'pos_fraction': 0.5, 'neg_pos_ub': -1, 'add_gt_as_proposals': False}, 'allowed_border': -1, 'pos_weight': -1, 'debug': False}, 'rpn_proposal': {'nms_pre': 2000, 'max_per_img': 1000, 'nms': {'type': 'nms', 'iou_threshold': 0.7}, 'min_bbox_size': 0}, 'rcnn': {'assigner': {'type': 'MaxIoUAssigner', 'pos_iou_thr': 0.5, 'neg_iou_thr': 0.5, 'min_pos_iou': 0.5, 'match_low_quality': False, 'ignore_iof_thr': -1}, 'sampler': {'type': 'RandomSampler', 'num': 512, 'pos_fraction': 0.25, 'neg_pos_ub': -1, 'add_gt_as_proposals': True}, 'pos_weight': -1, 'debug': False}}, 'test_cfg': {'rpn': {'nms_pre': 1000, 'max_per_img': 1000, 'nms': {'type': 'nms', 'iou_threshold': 0.7}, 'min_bbox_size': 0}, 'rcnn': {'score_thr': 0.05, 'nms': {'type': 'nms', 'iou_threshold': 0.5}, 'max_per_img': 100}}}, 'dataset_type': 'CocoDataset', 'data_root': 'data/coco/', 'img_norm_cfg': {'mean': [123.675, 116.28, 103.53], 'std': [58.395, 57.12, 57.375], 'to_rgb': True}, 'train_pipeline': [{'type': 'LoadImageFromFile', 'file_client_args': AttrDict({'backend': 'disk'})}, {'type': 'LoadAnnotations', 'with_bbox': True}, {'type': 'Resize', 'img_scale': (1333, 800), 'keep_ratio': True}, {'type': 'RandomFlip', 'flip_ratio': 0.5}, {'type': 'Normalize', 'mean': [123.675, 116.28, 103.53], 'std': [58.395, 57.12, 57.375], 'to_rgb': True}, {'type': 'Pad', 'size_divisor': 32}, {'type': 'DefaultFormatBundle'}, {'type': 'Collect', 'keys': ['img', 'gt_bboxes', 'gt_labels']}], 'test_pipeline': [{'type': 'LoadImageFromFile', 'file_client_args': AttrDict({'backend': 'disk'})}, {'type': 'MultiScaleFlipAug', 'img_scale': (1333, 800), 'flip': False, 'transforms': [{'type': 'Resize', 'keep_ratio': True}, {'type': 'RandomFlip'}, {'type': 'Normalize', 'mean': [123.675, 116.28, 103.53], 'std': [58.395, 57.12, 57.375], 'to_rgb': True}, {'type': 'Pad', 'size_divisor': 32}, {'type': 'ImageToTensor', 'keys': ['img']}, {'type': 'Collect', 'keys': ['img']}]}], 'data': {'samples_per_gpu': 2, 'workers_per_gpu': 2, 'train': {'type': 'CocoDataset', 'ann_file': '/tmp/coco/annotations/instances_train2017.json', 'img_prefix': 'data/coco/train2017/', 'pipeline': [{'type': 'LoadImageFromFile', 'file_client_args': AttrDict({'backend': 'disk'})}, {'type': 'LoadAnnotations', 'with_bbox': True}, {'type': 'Resize', 'img_scale': (1333, 800), 'keep_ratio': True}, {'type': 'RandomFlip', 'flip_ratio': 0.5}, {'type': 'Normalize', 'mean': [123.675, 116.28, 103.53], 'std': [58.395, 57.12, 57.375], 'to_rgb': True}, {'type': 'Pad', 'size_divisor': 32}, {'type': 'DefaultFormatBundle'}, {'type': 'Collect', 'keys': ['img', 'gt_bboxes', 'gt_labels']}]}, 'val': {'type': 'CocoDataset', 'ann_file': '/tmp/coco/annotations/instances_val2017.json', 'img_prefix': 'data/coco/val2017/', 'pipeline': [{'type': 'LoadImageFromFile', 'file_client_args': AttrDict({'backend': 'disk'})}, {'type': 'MultiScaleFlipAug', 'img_scale': (1333, 800), 'flip': False, 'transforms': [{'type': 'Resize', 'keep_ratio': True}, {'type': 'RandomFlip'}, {'type': 'Normalize', 'mean': [123.675, 116.28, 103.53], 'std': [58.395, 57.12, 57.375], 'to_rgb': True}, {'type': 'Pad', 'size_divisor': 32}, {'type': 'DefaultFormatBundle'}, {'type': 'Collect', 'keys': ['img']}]}], 'test_mode': True}, 'test': {'type': 'CocoDataset', 'ann_file': 'data/coco/annotations/instances_val2017.json', 'img_prefix': 'data/coco/val2017/', 'pipeline': [{'type': 'LoadImageFromFile', 'file_client_args': AttrDict({'backend': 'disk'})}, {'type': 'MultiScaleFlipAug', 'img_scale': (1333, 800), 'flip': False, 'transforms': [{'type': 'Resize', 'keep_ratio': True}, {'type': 'RandomFlip'}, {'type': 'Normalize', 'mean': [123.675, 116.28, 103.53], 'std': [58.395, 57.12, 57.375], 'to_rgb': True}, {'type': 'Pad', 'size_divisor': 32}, {'type': 'DefaultFormatBundle'}, {'type': 'Collect', 'keys': ['img']}]}]}}, 'evaluation': {'interval': 1, 'metric': 'bbox'}, 'optimizer': {'type': 'SGD', 'lr': 0.02, 'momentum': 0.9, 'weight_decay': 0.0001}, 'optimizer_config': {'grad_clip': None}, 'lr_config': {'policy': 'step', 'warmup': 'linear', 'warmup_iters': 500, 'warmup_ratio': 0.001, 'step': [8, 11]}, 'runner': {'type': 'EpochBasedRunner', 'max_epochs': 12}, 'checkpoint_config': {'interval': 1}, 'log_config': {'interval': 50, 'hooks': [{'type': 'TextLoggerHook'}]}, 'custom_hooks': [{'type': 'NumClassCheckHook'}], 'dist_params': {'backend': 'nccl'}, 'log_level': 'INFO', 'load_from': None, 'resume_from': None, 'workflow': [('train', 1)], 'opencv_num_threads': 0, 'mp_start_method': 'fork'} [2022-05-31T14:24:26.348756Z] dc5b74e6 [rank=1] || INFO: [52] root: disabled automatically moving data to device [2022-05-31T14:24:26.350086Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: http://10.6.5.8:8080 "GET /api/v1/allocations/34.29c616a1-c0bf-418c-a8bd-b855ef8ac23d.1/signals/preemption?timeout_seconds=0 HTTP/1.1" 200 17 [2022-05-31T14:24:26.352527Z] dc5b74e6 [rank=0] || DEBUG: [49] determined.core: (_PreemptionWatcher thread) _get_preemption(60) [2022-05-31T14:24:26.357083Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: Starting new HTTP connection (1): 10.6.5.8:8080 [2022-05-31T14:24:26.367921Z] dc5b74e6 [rank=0] || /mmdetection/mmdet/datasets/utils.py:65: UserWarning: "ImageToTensor" pipeline is replaced by "DefaultFormatBundle" for batch inference. It is recommended to manually replace it in the test data pipeline in your config file. [2022-05-31T14:24:26.367924Z] dc5b74e6 [rank=0] || warnings.warn( [2022-05-31T14:24:26.370021Z] dc5b74e6 [rank=0] || INFO: [49] root: Config (path: /mmdetection/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py): {'model': {'type': 'FasterRCNN', 'backbone': {'type': 'ResNet', 'depth': 50, 'num_stages': 4, 'out_indices': (0, 1, 2, 3), 'frozen_stages': 1, 'norm_cfg': {'type': 'BN', 'requires_grad': True}, 'norm_eval': True, 'style': 'pytorch', 'init_cfg': {'type': 'Pretrained', 'checkpoint': 'torchvision://resnet50'}}, 'neck': {'type': 'FPN', 'in_channels': [256, 512, 1024, 2048], 'out_channels': 256, 'num_outs': 5}, 'rpn_head': {'type': 'RPNHead', 'in_channels': 256, 'feat_channels': 256, 'anchor_generator': {'type': 'AnchorGenerator', 'scales': [8], 'ratios': [0.5, 1.0, 2.0], 'strides': [4, 8, 16, 32, 64]}, 'bbox_coder': {'type': 'DeltaXYWHBBoxCoder', 'target_means': [0.0, 0.0, 0.0, 0.0], 'target_stds': [1.0, 1.0, 1.0, 1.0]}, 'loss_cls': {'type': 'CrossEntropyLoss', 'use_sigmoid': True, 'loss_weight': 1.0}, 'loss_bbox': {'type': 'L1Loss', 'loss_weight': 1.0}}, 'roi_head': {'type': 'StandardRoIHead', 'bbox_roi_extractor': {'type': 'SingleRoIExtractor', 'roi_layer': {'type': 'RoIAlign', 'output_size': 7, 'sampling_ratio': 0}, 'out_channels': 256, 'featmap_strides': [4, 8, 16, 32]}, 'bbox_head': {'type': 'Shared2FCBBoxHead', 'in_channels': 256, 'fc_out_channels': 1024, 'roi_feat_size': 7, 'num_classes': 80, 'bbox_coder': {'type': 'DeltaXYWHBBoxCoder', 'target_means': [0.0, 0.0, 0.0, 0.0], 'target_stds': [0.1, 0.1, 0.2, 0.2]}, 'reg_class_agnostic': False, 'loss_cls': {'type': 'CrossEntropyLoss', 'use_sigmoid': False, 'loss_weight': 1.0}, 'loss_bbox': {'type': 'L1Loss', 'loss_weight': 1.0}}}, 'train_cfg': {'rpn': {'assigner': {'type': 'MaxIoUAssigner', 'pos_iou_thr': 0.7, 'neg_iou_thr': 0.3, 'min_pos_iou': 0.3, 'match_low_quality': True, 'ignore_iof_thr': -1}, 'sampler': {'type': 'RandomSampler', 'num': 256, 'pos_fraction': 0.5, 'neg_pos_ub': -1, 'add_gt_as_proposals': False}, 'allowed_border': -1, 'pos_weight': -1, 'debug': False}, 'rpn_proposal': {'nms_pre': 2000, 'max_per_img': 1000, 'nms': {'type': 'nms', 'iou_threshold': 0.7}, 'min_bbox_size': 0}, 'rcnn': {'assigner': {'type': 'MaxIoUAssigner', 'pos_iou_thr': 0.5, 'neg_iou_thr': 0.5, 'min_pos_iou': 0.5, 'match_low_quality': False, 'ignore_iof_thr': -1}, 'sampler': {'type': 'RandomSampler', 'num': 512, 'pos_fraction': 0.25, 'neg_pos_ub': -1, 'add_gt_as_proposals': True}, 'pos_weight': -1, 'debug': False}}, 'test_cfg': {'rpn': {'nms_pre': 1000, 'max_per_img': 1000, 'nms': {'type': 'nms', 'iou_threshold': 0.7}, 'min_bbox_size': 0}, 'rcnn': {'score_thr': 0.05, 'nms': {'type': 'nms', 'iou_threshold': 0.5}, 'max_per_img': 100}}}, 'dataset_type': 'CocoDataset', 'data_root': 'data/coco/', 'img_norm_cfg': {'mean': [123.675, 116.28, 103.53], 'std': [58.395, 57.12, 57.375], 'to_rgb': True}, 'train_pipeline': [{'type': 'LoadImageFromFile', 'file_client_args': AttrDict({'backend': 'disk'})}, {'type': 'LoadAnnotations', 'with_bbox': True}, {'type': 'Resize', 'img_scale': (1333, 800), 'keep_ratio': True}, {'type': 'RandomFlip', 'flip_ratio': 0.5}, {'type': 'Normalize', 'mean': [123.675, 116.28, 103.53], 'std': [58.395, 57.12, 57.375], 'to_rgb': True}, {'type': 'Pad', 'size_divisor': 32}, {'type': 'DefaultFormatBundle'}, {'type': 'Collect', 'keys': ['img', 'gt_bboxes', 'gt_labels']}], 'test_pipeline': [{'type': 'LoadImageFromFile', 'file_client_args': AttrDict({'backend': 'disk'})}, {'type': 'MultiScaleFlipAug', 'img_scale': (1333, 800), 'flip': False, 'transforms': [{'type': 'Resize', 'keep_ratio': True}, {'type': 'RandomFlip'}, {'type': 'Normalize', 'mean': [123.675, 116.28, 103.53], 'std': [58.395, 57.12, 57.375], 'to_rgb': True}, {'type': 'Pad', 'size_divisor': 32}, {'type': 'ImageToTensor', 'keys': ['img']}, {'type': 'Collect', 'keys': ['img']}]}], 'data': {'samples_per_gpu': 2, 'workers_per_gpu': 2, 'train': {'type': 'CocoDataset', 'ann_file': '/tmp/coco/annotations/instances_train2017.json', 'img_prefix': 'data/coco/train2017/', 'pipeline': [{'type': 'LoadImageFromFile', 'file_client_args': AttrDict({'backend': 'disk'})}, {'type': 'LoadAnnotations', 'with_bbox': True}, {'type': 'Resize', 'img_scale': (1333, 800), 'keep_ratio': True}, {'type': 'RandomFlip', 'flip_ratio': 0.5}, {'type': 'Normalize', 'mean': [123.675, 116.28, 103.53], 'std': [58.395, 57.12, 57.375], 'to_rgb': True}, {'type': 'Pad', 'size_divisor': 32}, {'type': 'DefaultFormatBundle'}, {'type': 'Collect', 'keys': ['img', 'gt_bboxes', 'gt_labels']}]}, 'val': {'type': 'CocoDataset', 'ann_file': '/tmp/coco/annotations/instances_val2017.json', 'img_prefix': 'data/coco/val2017/', 'pipeline': [{'type': 'LoadImageFromFile', 'file_client_args': AttrDict({'backend': 'disk'})}, {'type': 'MultiScaleFlipAug', 'img_scale': (1333, 800), 'flip': False, 'transforms': [{'type': 'Resize', 'keep_ratio': True}, {'type': 'RandomFlip'}, {'type': 'Normalize', 'mean': [123.675, 116.28, 103.53], 'std': [58.395, 57.12, 57.375], 'to_rgb': True}, {'type': 'Pad', 'size_divisor': 32}, {'type': 'DefaultFormatBundle'}, {'type': 'Collect', 'keys': ['img']}]}], 'test_mode': True}, 'test': {'type': 'CocoDataset', 'ann_file': 'data/coco/annotations/instances_val2017.json', 'img_prefix': 'data/coco/val2017/', 'pipeline': [{'type': 'LoadImageFromFile', 'file_client_args': AttrDict({'backend': 'disk'})}, {'type': 'MultiScaleFlipAug', 'img_scale': (1333, 800), 'flip': False, 'transforms': [{'type': 'Resize', 'keep_ratio': True}, {'type': 'RandomFlip'}, {'type': 'Normalize', 'mean': [123.675, 116.28, 103.53], 'std': [58.395, 57.12, 57.375], 'to_rgb': True}, {'type': 'Pad', 'size_divisor': 32}, {'type': 'DefaultFormatBundle'}, {'type': 'Collect', 'keys': ['img']}]}]}}, 'evaluation': {'interval': 1, 'metric': 'bbox'}, 'optimizer': {'type': 'SGD', 'lr': 0.02, 'momentum': 0.9, 'weight_decay': 0.0001}, 'optimizer_config': {'grad_clip': None}, 'lr_config': {'policy': 'step', 'warmup': 'linear', 'warmup_iters': 500, 'warmup_ratio': 0.001, 'step': [8, 11]}, 'runner': {'type': 'EpochBasedRunner', 'max_epochs': 12}, 'checkpoint_config': {'interval': 1}, 'log_config': {'interval': 50, 'hooks': [{'type': 'TextLoggerHook'}]}, 'custom_hooks': [{'type': 'NumClassCheckHook'}], 'dist_params': {'backend': 'nccl'}, 'log_level': 'INFO', 'load_from': None, 'resume_from': None, 'workflow': [('train', 1)], 'opencv_num_threads': 0, 'mp_start_method': 'fork'} [2022-05-31T14:24:26.370025Z] dc5b74e6 [rank=0] || INFO: [49] root: disabled automatically moving data to device [2022-05-31T14:24:26.785393Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:26,784 - mmcv - INFO - initialize ResNet with init_cfg {'type': 'Pretrained', 'checkpoint': 'torchvision://resnet50'} [2022-05-31T14:24:26.785397Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:26,785 - mmcv - INFO - load model from: torchvision://resnet50 [2022-05-31T14:24:26.785492Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:26,785 - mmcv - INFO - load checkpoint from torchvision path: torchvision://resnet50 [2022-05-31T14:24:26.787335Z] dc5b74e6 [rank=0] || Downloading: "https://download.pytorch.org/models/resnet50-0676ba61.pth" to /root/.cache/torch/hub/checkpoints/resnet50-0676ba61.pth [2022-05-31T14:24:38.141461Z] dc5b74e6 [rank=0] || [2022-05-31T14:24:38.141470Z] dc5b74e6 || 0%| | 0.00/97.8M [00:00<?, ?B/s] [2022-05-31T14:24:38.141472Z] dc5b74e6 || 0%| | 152k/97.8M [00:00<01:07, 1.52MB/s] [2022-05-31T14:24:38.141474Z] dc5b74e6 || 1%| | 784k/97.8M [00:00<00:23, 4.40MB/s] [2022-05-31T14:24:38.141478Z] dc5b74e6 || 1%|鈻? | 1.39M/97.8M [00:00<00:18, 5.36MB/s] [2022-05-31T14:24:38.141483Z] dc5b74e6 || 3%|鈻? | 2.49M/97.8M [00:00<00:15, 6.58MB/s] [2022-05-31T14:24:38.141488Z] dc5b74e6 || 4%|鈻? | 3.62M/97.8M [00:00<00:11, 8.26MB/s] [2022-05-31T14:24:38.141492Z] dc5b74e6 || 5%|鈻? | 4.42M/97.8M [00:00<00:12, 7.88MB/s] [2022-05-31T14:24:38.141499Z] dc5b74e6 || 5%|鈻? | 5.27M/97.8M [00:00<00:12, 7.68MB/s] [2022-05-31T14:24:38.141507Z] dc5b74e6 || 7%|鈻? | 6.39M/97.8M [00:00<00:10, 8.83MB/s] [2022-05-31T14:24:38.141511Z] dc5b74e6 || 7%|鈻? | 7.25M/97.8M [00:01<00:11, 8.19MB/s] [2022-05-31T14:24:38.141516Z] dc5b74e6 || 8%|鈻? | 8.22M/97.8M [00:01<00:10, 8.73MB/s] [2022-05-31T14:24:38.141522Z] dc5b74e6 || 9%|鈻? | 9.29M/97.8M [00:01<00:09, 9.43MB/s] [2022-05-31T14:24:38.141528Z] dc5b74e6 || 11%|鈻? | 10.4M/97.8M [00:01<00:09, 10.1MB/s] [2022-05-31T14:24:38.141535Z] dc5b74e6 || 12%|鈻堚枏 | 11.4M/97.8M [00:01<00:14, 6.40MB/s] [2022-05-31T14:24:38.141540Z] dc5b74e6 || 14%|鈻堚枍 | 13.5M/97.8M [00:01<00:09, 9.37MB/s] [2022-05-31T14:24:38.141547Z] dc5b74e6 || 15%|鈻堚枍 | 14.6M/97.8M [00:01<00:08, 9.90MB/s] [2022-05-31T14:24:38.141554Z] dc5b74e6 || 16%|鈻堚枌 | 15.8M/97.8M [00:01<00:09, 9.35MB/s] [2022-05-31T14:24:38.141561Z] dc5b74e6 || 17%|鈻堚枊 | 16.9M/97.8M [00:02<00:08, 9.89MB/s] [2022-05-31T14:24:38.141568Z] dc5b74e6 || 18%|鈻堚枈 | 17.9M/97.8M [00:02<00:09, 9.23MB/s] [2022-05-31T14:24:38.141573Z] dc5b74e6 || 19%|鈻堚枆 | 19.0M/97.8M [00:02<00:08, 9.85MB/s] [2022-05-31T14:24:38.141578Z] dc5b74e6 || 20%|鈻堚枅 | 20.0M/97.8M [00:02<00:08, 9.20MB/s] [2022-05-31T14:24:38.141582Z] dc5b74e6 || 22%|鈻堚枅鈻? | 21.1M/97.8M [00:02<00:08, 9.79MB/s] [2022-05-31T14:24:38.141588Z] dc5b74e6 || 23%|鈻堚枅鈻? | 22.3M/97.8M [00:02<00:07, 10.3MB/s] [2022-05-31T14:24:38.141592Z] dc5b74e6 || 24%|鈻堚枅鈻? | 23.3M/97.8M [00:02<00:08, 9.50MB/s] [2022-05-31T14:24:38.141596Z] dc5b74e6 || 25%|鈻堚枅鈻? | 24.4M/97.8M [00:02<00:07, 10.1MB/s] [2022-05-31T14:24:38.141601Z] dc5b74e6 || 26%|鈻堚枅鈻? | 25.4M/97.8M [00:03<00:08, 9.34MB/s] [2022-05-31T14:24:38.141605Z] dc5b74e6 || 27%|鈻堚枅鈻? | 26.5M/97.8M [00:03<00:07, 9.99MB/s] [2022-05-31T14:24:38.141612Z] dc5b74e6 || 28%|鈻堚枅鈻? | 27.5M/97.8M [00:03<00:07, 9.25MB/s] [2022-05-31T14:24:38.141622Z] dc5b74e6 || 29%|鈻堚枅鈻? | 28.6M/97.8M [00:03<00:07, 9.90MB/s] [2022-05-31T14:24:38.141629Z] dc5b74e6 || 30%|鈻堚枅鈻? | 29.6M/97.8M [00:03<00:07, 9.15MB/s] [2022-05-31T14:24:38.141635Z] dc5b74e6 || 31%|鈻堚枅鈻堚枏 | 30.8M/97.8M [00:03<00:07, 9.84MB/s] [2022-05-31T14:24:38.141640Z] dc5b74e6 || 32%|鈻堚枅鈻堚枏 | 31.7M/97.8M [00:03<00:07, 9.08MB/s] [2022-05-31T14:24:38.141647Z] dc5b74e6 || 34%|鈻堚枅鈻堚枎 | 32.8M/97.8M [00:03<00:06, 9.75MB/s] [2022-05-31T14:24:38.141652Z] dc5b74e6 || 35%|鈻堚枅鈻堚枍 | 33.8M/97.8M [00:03<00:07, 9.00MB/s] [2022-05-31T14:24:38.141659Z] dc5b74e6 || 36%|鈻堚枅鈻堚枌 | 34.9M/97.8M [00:04<00:06, 9.71MB/s] [2022-05-31T14:24:38.141668Z] dc5b74e6 || 37%|鈻堚枅鈻堚枊 | 35.9M/97.8M [00:04<00:07, 9.04MB/s] [2022-05-31T14:24:38.141675Z] dc5b74e6 || 38%|鈻堚枅鈻堚枈 | 37.0M/97.8M [00:04<00:06, 9.74MB/s] [2022-05-31T14:24:38.141681Z] dc5b74e6 || 39%|鈻堚枅鈻堚枆 | 38.1M/97.8M [00:04<00:06, 10.3MB/s] [2022-05-31T14:24:38.141686Z] dc5b74e6 || 40%|鈻堚枅鈻堚枅 | 39.1M/97.8M [00:04<00:06, 9.45MB/s] [2022-05-31T14:24:38.141694Z] dc5b74e6 || 41%|鈻堚枅鈻堚枅 | 40.3M/97.8M [00:04<00:06, 10.0MB/s] [2022-05-31T14:24:38.141701Z] dc5b74e6 || 42%|鈻堚枅鈻堚枅鈻? | 41.2M/97.8M [00:04<00:06, 9.27MB/s] [2022-05-31T14:24:38.141708Z] dc5b74e6 || 43%|鈻堚枅鈻堚枅鈻? | 42.4M/97.8M [00:04<00:05, 9.92MB/s] [2022-05-31T14:24:38.141713Z] dc5b74e6 || 44%|鈻堚枅鈻堚枅鈻? | 43.4M/97.8M [00:05<00:06, 9.15MB/s] [2022-05-31T14:24:38.141720Z] dc5b74e6 || 45%|鈻堚枅鈻堚枅鈻? | 44.5M/97.8M [00:05<00:05, 9.82MB/s] [2022-05-31T14:24:38.141726Z] dc5b74e6 || 47%|鈻堚枅鈻堚枅鈻? | 45.5M/97.8M [00:05<00:06, 9.10MB/s] [2022-05-31T14:24:38.141732Z] dc5b74e6 || 48%|鈻堚枅鈻堚枅鈻? | 46.6M/97.8M [00:05<00:05, 9.77MB/s] [2022-05-31T14:24:38.141738Z] dc5b74e6 || 49%|鈻堚枅鈻堚枅鈻? | 47.7M/97.8M [00:05<00:05, 10.3MB/s] [2022-05-31T14:24:38.141744Z] dc5b74e6 || 50%|鈻堚枅鈻堚枅鈻? | 48.8M/97.8M [00:05<00:05, 9.43MB/s] [2022-05-31T14:24:38.141749Z] dc5b74e6 || 51%|鈻堚枅鈻堚枅鈻? | 49.9M/97.8M [00:05<00:04, 10.1MB/s] [2022-05-31T14:24:38.141755Z] dc5b74e6 || 52%|鈻堚枅鈻堚枅鈻堚枏 | 50.9M/97.8M [00:05<00:05, 9.23MB/s] [2022-05-31T14:24:38.141761Z] dc5b74e6 || 53%|鈻堚枅鈻堚枅鈻堚枎 | 52.0M/97.8M [00:05<00:04, 9.88MB/s] [2022-05-31T14:24:38.141767Z] dc5b74e6 || 54%|鈻堚枅鈻堚枅鈻堚枍 | 53.0M/97.8M [00:06<00:05, 9.18MB/s] [2022-05-31T14:24:38.141772Z] dc5b74e6 || 55%|鈻堚枅鈻堚枅鈻堚枌 | 54.0M/97.8M [00:06<00:04, 9.69MB/s] [2022-05-31T14:24:38.141779Z] dc5b74e6 || 56%|鈻堚枅鈻堚枅鈻堚枌 | 55.0M/97.8M [00:06<00:05, 8.91MB/s] [2022-05-31T14:24:38.141785Z] dc5b74e6 || 57%|鈻堚枅鈻堚枅鈻堚枊 | 56.1M/97.8M [00:06<00:04, 9.66MB/s] [2022-05-31T14:24:38.141791Z] dc5b74e6 || 58%|鈻堚枅鈻堚枅鈻堚枈 | 57.2M/97.8M [00:06<00:04, 9.09MB/s] [2022-05-31T14:24:38.141797Z] dc5b74e6 || 60%|鈻堚枅鈻堚枅鈻堚枆 | 58.3M/97.8M [00:06<00:04, 9.77MB/s] [2022-05-31T14:24:38.141802Z] dc5b74e6 || 61%|鈻堚枅鈻堚枅鈻堚枅 | 59.4M/97.8M [00:06<00:04, 9.19MB/s] [2022-05-31T14:24:38.141808Z] dc5b74e6 || 62%|鈻堚枅鈻堚枅鈻堚枅鈻? | 60.5M/97.8M [00:06<00:03, 9.87MB/s] [2022-05-31T14:24:38.141814Z] dc5b74e6 || 63%|鈻堚枅鈻堚枅鈻堚枅鈻? | 61.5M/97.8M [00:06<00:03, 10.0MB/s] [2022-05-31T14:24:38.141821Z] dc5b74e6 || 64%|鈻堚枅鈻堚枅鈻堚枅鈻? | 62.6M/97.8M [00:07<00:03, 10.5MB/s] [2022-05-31T14:24:38.141826Z] dc5b74e6 || 65%|鈻堚枅鈻堚枅鈻堚枅鈻? | 63.7M/97.8M [00:07<00:03, 9.59MB/s] [2022-05-31T14:24:38.141832Z] dc5b74e6 || 66%|鈻堚枅鈻堚枅鈻堚枅鈻? | 64.8M/97.8M [00:07<00:03, 10.2MB/s] [2022-05-31T14:24:38.141838Z] dc5b74e6 || 67%|鈻堚枅鈻堚枅鈻堚枅鈻? | 65.8M/97.8M [00:07<00:03, 9.32MB/s] [2022-05-31T14:24:38.141844Z] dc5b74e6 || 68%|鈻堚枅鈻堚枅鈻堚枅鈻? | 66.9M/97.8M [00:07<00:03, 9.96MB/s] [2022-05-31T14:24:38.141850Z] dc5b74e6 || 69%|鈻堚枅鈻堚枅鈻堚枅鈻? | 67.9M/97.8M [00:07<00:03, 9.22MB/s] [2022-05-31T14:24:38.141856Z] dc5b74e6 || 71%|鈻堚枅鈻堚枅鈻堚枅鈻? | 69.0M/97.8M [00:07<00:03, 9.89MB/s] [2022-05-31T14:24:38.141862Z] dc5b74e6 || 72%|鈻堚枅鈻堚枅鈻堚枅鈻堚枏 | 70.0M/97.8M [00:07<00:03, 9.18MB/s] [2022-05-31T14:24:38.141868Z] dc5b74e6 || 73%|鈻堚枅鈻堚枅鈻堚枅鈻堚枎 | 71.1M/97.8M [00:08<00:02, 9.85MB/s] [2022-05-31T14:24:38.141876Z] dc5b74e6 || 74%|鈻堚枅鈻堚枅鈻堚枅鈻堚枎 | 72.1M/97.8M [00:08<00:02, 9.12MB/s] [2022-05-31T14:24:38.141881Z] dc5b74e6 || 75%|鈻堚枅鈻堚枅鈻堚枅鈻堚枍 | 73.2M/97.8M [00:08<00:02, 9.80MB/s] [2022-05-31T14:24:38.141887Z] dc5b74e6 || 76%|鈻堚枅鈻堚枅鈻堚枅鈻堚枌 | 74.1M/97.8M [00:08<00:02, 9.06MB/s] [2022-05-31T14:24:38.141893Z] dc5b74e6 || 77%|鈻堚枅鈻堚枅鈻堚枅鈻堚枊 | 75.2M/97.8M [00:08<00:02, 9.63MB/s] [2022-05-31T14:24:38.141901Z] dc5b74e6 || 78%|鈻堚枅鈻堚枅鈻堚枅鈻堚枈 | 76.3M/97.8M [00:08<00:02, 10.2MB/s] [2022-05-31T14:24:38.141907Z] dc5b74e6 || 79%|鈻堚枅鈻堚枅鈻堚枅鈻堚枆 | 77.3M/97.8M [00:08<00:02, 9.33MB/s] [2022-05-31T14:24:38.141912Z] dc5b74e6 || 80%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅 | 78.5M/97.8M [00:08<00:02, 9.96MB/s] [2022-05-31T14:24:38.141918Z] dc5b74e6 || 81%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅 | 79.4M/97.8M [00:08<00:02, 9.22MB/s] [2022-05-31T14:24:38.141924Z] dc5b74e6 || 82%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻?| 80.6M/97.8M [00:09<00:01, 9.86MB/s] [2022-05-31T14:24:38.141930Z] dc5b74e6 || 83%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻?| 81.5M/97.8M [00:09<00:01, 9.20MB/s] [2022-05-31T14:24:38.141938Z] dc5b74e6 || 85%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻?| 82.6M/97.8M [00:09<00:01, 9.84MB/s] [2022-05-31T14:24:38.141943Z] dc5b74e6 || 85%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻?| 83.6M/97.8M [00:09<00:01, 9.17MB/s] [2022-05-31T14:24:38.141949Z] dc5b74e6 || 87%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻?| 84.7M/97.8M [00:09<00:01, 9.85MB/s] [2022-05-31T14:24:38.141956Z] dc5b74e6 || 88%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻?| 85.7M/97.8M [00:09<00:01, 9.14MB/s] [2022-05-31T14:24:38.141962Z] dc5b74e6 || 89%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻?| 86.8M/97.8M [00:09<00:01, 9.82MB/s] [2022-05-31T14:24:38.141968Z] dc5b74e6 || 90%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻?| 87.8M/97.8M [00:09<00:01, 9.16MB/s] [2022-05-31T14:24:38.141975Z] dc5b74e6 || 91%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻?| 89.0M/97.8M [00:09<00:00, 9.85MB/s] [2022-05-31T14:24:38.141981Z] dc5b74e6 || 92%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枏| 90.1M/97.8M [00:10<00:00, 10.4MB/s] [2022-05-31T14:24:38.141987Z] dc5b74e6 || 93%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枎| 91.1M/97.8M [00:10<00:00, 9.46MB/s] [2022-05-31T14:24:38.141992Z] dc5b74e6 || 94%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枍| 92.2M/97.8M [00:10<00:00, 10.1MB/s] [2022-05-31T14:24:38.141998Z] dc5b74e6 || 95%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枌| 93.2M/97.8M [00:10<00:00, 5.80MB/s] [2022-05-31T14:24:38.142005Z] dc5b74e6 || 96%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枊| 94.4M/97.8M [00:10<00:00, 6.88MB/s] [2022-05-31T14:24:38.142012Z] dc5b74e6 || 97%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枊| 95.2M/97.8M [00:10<00:00, 7.00MB/s] [2022-05-31T14:24:38.142017Z] dc5b74e6 || 99%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枈| 96.4M/97.8M [00:11<00:00, 8.04MB/s] [2022-05-31T14:24:38.142023Z] dc5b74e6 || 99%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枆| 97.3M/97.8M [00:11<00:00, 7.84MB/s] [2022-05-31T14:24:38.142029Z] dc5b74e6 || 100%|鈻堚枅鈻堚枅鈻堚枅鈻堚枅鈻堚枅| 97.8M/97.8M [00:11<00:00, 9.16MB/s] [2022-05-31T14:24:44.115565Z] dc5b74e6 [rank=0] || Timeout (0:00:30)! [2022-05-31T14:24:44.115579Z] dc5b74e6 [rank=0] || Thread 0x00007f32e68e1700 (most recent call first): [2022-05-31T14:24:44.115584Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31T14:24:44.115589Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31T14:24:44.115593Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31T14:24:44.115598Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:24:44.115601Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:24:44.115606Z] dc5b74e6 [rank=0] || [2022-05-31T14:24:44.115610Z] dc5b74e6 [rank=0] || Thread 0x00007f32eedf4700 (most recent call first): [2022-05-31T14:24:44.115710Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31T14:24:44.115717Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31T14:24:44.115810Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31T14:24:44.115869Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31T14:24:44.115976Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31T14:24:44.116089Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31T14:24:44.116187Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31T14:24:44.116274Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31T14:24:44.116378Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request [2022-05-31T14:24:44.116527Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/requests.py", line 41 in request [2022-05-31T14:24:44.116648Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/api/request.py", line 104 in do_request [2022-05-31T14:24:44.116816Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 65 in _do_request [2022-05-31T14:24:44.116955Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 86 in get [2022-05-31T14:24:44.117111Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 57 in _get_preemption [2022-05-31T14:24:44.117235Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 89 in run [2022-05-31T14:24:44.117305Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:24:44.117400Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:24:44.117404Z] dc5b74e6 [rank=0] || [2022-05-31T14:24:44.117406Z] dc5b74e6 [rank=0] || Thread 0x00007f32ef5f5700 (most recent call first): [2022-05-31T14:24:44.117460Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:24:44.117605Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:24:44.117733Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:24:44.117792Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:24:44.117871Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:24:44.117875Z] dc5b74e6 [rank=0] || [2022-05-31T14:24:44.117878Z] dc5b74e6 [rank=0] || Thread 0x00007f34546e3240 (most recent call first): [2022-05-31T14:24:44.118015Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 2709 in barrier [2022-05-31T14:24:44.118144Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/checkpoint.py", line 292 in load_from_http [2022-05-31T14:24:44.118263Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/checkpoint.py", line 399 in load_from_torchvision [2022-05-31T14:24:44.118395Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/checkpoint.py", line 251 in load_checkpoint [2022-05-31T14:24:44.118504Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/checkpoint.py", line 481 in _load_checkpoint [2022-05-31T14:24:44.118615Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/checkpoint.py", line 542 in load_checkpoint [2022-05-31T14:24:44.118732Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/mmcv/cnn/utils/weight_init.py", line 489 in call [2022-05-31T14:24:44.118842Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/mmcv/cnn/utils/weight_init.py", line 517 in _initialize [2022-05-31T14:24:44.118955Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/mmcv/cnn/utils/weight_init.py", line 613 in initialize [2022-05-31T14:24:44.119069Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/base_module.py", line 105 in init_weights [2022-05-31T14:24:44.119202Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/base_module.py", line 116 in init_weights [2022-05-31T14:24:44.119330Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_trial.py", line 68 in init [2022-05-31T14:24:44.119460Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 111 in main [2022-05-31T14:24:44.119586Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:24:44.119656Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:24:44.119737Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:24:44.122147Z] dc5b74e6 [rank=1] || Timeout (0:00:30)! [2022-05-31T14:24:44.122153Z] dc5b74e6 [rank=1] || Thread 0x00007f20ab5f5700 (most recent call first): [2022-05-31T14:24:44.122156Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:24:44.122188Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:24:44.122308Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:24:44.122394Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:24:44.122475Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:24:44.122478Z] dc5b74e6 [rank=1] || [2022-05-31T14:24:44.122480Z] dc5b74e6 [rank=1] || Thread 0x00007f2210724240 (most recent call first): [2022-05-31T14:24:44.122611Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 2709 in barrier [2022-05-31T14:24:44.122711Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/checkpoint.py", line 292 in load_from_http [2022-05-31T14:24:44.122851Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/checkpoint.py", line 399 in load_from_torchvision [2022-05-31T14:24:44.122966Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/checkpoint.py", line 251 in load_checkpoint [2022-05-31T14:24:44.123071Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/checkpoint.py", line 481 in _load_checkpoint [2022-05-31T14:24:44.123185Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/checkpoint.py", line 542 in load_checkpoint [2022-05-31T14:24:44.123299Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/mmcv/cnn/utils/weight_init.py", line 489 in call [2022-05-31T14:24:44.123422Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/mmcv/cnn/utils/weight_init.py", line 517 in _initialize [2022-05-31T14:24:44.123526Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/mmcv/cnn/utils/weight_init.py", line 613 in initialize [2022-05-31T14:24:44.123635Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/base_module.py", line 105 in init_weights [2022-05-31T14:24:44.123744Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/mmcv/runner/base_module.py", line 116 in init_weights [2022-05-31T14:24:44.123859Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_trial.py", line 68 in init [2022-05-31T14:24:44.123990Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 111 in main [2022-05-31T14:24:44.124116Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:24:44.124184Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:24:44.124279Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:24:44.926154Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:44,925 - mmcv - WARNING - The model and loaded state dict do not match exactly [2022-05-31T14:24:44.926164Z] dc5b74e6 [rank=0] || [2022-05-31T14:24:44.926165Z] dc5b74e6 [rank=0] || unexpected key in source state_dict: fc.weight, fc.bias [2022-05-31T14:24:44.926169Z] dc5b74e6 [rank=0] || [2022-05-31T14:24:44.950564Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:44,950 - mmcv - INFO - initialize FPN with init_cfg {'type': 'Xavier', 'layer': 'Conv2d', 'distribution': 'uniform'} [2022-05-31T14:24:44.976988Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:44,976 - mmcv - INFO - initialize RPNHead with init_cfg {'type': 'Normal', 'layer': 'Conv2d', 'std': 0.01} [2022-05-31T14:24:44.982467Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:44,982 - mmcv - INFO - initialize Shared2FCBBoxHead with init_cfg [{'type': 'Normal', 'std': 0.01, 'override': {'name': 'fc_cls'}}, {'type': 'Normal', 'std': 0.001, 'override': {'name': 'fc_reg'}}, {'type': 'Xavier', 'distribution': 'uniform', 'override': [{'name': 'shared_fcs'}, {'name': 'cls_fcs'}, {'name': 'reg_fcs'}]}] [2022-05-31T14:24:45.106728Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,106 - mmcv - INFO -

[kelvinwang139]

more logs because comments size limitations： [2022-05-31T14:24:45.106732Z] dc5b74e6 [rank=0] || backbone.conv1.weight - torch.Size([64, 3, 7, 7]): [2022-05-31T14:24:45.106733Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.106735Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.106739Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,106 - mmcv - INFO - [2022-05-31T14:24:45.106742Z] dc5b74e6 [rank=0] || backbone.bn1.weight - torch.Size([64]): [2022-05-31T14:24:45.106745Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.106748Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.106755Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,106 - mmcv - INFO - [2022-05-31T14:24:45.106759Z] dc5b74e6 [rank=0] || backbone.bn1.bias - torch.Size([64]): [2022-05-31T14:24:45.106764Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.106768Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.106774Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,106 - mmcv - INFO - [2022-05-31T14:24:45.106778Z] dc5b74e6 [rank=0] || backbone.layer1.0.conv1.weight - torch.Size([64, 64, 1, 1]): [2022-05-31T14:24:45.106783Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.106788Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.106793Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,106 - mmcv - INFO - [2022-05-31T14:24:45.106798Z] dc5b74e6 [rank=0] || backbone.layer1.0.bn1.weight - torch.Size([64]): [2022-05-31T14:24:45.106802Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.106807Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.106812Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,106 - mmcv - INFO - [2022-05-31T14:24:45.106817Z] dc5b74e6 [rank=0] || backbone.layer1.0.bn1.bias - torch.Size([64]): [2022-05-31T14:24:45.106822Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.106827Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.106831Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,106 - mmcv - INFO - [2022-05-31T14:24:45.106836Z] dc5b74e6 [rank=0] || backbone.layer1.0.conv2.weight - torch.Size([64, 64, 3, 3]): [2022-05-31T14:24:45.106841Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.106845Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.106856Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,106 - mmcv - INFO - [2022-05-31T14:24:45.106861Z] dc5b74e6 [rank=0] || backbone.layer1.0.bn2.weight - torch.Size([64]): [2022-05-31T14:24:45.106865Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.106870Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.106875Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,106 - mmcv - INFO - [2022-05-31T14:24:45.106879Z] dc5b74e6 [rank=0] || backbone.layer1.0.bn2.bias - torch.Size([64]): [2022-05-31T14:24:45.106884Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.106889Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.106946Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,106 - mmcv - INFO - [2022-05-31T14:24:45.106953Z] dc5b74e6 [rank=0] || backbone.layer1.0.conv3.weight - torch.Size([256, 64, 1, 1]): [2022-05-31T14:24:45.106955Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.106958Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.106960Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,106 - mmcv - INFO - [2022-05-31T14:24:45.106963Z] dc5b74e6 [rank=0] || backbone.layer1.0.bn3.weight - torch.Size([256]): [2022-05-31T14:24:45.106969Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.106973Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.106977Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,106 - mmcv - INFO - [2022-05-31T14:24:45.106985Z] dc5b74e6 [rank=0] || backbone.layer1.0.bn3.bias - torch.Size([256]): [2022-05-31T14:24:45.106989Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.106994Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107008Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,106 - mmcv - INFO - [2022-05-31T14:24:45.107011Z] dc5b74e6 [rank=0] || backbone.layer1.0.downsample.0.weight - torch.Size([256, 64, 1, 1]): [2022-05-31T14:24:45.107016Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107021Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107027Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,106 - mmcv - INFO - [2022-05-31T14:24:45.107032Z] dc5b74e6 [rank=0] || backbone.layer1.0.downsample.1.weight - torch.Size([256]): [2022-05-31T14:24:45.107038Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107044Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107061Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,106 - mmcv - INFO - [2022-05-31T14:24:45.107068Z] dc5b74e6 [rank=0] || backbone.layer1.0.downsample.1.bias - torch.Size([256]): [2022-05-31T14:24:45.107072Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107075Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107097Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107100Z] dc5b74e6 [rank=0] || backbone.layer1.1.conv1.weight - torch.Size([64, 256, 1, 1]): [2022-05-31T14:24:45.107104Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107106Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107156Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107159Z] dc5b74e6 [rank=0] || backbone.layer1.1.bn1.weight - torch.Size([64]): [2022-05-31T14:24:45.107164Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107169Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107175Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107177Z] dc5b74e6 [rank=0] || backbone.layer1.1.bn1.bias - torch.Size([64]): [2022-05-31T14:24:45.107180Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107183Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107250Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107255Z] dc5b74e6 [rank=0] || backbone.layer1.1.conv2.weight - torch.Size([64, 64, 3, 3]): [2022-05-31T14:24:45.107258Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107261Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107266Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107269Z] dc5b74e6 [rank=0] || backbone.layer1.1.bn2.weight - torch.Size([64]): [2022-05-31T14:24:45.107271Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107276Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107309Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107312Z] dc5b74e6 [rank=0] || backbone.layer1.1.bn2.bias - torch.Size([64]): [2022-05-31T14:24:45.107315Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107317Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107345Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107349Z] dc5b74e6 [rank=0] || backbone.layer1.1.conv3.weight - torch.Size([256, 64, 1, 1]): [2022-05-31T14:24:45.107352Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107355Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107400Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107402Z] dc5b74e6 [rank=0] || backbone.layer1.1.bn3.weight - torch.Size([256]): [2022-05-31T14:24:45.107403Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107405Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107432Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107440Z] dc5b74e6 [rank=0] || backbone.layer1.1.bn3.bias - torch.Size([256]): [2022-05-31T14:24:45.107444Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107447Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107484Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107488Z] dc5b74e6 [rank=0] || backbone.layer1.2.conv1.weight - torch.Size([64, 256, 1, 1]): [2022-05-31T14:24:45.107490Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107497Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107516Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107520Z] dc5b74e6 [rank=0] || backbone.layer1.2.bn1.weight - torch.Size([64]): [2022-05-31T14:24:45.107524Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107528Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107558Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107564Z] dc5b74e6 [rank=0] || backbone.layer1.2.bn1.bias - torch.Size([64]): [2022-05-31T14:24:45.107571Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107576Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107604Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107608Z] dc5b74e6 [rank=0] || backbone.layer1.2.conv2.weight - torch.Size([64, 64, 3, 3]): [2022-05-31T14:24:45.107610Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107615Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107652Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107655Z] dc5b74e6 [rank=0] || backbone.layer1.2.bn2.weight - torch.Size([64]): [2022-05-31T14:24:45.107656Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107659Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107692Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107698Z] dc5b74e6 [rank=0] || backbone.layer1.2.bn2.bias - torch.Size([64]): [2022-05-31T14:24:45.107702Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107709Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107721Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107729Z] dc5b74e6 [rank=0] || backbone.layer1.2.conv3.weight - torch.Size([256, 64, 1, 1]): [2022-05-31T14:24:45.107733Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107738Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107876Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107880Z] dc5b74e6 [rank=0] || backbone.layer1.2.bn3.weight - torch.Size([256]): [2022-05-31T14:24:45.107882Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107883Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107885Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107886Z] dc5b74e6 [rank=0] || backbone.layer1.2.bn3.bias - torch.Size([256]): [2022-05-31T14:24:45.107890Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107895Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107899Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107904Z] dc5b74e6 [rank=0] || backbone.layer2.0.conv1.weight - torch.Size([128, 256, 1, 1]): [2022-05-31T14:24:45.107907Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107911Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107956Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107961Z] dc5b74e6 [rank=0] || backbone.layer2.0.bn1.weight - torch.Size([128]): [2022-05-31T14:24:45.107965Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107969Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107973Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.107978Z] dc5b74e6 [rank=0] || backbone.layer2.0.bn1.bias - torch.Size([128]): [2022-05-31T14:24:45.107983Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.107988Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.107994Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.108002Z] dc5b74e6 [rank=0] || backbone.layer2.0.conv2.weight - torch.Size([128, 128, 3, 3]): [2022-05-31T14:24:45.108008Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108013Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108023Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.108030Z] dc5b74e6 [rank=0] || backbone.layer2.0.bn2.weight - torch.Size([128]): [2022-05-31T14:24:45.108035Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108041Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108059Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,107 - mmcv - INFO - [2022-05-31T14:24:45.108063Z] dc5b74e6 [rank=0] || backbone.layer2.0.bn2.bias - torch.Size([128]): [2022-05-31T14:24:45.108066Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108070Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108111Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108115Z] dc5b74e6 [rank=0] || backbone.layer2.0.conv3.weight - torch.Size([512, 128, 1, 1]): [2022-05-31T14:24:45.108120Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108123Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108140Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108146Z] dc5b74e6 [rank=0] || backbone.layer2.0.bn3.weight - torch.Size([512]): [2022-05-31T14:24:45.108150Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108155Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108189Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108190Z] dc5b74e6 [rank=0] || backbone.layer2.0.bn3.bias - torch.Size([512]): [2022-05-31T14:24:45.108193Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108197Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108238Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108241Z] dc5b74e6 [rank=0] || backbone.layer2.0.downsample.0.weight - torch.Size([512, 256, 1, 1]): [2022-05-31T14:24:45.108243Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108246Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108280Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108285Z] dc5b74e6 [rank=0] || backbone.layer2.0.downsample.1.weight - torch.Size([512]): [2022-05-31T14:24:45.108293Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108298Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108305Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108309Z] dc5b74e6 [rank=0] || backbone.layer2.0.downsample.1.bias - torch.Size([512]): [2022-05-31T14:24:45.108313Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108318Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108379Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108383Z] dc5b74e6 [rank=0] || backbone.layer2.1.conv1.weight - torch.Size([128, 512, 1, 1]): [2022-05-31T14:24:45.108386Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108388Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108392Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108397Z] dc5b74e6 [rank=0] || backbone.layer2.1.bn1.weight - torch.Size([128]): [2022-05-31T14:24:45.108406Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108411Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108432Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108435Z] dc5b74e6 [rank=0] || backbone.layer2.1.bn1.bias - torch.Size([128]): [2022-05-31T14:24:45.108436Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108440Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108483Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108486Z] dc5b74e6 [rank=0] || backbone.layer2.1.conv2.weight - torch.Size([128, 128, 3, 3]): [2022-05-31T14:24:45.108488Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108489Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108521Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108526Z] dc5b74e6 [rank=0] || backbone.layer2.1.bn2.weight - torch.Size([128]): [2022-05-31T14:24:45.108530Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108534Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108545Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108550Z] dc5b74e6 [rank=0] || backbone.layer2.1.bn2.bias - torch.Size([128]): [2022-05-31T14:24:45.108556Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108561Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108598Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108599Z] dc5b74e6 [rank=0] || backbone.layer2.1.conv3.weight - torch.Size([512, 128, 1, 1]): [2022-05-31T14:24:45.108601Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108606Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108639Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108642Z] dc5b74e6 [rank=0] || backbone.layer2.1.bn3.weight - torch.Size([512]): [2022-05-31T14:24:45.108645Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108649Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108681Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108682Z] dc5b74e6 [rank=0] || backbone.layer2.1.bn3.bias - torch.Size([512]): [2022-05-31T14:24:45.108683Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108687Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108718Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108719Z] dc5b74e6 [rank=0] || backbone.layer2.2.conv1.weight - torch.Size([128, 512, 1, 1]): [2022-05-31T14:24:45.108793Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108797Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108804Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108809Z] dc5b74e6 [rank=0] || backbone.layer2.2.bn1.weight - torch.Size([128]): [2022-05-31T14:24:45.108814Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108818Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108825Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108829Z] dc5b74e6 [rank=0] || backbone.layer2.2.bn1.bias - torch.Size([128]): [2022-05-31T14:24:45.108834Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108838Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108877Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108880Z] dc5b74e6 [rank=0] || backbone.layer2.2.conv2.weight - torch.Size([128, 128, 3, 3]): [2022-05-31T14:24:45.108883Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108887Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108891Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108894Z] dc5b74e6 [rank=0] || backbone.layer2.2.bn2.weight - torch.Size([128]): [2022-05-31T14:24:45.108897Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108902Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108929Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108933Z] dc5b74e6 [rank=0] || backbone.layer2.2.bn2.bias - torch.Size([128]): [2022-05-31T14:24:45.108935Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108938Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.108966Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.108969Z] dc5b74e6 [rank=0] || backbone.layer2.2.conv3.weight - torch.Size([512, 128, 1, 1]): [2022-05-31T14:24:45.108971Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.108977Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109017Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.109021Z] dc5b74e6 [rank=0] || backbone.layer2.2.bn3.weight - torch.Size([512]): [2022-05-31T14:24:45.109025Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109031Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109043Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,108 - mmcv - INFO - [2022-05-31T14:24:45.109046Z] dc5b74e6 [rank=0] || backbone.layer2.2.bn3.bias - torch.Size([512]): [2022-05-31T14:24:45.109050Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109054Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109090Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109092Z] dc5b74e6 [rank=0] || backbone.layer2.3.conv1.weight - torch.Size([128, 512, 1, 1]): [2022-05-31T14:24:45.109094Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109097Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109143Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109145Z] dc5b74e6 [rank=0] || backbone.layer2.3.bn1.weight - torch.Size([128]): [2022-05-31T14:24:45.109148Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109153Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109184Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109185Z] dc5b74e6 [rank=0] || backbone.layer2.3.bn1.bias - torch.Size([128]): [2022-05-31T14:24:45.109187Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109192Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109224Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109225Z] dc5b74e6 [rank=0] || backbone.layer2.3.conv2.weight - torch.Size([128, 128, 3, 3]): [2022-05-31T14:24:45.109228Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109236Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109253Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109256Z] dc5b74e6 [rank=0] || backbone.layer2.3.bn2.weight - torch.Size([128]): [2022-05-31T14:24:45.109258Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109262Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109282Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109287Z] dc5b74e6 [rank=0] || backbone.layer2.3.bn2.bias - torch.Size([128]): [2022-05-31T14:24:45.109292Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109297Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109335Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109339Z] dc5b74e6 [rank=0] || backbone.layer2.3.conv3.weight - torch.Size([512, 128, 1, 1]): [2022-05-31T14:24:45.109343Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109347Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109379Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109382Z] dc5b74e6 [rank=0] || backbone.layer2.3.bn3.weight - torch.Size([512]): [2022-05-31T14:24:45.109384Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109390Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109419Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109420Z] dc5b74e6 [rank=0] || backbone.layer2.3.bn3.bias - torch.Size([512]): [2022-05-31T14:24:45.109422Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109427Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109459Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109464Z] dc5b74e6 [rank=0] || backbone.layer3.0.conv1.weight - torch.Size([256, 512, 1, 1]): [2022-05-31T14:24:45.109467Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109472Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109502Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109503Z] dc5b74e6 [rank=0] || backbone.layer3.0.bn1.weight - torch.Size([256]): [2022-05-31T14:24:45.109506Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109510Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109543Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109544Z] dc5b74e6 [rank=0] || backbone.layer3.0.bn1.bias - torch.Size([256]): [2022-05-31T14:24:45.109546Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109550Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109590Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109593Z] dc5b74e6 [rank=0] || backbone.layer3.0.conv2.weight - torch.Size([256, 256, 3, 3]): [2022-05-31T14:24:45.109597Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109602Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109647Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109650Z] dc5b74e6 [rank=0] || backbone.layer3.0.bn2.weight - torch.Size([256]): [2022-05-31T14:24:45.109651Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109655Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109671Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109675Z] dc5b74e6 [rank=0] || backbone.layer3.0.bn2.bias - torch.Size([256]): [2022-05-31T14:24:45.109680Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109689Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109711Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109715Z] dc5b74e6 [rank=0] || backbone.layer3.0.conv3.weight - torch.Size([1024, 256, 1, 1]): [2022-05-31T14:24:45.109718Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109723Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109753Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109754Z] dc5b74e6 [rank=0] || backbone.layer3.0.bn3.weight - torch.Size([1024]): [2022-05-31T14:24:45.109757Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109761Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109800Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109803Z] dc5b74e6 [rank=0] || backbone.layer3.0.bn3.bias - torch.Size([1024]): [2022-05-31T14:24:45.109806Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109810Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109845Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109850Z] dc5b74e6 [rank=0] || backbone.layer3.0.downsample.0.weight - torch.Size([1024, 512, 1, 1]): [2022-05-31T14:24:45.109855Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109860Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109892Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109893Z] dc5b74e6 [rank=0] || backbone.layer3.0.downsample.1.weight - torch.Size([1024]): [2022-05-31T14:24:45.109896Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109901Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109931Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109934Z] dc5b74e6 [rank=0] || backbone.layer3.0.downsample.1.bias - torch.Size([1024]): [2022-05-31T14:24:45.109936Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109939Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.109977Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.109983Z] dc5b74e6 [rank=0] || backbone.layer3.1.conv1.weight - torch.Size([256, 1024, 1, 1]): [2022-05-31T14:24:45.109987Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.109992Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110017Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.110020Z] dc5b74e6 [rank=0] || backbone.layer3.1.bn1.weight - torch.Size([256]): [2022-05-31T14:24:45.110024Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110029Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110059Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,109 - mmcv - INFO - [2022-05-31T14:24:45.110060Z] dc5b74e6 [rank=0] || backbone.layer3.1.bn1.bias - torch.Size([256]): [2022-05-31T14:24:45.110063Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110068Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110099Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110102Z] dc5b74e6 [rank=0] || backbone.layer3.1.conv2.weight - torch.Size([256, 256, 3, 3]): [2022-05-31T14:24:45.110103Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110109Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110139Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110140Z] dc5b74e6 [rank=0] || backbone.layer3.1.bn2.weight - torch.Size([256]): [2022-05-31T14:24:45.110141Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110146Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110183Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110188Z] dc5b74e6 [rank=0] || backbone.layer3.1.bn2.bias - torch.Size([256]): [2022-05-31T14:24:45.110193Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110198Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110214Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110219Z] dc5b74e6 [rank=0] || backbone.layer3.1.conv3.weight - torch.Size([1024, 256, 1, 1]): [2022-05-31T14:24:45.110222Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110226Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110246Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110251Z] dc5b74e6 [rank=0] || backbone.layer3.1.bn3.weight - torch.Size([1024]): [2022-05-31T14:24:45.110255Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110259Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110299Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110301Z] dc5b74e6 [rank=0] || backbone.layer3.1.bn3.bias - torch.Size([1024]): [2022-05-31T14:24:45.110304Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110309Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110363Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110367Z] dc5b74e6 [rank=0] || backbone.layer3.2.conv1.weight - torch.Size([256, 1024, 1, 1]): [2022-05-31T14:24:45.110391Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110393Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110398Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110403Z] dc5b74e6 [rank=0] || backbone.layer3.2.bn1.weight - torch.Size([256]): [2022-05-31T14:24:45.110407Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110412Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110439Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110442Z] dc5b74e6 [rank=0] || backbone.layer3.2.bn1.bias - torch.Size([256]): [2022-05-31T14:24:45.110444Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110447Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110474Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110476Z] dc5b74e6 [rank=0] || backbone.layer3.2.conv2.weight - torch.Size([256, 256, 3, 3]): [2022-05-31T14:24:45.110479Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110483Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110515Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110516Z] dc5b74e6 [rank=0] || backbone.layer3.2.bn2.weight - torch.Size([256]): [2022-05-31T14:24:45.110517Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110523Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110556Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110558Z] dc5b74e6 [rank=0] || backbone.layer3.2.bn2.bias - torch.Size([256]): [2022-05-31T14:24:45.110559Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110562Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110595Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110600Z] dc5b74e6 [rank=0] || backbone.layer3.2.conv3.weight - torch.Size([1024, 256, 1, 1]): [2022-05-31T14:24:45.110605Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110610Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110639Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110643Z] dc5b74e6 [rank=0] || backbone.layer3.2.bn3.weight - torch.Size([1024]): [2022-05-31T14:24:45.110644Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110648Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110682Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110687Z] dc5b74e6 [rank=0] || backbone.layer3.2.bn3.bias - torch.Size([1024]): [2022-05-31T14:24:45.110691Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110695Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110723Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110734Z] dc5b74e6 [rank=0] || backbone.layer3.3.conv1.weight - torch.Size([256, 1024, 1, 1]): [2022-05-31T14:24:45.110739Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110745Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110766Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110769Z] dc5b74e6 [rank=0] || backbone.layer3.3.bn1.weight - torch.Size([256]): [2022-05-31T14:24:45.110771Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110774Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110793Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110795Z] dc5b74e6 [rank=0] || backbone.layer3.3.bn1.bias - torch.Size([256]): [2022-05-31T14:24:45.110799Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110804Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110845Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110848Z] dc5b74e6 [rank=0] || backbone.layer3.3.conv2.weight - torch.Size([256, 256, 3, 3]): [2022-05-31T14:24:45.110853Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110857Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110886Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110896Z] dc5b74e6 [rank=0] || backbone.layer3.3.bn2.weight - torch.Size([256]): [2022-05-31T14:24:45.110901Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110906Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110925Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110927Z] dc5b74e6 [rank=0] || backbone.layer3.3.bn2.bias - torch.Size([256]): [2022-05-31T14:24:45.110929Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110932Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.110967Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.110969Z] dc5b74e6 [rank=0] || backbone.layer3.3.conv3.weight - torch.Size([1024, 256, 1, 1]): [2022-05-31T14:24:45.110972Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.110977Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111006Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.111007Z] dc5b74e6 [rank=0] || backbone.layer3.3.bn3.weight - torch.Size([1024]): [2022-05-31T14:24:45.111010Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111015Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111051Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,110 - mmcv - INFO - [2022-05-31T14:24:45.111054Z] dc5b74e6 [rank=0] || backbone.layer3.3.bn3.bias - torch.Size([1024]): [2022-05-31T14:24:45.111055Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111059Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111098Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111099Z] dc5b74e6 [rank=0] || backbone.layer3.4.conv1.weight - torch.Size([256, 1024, 1, 1]): [2022-05-31T14:24:45.111103Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111107Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111143Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111144Z] dc5b74e6 [rank=0] || backbone.layer3.4.bn1.weight - torch.Size([256]): [2022-05-31T14:24:45.111145Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111152Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111192Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111193Z] dc5b74e6 [rank=0] || backbone.layer3.4.bn1.bias - torch.Size([256]): [2022-05-31T14:24:45.111195Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111200Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111229Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111237Z] dc5b74e6 [rank=0] || backbone.layer3.4.conv2.weight - torch.Size([256, 256, 3, 3]): [2022-05-31T14:24:45.111241Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111246Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111312Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111316Z] dc5b74e6 [rank=0] || backbone.layer3.4.bn2.weight - torch.Size([256]): [2022-05-31T14:24:45.111318Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111320Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111330Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111343Z] dc5b74e6 [rank=0] || backbone.layer3.4.bn2.bias - torch.Size([256]): [2022-05-31T14:24:45.111347Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111352Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111387Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111389Z] dc5b74e6 [rank=0] || backbone.layer3.4.conv3.weight - torch.Size([1024, 256, 1, 1]): [2022-05-31T14:24:45.111391Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111393Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111396Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111401Z] dc5b74e6 [rank=0] || backbone.layer3.4.bn3.weight - torch.Size([1024]): [2022-05-31T14:24:45.111404Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111407Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111421Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111426Z] dc5b74e6 [rank=0] || backbone.layer3.4.bn3.bias - torch.Size([1024]): [2022-05-31T14:24:45.111430Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111434Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111473Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111478Z] dc5b74e6 [rank=0] || backbone.layer3.5.conv1.weight - torch.Size([256, 1024, 1, 1]): [2022-05-31T14:24:45.111482Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111487Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111514Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111515Z] dc5b74e6 [rank=0] || backbone.layer3.5.bn1.weight - torch.Size([256]): [2022-05-31T14:24:45.111517Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111521Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111550Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111552Z] dc5b74e6 [rank=0] || backbone.layer3.5.bn1.bias - torch.Size([256]): [2022-05-31T14:24:45.111554Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111559Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111591Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111592Z] dc5b74e6 [rank=0] || backbone.layer3.5.conv2.weight - torch.Size([256, 256, 3, 3]): [2022-05-31T14:24:45.111593Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111597Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111624Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111625Z] dc5b74e6 [rank=0] || backbone.layer3.5.bn2.weight - torch.Size([256]): [2022-05-31T14:24:45.111626Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111644Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111672Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111673Z] dc5b74e6 [rank=0] || backbone.layer3.5.bn2.bias - torch.Size([256]): [2022-05-31T14:24:45.111675Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111680Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111722Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111727Z] dc5b74e6 [rank=0] || backbone.layer3.5.conv3.weight - torch.Size([1024, 256, 1, 1]): [2022-05-31T14:24:45.111732Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111737Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111752Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111754Z] dc5b74e6 [rank=0] || backbone.layer3.5.bn3.weight - torch.Size([1024]): [2022-05-31T14:24:45.111757Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111761Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111780Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111782Z] dc5b74e6 [rank=0] || backbone.layer3.5.bn3.bias - torch.Size([1024]): [2022-05-31T14:24:45.111783Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111788Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111847Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111854Z] dc5b74e6 [rank=0] || backbone.layer4.0.conv1.weight - torch.Size([512, 1024, 1, 1]): [2022-05-31T14:24:45.111856Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111861Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111894Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111900Z] dc5b74e6 [rank=0] || backbone.layer4.0.bn1.weight - torch.Size([512]): [2022-05-31T14:24:45.111905Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111911Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111922Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111928Z] dc5b74e6 [rank=0] || backbone.layer4.0.bn1.bias - torch.Size([512]): [2022-05-31T14:24:45.111932Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111936Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.111956Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.111959Z] dc5b74e6 [rank=0] || backbone.layer4.0.conv2.weight - torch.Size([512, 512, 3, 3]): [2022-05-31T14:24:45.111961Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.111962Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112012Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.112015Z] dc5b74e6 [rank=0] || backbone.layer4.0.bn2.weight - torch.Size([512]): [2022-05-31T14:24:45.112016Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112022Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112043Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,111 - mmcv - INFO - [2022-05-31T14:24:45.112045Z] dc5b74e6 [rank=0] || backbone.layer4.0.bn2.bias - torch.Size([512]): [2022-05-31T14:24:45.112046Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112051Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112092Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112093Z] dc5b74e6 [rank=0] || backbone.layer4.0.conv3.weight - torch.Size([2048, 512, 1, 1]): [2022-05-31T14:24:45.112096Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112101Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112131Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112134Z] dc5b74e6 [rank=0] || backbone.layer4.0.bn3.weight - torch.Size([2048]): [2022-05-31T14:24:45.112135Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112142Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112171Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112174Z] dc5b74e6 [rank=0] || backbone.layer4.0.bn3.bias - torch.Size([2048]): [2022-05-31T14:24:45.112180Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112184Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112215Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112220Z] dc5b74e6 [rank=0] || backbone.layer4.0.downsample.0.weight - torch.Size([2048, 1024, 1, 1]): [2022-05-31T14:24:45.112224Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112229Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112258Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112263Z] dc5b74e6 [rank=0] || backbone.layer4.0.downsample.1.weight - torch.Size([2048]): [2022-05-31T14:24:45.112267Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112273Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112306Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112310Z] dc5b74e6 [rank=0] || backbone.layer4.0.downsample.1.bias - torch.Size([2048]): [2022-05-31T14:24:45.112311Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112316Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112347Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112352Z] dc5b74e6 [rank=0] || backbone.layer4.1.conv1.weight - torch.Size([512, 2048, 1, 1]): [2022-05-31T14:24:45.112356Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112362Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112396Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112402Z] dc5b74e6 [rank=0] || backbone.layer4.1.bn1.weight - torch.Size([512]): [2022-05-31T14:24:45.112407Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112412Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112431Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112437Z] dc5b74e6 [rank=0] || backbone.layer4.1.bn1.bias - torch.Size([512]): [2022-05-31T14:24:45.112440Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112444Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112451Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112456Z] dc5b74e6 [rank=0] || backbone.layer4.1.conv2.weight - torch.Size([512, 512, 3, 3]): [2022-05-31T14:24:45.112460Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112465Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112491Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112494Z] dc5b74e6 [rank=0] || backbone.layer4.1.bn2.weight - torch.Size([512]): [2022-05-31T14:24:45.112496Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112501Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112542Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112546Z] dc5b74e6 [rank=0] || backbone.layer4.1.bn2.bias - torch.Size([512]): [2022-05-31T14:24:45.112552Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112557Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112580Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112585Z] dc5b74e6 [rank=0] || backbone.layer4.1.conv3.weight - torch.Size([2048, 512, 1, 1]): [2022-05-31T14:24:45.112590Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112594Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112621Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112625Z] dc5b74e6 [rank=0] || backbone.layer4.1.bn3.weight - torch.Size([2048]): [2022-05-31T14:24:45.112629Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112634Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112661Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112665Z] dc5b74e6 [rank=0] || backbone.layer4.1.bn3.bias - torch.Size([2048]): [2022-05-31T14:24:45.112668Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112673Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112714Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112715Z] dc5b74e6 [rank=0] || backbone.layer4.2.conv1.weight - torch.Size([512, 2048, 1, 1]): [2022-05-31T14:24:45.112716Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112719Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112750Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112755Z] dc5b74e6 [rank=0] || backbone.layer4.2.bn1.weight - torch.Size([512]): [2022-05-31T14:24:45.112759Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112764Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112797Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112801Z] dc5b74e6 [rank=0] || backbone.layer4.2.bn1.bias - torch.Size([512]): [2022-05-31T14:24:45.112803Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112809Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112833Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112837Z] dc5b74e6 [rank=0] || backbone.layer4.2.conv2.weight - torch.Size([512, 512, 3, 3]): [2022-05-31T14:24:45.112841Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112845Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112868Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112873Z] dc5b74e6 [rank=0] || backbone.layer4.2.bn2.weight - torch.Size([512]): [2022-05-31T14:24:45.112877Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112883Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112919Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112923Z] dc5b74e6 [rank=0] || backbone.layer4.2.bn2.bias - torch.Size([512]): [2022-05-31T14:24:45.112925Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112930Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112958Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112965Z] dc5b74e6 [rank=0] || backbone.layer4.2.conv3.weight - torch.Size([2048, 512, 1, 1]): [2022-05-31T14:24:45.112970Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.112975Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.112989Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.112992Z] dc5b74e6 [rank=0] || backbone.layer4.2.bn3.weight - torch.Size([2048]): [2022-05-31T14:24:45.112997Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.113001Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113013Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,112 - mmcv - INFO - [2022-05-31T14:24:45.113015Z] dc5b74e6 [rank=0] || backbone.layer4.2.bn3.bias - torch.Size([2048]): [2022-05-31T14:24:45.113019Z] dc5b74e6 [rank=0] || PretrainedInit: load from torchvision://resnet50 [2022-05-31T14:24:45.113023Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113088Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113091Z] dc5b74e6 [rank=0] || neck.lateral_convs.0.conv.weight - torch.Size([256, 256, 1, 1]): [2022-05-31T14:24:45.113092Z] dc5b74e6 [rank=0] || XavierInit: gain=1, distribution=uniform, bias=0 [2022-05-31T14:24:45.113098Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113131Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113132Z] dc5b74e6 [rank=0] || neck.lateral_convs.0.conv.bias - torch.Size([256]): [2022-05-31T14:24:45.113135Z] dc5b74e6 [rank=0] || The value is the same before and after calling init_weights of FasterRCNN
[2022-05-31T14:24:45.113141Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113192Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113194Z] dc5b74e6 [rank=0] || neck.lateral_convs.1.conv.weight - torch.Size([256, 512, 1, 1]): [2022-05-31T14:24:45.113196Z] dc5b74e6 [rank=0] || XavierInit: gain=1, distribution=uniform, bias=0 [2022-05-31T14:24:45.113201Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113205Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113210Z] dc5b74e6 [rank=0] || neck.lateral_convs.1.conv.bias - torch.Size([256]): [2022-05-31T14:24:45.113213Z] dc5b74e6 [rank=0] || The value is the same before and after calling init_weights of FasterRCNN
[2022-05-31T14:24:45.113218Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113254Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113259Z] dc5b74e6 [rank=0] || neck.lateral_convs.2.conv.weight - torch.Size([256, 1024, 1, 1]): [2022-05-31T14:24:45.113266Z] dc5b74e6 [rank=0] || XavierInit: gain=1, distribution=uniform, bias=0 [2022-05-31T14:24:45.113270Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113277Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113283Z] dc5b74e6 [rank=0] || neck.lateral_convs.2.conv.bias - torch.Size([256]): [2022-05-31T14:24:45.113307Z] dc5b74e6 [rank=0] || The value is the same before and after calling init_weights of FasterRCNN
[2022-05-31T14:24:45.113311Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113328Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113340Z] dc5b74e6 [rank=0] || neck.lateral_convs.3.conv.weight - torch.Size([256, 2048, 1, 1]): [2022-05-31T14:24:45.113344Z] dc5b74e6 [rank=0] || XavierInit: gain=1, distribution=uniform, bias=0 [2022-05-31T14:24:45.113349Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113366Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113370Z] dc5b74e6 [rank=0] || neck.lateral_convs.3.conv.bias - torch.Size([256]): [2022-05-31T14:24:45.113375Z] dc5b74e6 [rank=0] || The value is the same before and after calling init_weights of FasterRCNN
[2022-05-31T14:24:45.113379Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113420Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113422Z] dc5b74e6 [rank=0] || neck.fpn_convs.0.conv.weight - torch.Size([256, 256, 3, 3]): [2022-05-31T14:24:45.113425Z] dc5b74e6 [rank=0] || XavierInit: gain=1, distribution=uniform, bias=0 [2022-05-31T14:24:45.113432Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113463Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113465Z] dc5b74e6 [rank=0] || neck.fpn_convs.0.conv.bias - torch.Size([256]): [2022-05-31T14:24:45.113468Z] dc5b74e6 [rank=0] || The value is the same before and after calling init_weights of FasterRCNN
[2022-05-31T14:24:45.113473Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113506Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113510Z] dc5b74e6 [rank=0] || neck.fpn_convs.1.conv.weight - torch.Size([256, 256, 3, 3]): [2022-05-31T14:24:45.113514Z] dc5b74e6 [rank=0] || XavierInit: gain=1, distribution=uniform, bias=0 [2022-05-31T14:24:45.113519Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113554Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113557Z] dc5b74e6 [rank=0] || neck.fpn_convs.1.conv.bias - torch.Size([256]): [2022-05-31T14:24:45.113560Z] dc5b74e6 [rank=0] || The value is the same before and after calling init_weights of FasterRCNN
[2022-05-31T14:24:45.113564Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113590Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113596Z] dc5b74e6 [rank=0] || neck.fpn_convs.2.conv.weight - torch.Size([256, 256, 3, 3]): [2022-05-31T14:24:45.113601Z] dc5b74e6 [rank=0] || XavierInit: gain=1, distribution=uniform, bias=0 [2022-05-31T14:24:45.113606Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113634Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113639Z] dc5b74e6 [rank=0] || neck.fpn_convs.2.conv.bias - torch.Size([256]): [2022-05-31T14:24:45.113642Z] dc5b74e6 [rank=0] || The value is the same before and after calling init_weights of FasterRCNN
[2022-05-31T14:24:45.113647Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113653Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113658Z] dc5b74e6 [rank=0] || neck.fpn_convs.3.conv.weight - torch.Size([256, 256, 3, 3]): [2022-05-31T14:24:45.113662Z] dc5b74e6 [rank=0] || XavierInit: gain=1, distribution=uniform, bias=0 [2022-05-31T14:24:45.113665Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113703Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113706Z] dc5b74e6 [rank=0] || neck.fpn_convs.3.conv.bias - torch.Size([256]): [2022-05-31T14:24:45.113710Z] dc5b74e6 [rank=0] || The value is the same before and after calling init_weights of FasterRCNN
[2022-05-31T14:24:45.113715Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113745Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113746Z] dc5b74e6 [rank=0] || rpn_head.rpn_conv.weight - torch.Size([256, 256, 3, 3]): [2022-05-31T14:24:45.113749Z] dc5b74e6 [rank=0] || NormalInit: mean=0, std=0.01, bias=0 [2022-05-31T14:24:45.113754Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113789Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113792Z] dc5b74e6 [rank=0] || rpn_head.rpn_conv.bias - torch.Size([256]): [2022-05-31T14:24:45.113793Z] dc5b74e6 [rank=0] || NormalInit: mean=0, std=0.01, bias=0 [2022-05-31T14:24:45.113797Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113828Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113829Z] dc5b74e6 [rank=0] || rpn_head.rpn_cls.weight - torch.Size([3, 256, 1, 1]): [2022-05-31T14:24:45.113830Z] dc5b74e6 [rank=0] || NormalInit: mean=0, std=0.01, bias=0 [2022-05-31T14:24:45.113835Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113861Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113862Z] dc5b74e6 [rank=0] || rpn_head.rpn_cls.bias - torch.Size([3]): [2022-05-31T14:24:45.113864Z] dc5b74e6 [rank=0] || NormalInit: mean=0, std=0.01, bias=0 [2022-05-31T14:24:45.113868Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113917Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113922Z] dc5b74e6 [rank=0] || rpn_head.rpn_reg.weight - torch.Size([12, 256, 1, 1]): [2022-05-31T14:24:45.113925Z] dc5b74e6 [rank=0] || NormalInit: mean=0, std=0.01, bias=0 [2022-05-31T14:24:45.113930Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.113960Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.113961Z] dc5b74e6 [rank=0] || rpn_head.rpn_reg.bias - torch.Size([12]): [2022-05-31T14:24:45.113963Z] dc5b74e6 [rank=0] || NormalInit: mean=0, std=0.01, bias=0 [2022-05-31T14:24:45.113968Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.114022Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.114024Z] dc5b74e6 [rank=0] || roi_head.bbox_head.fc_cls.weight - torch.Size([81, 1024]): [2022-05-31T14:24:45.114027Z] dc5b74e6 [rank=0] || NormalInit: mean=0, std=0.01, bias=0 [2022-05-31T14:24:45.114031Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.114063Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,113 - mmcv - INFO - [2022-05-31T14:24:45.114064Z] dc5b74e6 [rank=0] || roi_head.bbox_head.fc_cls.bias - torch.Size([81]): [2022-05-31T14:24:45.114067Z] dc5b74e6 [rank=0] || NormalInit: mean=0, std=0.01, bias=0 [2022-05-31T14:24:45.114072Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.114103Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,114 - mmcv - INFO - [2022-05-31T14:24:45.114104Z] dc5b74e6 [rank=0] || roi_head.bbox_head.fc_reg.weight - torch.Size([320, 1024]): [2022-05-31T14:24:45.114106Z] dc5b74e6 [rank=0] || NormalInit: mean=0, std=0.001, bias=0 [2022-05-31T14:24:45.114110Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.114132Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,114 - mmcv - INFO - [2022-05-31T14:24:45.114135Z] dc5b74e6 [rank=0] || roi_head.bbox_head.fc_reg.bias - torch.Size([320]): [2022-05-31T14:24:45.114139Z] dc5b74e6 [rank=0] || NormalInit: mean=0, std=0.001, bias=0 [2022-05-31T14:24:45.114144Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.114175Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,114 - mmcv - INFO - [2022-05-31T14:24:45.114176Z] dc5b74e6 [rank=0] || roi_head.bbox_head.shared_fcs.0.weight - torch.Size([1024, 12544]): [2022-05-31T14:24:45.114179Z] dc5b74e6 [rank=0] || XavierInit: gain=1, distribution=uniform, bias=0 [2022-05-31T14:24:45.114184Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.114222Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,114 - mmcv - INFO - [2022-05-31T14:24:45.114227Z] dc5b74e6 [rank=0] || roi_head.bbox_head.shared_fcs.0.bias - torch.Size([1024]): [2022-05-31T14:24:45.114230Z] dc5b74e6 [rank=0] || XavierInit: gain=1, distribution=uniform, bias=0 [2022-05-31T14:24:45.114256Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.114262Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,114 - mmcv - INFO - [2022-05-31T14:24:45.114268Z] dc5b74e6 [rank=0] || roi_head.bbox_head.shared_fcs.1.weight - torch.Size([1024, 1024]): [2022-05-31T14:24:45.114272Z] dc5b74e6 [rank=0] || XavierInit: gain=1, distribution=uniform, bias=0 [2022-05-31T14:24:45.114277Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.114304Z] dc5b74e6 [rank=0] || 2022-05-31 14:24:45,114 - mmcv - INFO - [2022-05-31T14:24:45.114324Z] dc5b74e6 [rank=0] || roi_head.bbox_head.shared_fcs.1.bias - torch.Size([1024]): [2022-05-31T14:24:45.114331Z] dc5b74e6 [rank=0] || XavierInit: gain=1, distribution=uniform, bias=0 [2022-05-31T14:24:45.114335Z] dc5b74e6 [rank=0] ||
[2022-05-31T14:24:45.194969Z] dc5b74e6 [rank=0] || INFO: [49] root: Creating PyTorchTrialController with MMDetTrial. [2022-05-31T14:24:45.195053Z] dc5b74e6 [rank=0] || INFO: [49] root: Adding lr updater callback. [2022-05-31T14:24:45.195422Z] dc5b74e6 [rank=0] || DEBUG: [49] root: Evaluate_batch_defined: True. [2022-05-31T14:24:45.195426Z] dc5b74e6 [rank=0] || DEBUG: [49] root: Evaluate full dataset defined: False. [2022-05-31T14:24:45.197983Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: Starting new HTTP connection (1): 10.6.5.8:8080 [2022-05-31T14:24:45.205408Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: http://10.6.5.8:8080 "GET /api/v1/trials/32 HTTP/1.1" 200 722 [2022-05-31T14:24:45.206021Z] dc5b74e6 [rank=0] || DEBUG: [49] determined.core: _get_last_validation() -> None [2022-05-31T14:24:45.206569Z] dc5b74e6 [rank=0] || DEBUG: [49] filelock: Attempting to acquire lock 139856684735264 on /tmp/coco/annotations/instances_train2017.json.lock [2022-05-31T14:24:45.206641Z] dc5b74e6 [rank=0] || DEBUG: [49] filelock: Lock 139856684735264 acquired on /tmp/coco/annotations/instances_train2017.json.lock [2022-05-31T14:24:45.206704Z] dc5b74e6 [rank=0] || DEBUG: [49] filelock: Attempting to release lock 139856684735264 on /tmp/coco/annotations/instances_train2017.json.lock [2022-05-31T14:24:45.206763Z] dc5b74e6 [rank=0] || DEBUG: [49] filelock: Lock 139856684735264 released on /tmp/coco/annotations/instances_train2017.json.lock [2022-05-31T14:24:45.207111Z] dc5b74e6 [rank=0] || loading annotations into memory... [2022-05-31T14:24:45.207413Z] dc5b74e6 [rank=0] || Done (t=0.00s) [2022-05-31T14:24:45.207416Z] dc5b74e6 [rank=0] || creating index... [2022-05-31T14:24:45.207461Z] dc5b74e6 [rank=0] || index created! [2022-05-31T14:24:45.209761Z] dc5b74e6 [rank=0] || DEBUG: [49] filelock: Attempting to acquire lock 139856685134704 on /tmp/coco/annotations/instances_val2017.json.lock [2022-05-31T14:24:45.209765Z] dc5b74e6 [rank=0] || DEBUG: [49] filelock: Lock 139856685134704 acquired on /tmp/coco/annotations/instances_val2017.json.lock [2022-05-31T14:24:45.209766Z] dc5b74e6 [rank=0] || DEBUG: [49] filelock: Attempting to release lock 139856685134704 on /tmp/coco/annotations/instances_val2017.json.lock [2022-05-31T14:24:45.209768Z] dc5b74e6 [rank=0] || DEBUG: [49] filelock: Lock 139856685134704 released on /tmp/coco/annotations/instances_val2017.json.lock [2022-05-31T14:24:45.209929Z] dc5b74e6 [rank=0] || loading annotations into memory... [2022-05-31T14:24:45.210094Z] dc5b74e6 [rank=0] || Done (t=0.00s) [2022-05-31T14:24:45.210098Z] dc5b74e6 [rank=0] || creating index... [2022-05-31T14:24:45.210099Z] dc5b74e6 [rank=0] || index created! [2022-05-31T14:24:45.232508Z] dc5b74e6 [rank=1] || INFO: [52] root: Creating PyTorchTrialController with MMDetTrial. [2022-05-31T14:24:45.232512Z] dc5b74e6 [rank=1] || INFO: [52] root: Adding lr updater callback. [2022-05-31T14:24:45.232706Z] dc5b74e6 [rank=1] || DEBUG: [52] root: Evaluate_batch_defined: True. [2022-05-31T14:24:45.232711Z] dc5b74e6 [rank=1] || DEBUG: [52] root: Evaluate full dataset defined: False. [2022-05-31T14:24:45.232997Z] dc5b74e6 [rank=1] || DEBUG: [52] filelock: Attempting to acquire lock 139778235221520 on /tmp/coco/annotations/instances_train2017.json.lock [2022-05-31T14:24:45.233040Z] dc5b74e6 [rank=1] || DEBUG: [52] filelock: Lock 139778235221520 acquired on /tmp/coco/annotations/instances_train2017.json.lock [2022-05-31T14:24:45.233095Z] dc5b74e6 [rank=1] || DEBUG: [52] filelock: Attempting to release lock 139778235221520 on /tmp/coco/annotations/instances_train2017.json.lock [2022-05-31T14:24:45.233127Z] dc5b74e6 [rank=1] || DEBUG: [52] filelock: Lock 139778235221520 released on /tmp/coco/annotations/instances_train2017.json.lock [2022-05-31T14:24:45.233319Z] dc5b74e6 [rank=1] || loading annotations into memory... [2022-05-31T14:24:45.233535Z] dc5b74e6 [rank=1] || Done (t=0.00s) [2022-05-31T14:24:45.233538Z] dc5b74e6 [rank=1] || creating index... [2022-05-31T14:24:45.233561Z] dc5b74e6 [rank=1] || index created! [2022-05-31T14:24:45.234936Z] dc5b74e6 [rank=1] || DEBUG: [52] filelock: Attempting to acquire lock 139778235223440 on /tmp/coco/annotations/instances_val2017.json.lock [2022-05-31T14:24:45.234941Z] dc5b74e6 [rank=1] || DEBUG: [52] filelock: Lock 139778235223440 acquired on /tmp/coco/annotations/instances_val2017.json.lock [2022-05-31T14:24:45.234943Z] dc5b74e6 [rank=1] || DEBUG: [52] filelock: Attempting to release lock 139778235223440 on /tmp/coco/annotations/instances_val2017.json.lock [2022-05-31T14:24:45.234945Z] dc5b74e6 [rank=1] || DEBUG: [52] filelock: Lock 139778235223440 released on /tmp/coco/annotations/instances_val2017.json.lock [2022-05-31T14:24:45.235082Z] dc5b74e6 [rank=1] || loading annotations into memory... [2022-05-31T14:24:45.235215Z] dc5b74e6 [rank=1] || Done (t=0.00s) [2022-05-31T14:24:45.235221Z] dc5b74e6 [rank=1] || creating index... [2022-05-31T14:24:45.235223Z] dc5b74e6 [rank=1] || index created! [2022-05-31T14:25:14.120198Z] dc5b74e6 [rank=0] || Timeout (0:00:30)! [2022-05-31T14:25:14.120206Z] dc5b74e6 [rank=0] || Thread 0x00007f32e68e1700 (most recent call first): [2022-05-31T14:25:14.120208Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31T14:25:14.120213Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31T14:25:14.120215Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31T14:25:14.120219Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:25:14.120330Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:25:14.120340Z] dc5b74e6 [rank=0] || [2022-05-31T14:25:14.120343Z] dc5b74e6 [rank=0] || Thread 0x00007f32eedf4700 (most recent call first): [2022-05-31T14:25:14.120353Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31T14:25:14.120427Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31T14:25:14.120486Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31T14:25:14.120548Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31T14:25:14.120669Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31T14:25:14.120766Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31T14:25:14.120865Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31T14:25:14.120936Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31T14:25:14.121030Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request [2022-05-31T14:25:14.121149Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/requests.py", line 41 in request [2022-05-31T14:25:14.121288Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/api/request.py", line 104 in do_request [2022-05-31T14:25:14.121434Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 65 in _do_request [2022-05-31T14:25:14.121576Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 86 in get [2022-05-31T14:25:14.121709Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 57 in _get_preemption [2022-05-31T14:25:14.121827Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 89 in run [2022-05-31T14:25:14.121892Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:25:14.121981Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:25:14.121985Z] dc5b74e6 [rank=0] || [2022-05-31T14:25:14.121988Z] dc5b74e6 [rank=0] || Thread 0x00007f32ef5f5700 (most recent call first): [2022-05-31T14:25:14.122052Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:25:14.122183Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:25:14.122302Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:25:14.122401Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:25:14.122453Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:25:14.122457Z] dc5b74e6 [rank=0] || [2022-05-31T14:25:14.122460Z] dc5b74e6 [rank=0] || Thread 0x00007f34546e3240 (most recent call first): [2022-05-31T14:25:14.122576Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:25:14.122673Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:25:14.122813Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:25:14.122943Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:25:14.123050Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:25:14.123163Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:25:14.123271Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:25:14.123384Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:25:14.123490Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:25:14.123599Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:25:14.123714Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:25:14.123832Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:25:14.123950Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:25:14.124001Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:25:14.124080Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:25:14.124453Z] dc5b74e6 [rank=1] || Timeout (0:00:30)! [2022-05-31T14:25:14.124456Z] dc5b74e6 [rank=1] || Thread 0x00007f20ab5f5700 (most recent call first): [2022-05-31T14:25:14.124458Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:25:14.124615Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:25:14.124756Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:25:14.124816Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:25:14.124904Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:25:14.124909Z] dc5b74e6 [rank=1] || [2022-05-31T14:25:14.124911Z] dc5b74e6 [rank=1] || Thread 0x00007f2210724240 (most recent call first): [2022-05-31T14:25:14.125022Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:25:14.125135Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:25:14.125300Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:25:14.125437Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:25:14.125564Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:25:14.125691Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:25:14.125804Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:25:14.125922Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:25:14.126038Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:25:14.126151Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:25:14.126294Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:25:14.126451Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:25:14.126573Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:25:14.126631Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:25:14.126711Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:25:26.363388Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: http://10.6.5.8:8080 "GET /api/v1/allocations/34.29c616a1-c0bf-418c-a8bd-b855ef8ac23d.1/signals/preemption?timeout_seconds=60 HTTP/1.1" 200 17 [2022-05-31T14:25:26.367086Z] dc5b74e6 [rank=0] || DEBUG: [49] determined.core: (_PreemptionWatcher thread) _get_preemption(60) [2022-05-31T14:25:26.373398Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: Starting new HTTP connection (1): 10.6.5.8:8080 [2022-05-31T14:25:44.124513Z] dc5b74e6 [rank=0] || Timeout (0:00:30)! [2022-05-31T14:25:44.124521Z] dc5b74e6 [rank=0] || Thread 0x00007f32e68e1700 (most recent call first): [2022-05-31T14:25:44.124525Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31T14:25:44.124528Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31T14:25:44.124531Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31T14:25:44.124534Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:25:44.124578Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:25:44.124585Z] dc5b74e6 [rank=0] || [2022-05-31T14:25:44.124587Z] dc5b74e6 [rank=0] || Thread 0x00007f32eedf4700 (most recent call first): [2022-05-31T14:25:44.124609Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31T14:25:44.124679Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31T14:25:44.124776Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31T14:25:44.124818Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31T14:25:44.124924Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31T14:25:44.125029Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31T14:25:44.125122Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31T14:25:44.125217Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31T14:25:44.125311Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request [2022-05-31T14:25:44.125447Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/requests.py", line 41 in request [2022-05-31T14:25:44.125587Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/api/request.py", line 104 in do_request [2022-05-31T14:25:44.125738Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 65 in _do_request [2022-05-31T14:25:44.125874Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 86 in get [2022-05-31T14:25:44.126001Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 57 in _get_preemption [2022-05-31T14:25:44.126123Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 89 in run [2022-05-31T14:25:44.126187Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:25:44.126254Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:25:44.126257Z] dc5b74e6 [rank=0] || [2022-05-31T14:25:44.126258Z] dc5b74e6 [rank=0] || Thread 0x00007f32ef5f5700 (most recent call first): [2022-05-31T14:25:44.126342Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:25:44.126467Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:25:44.126581Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:25:44.126646Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:25:44.126714Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:25:44.126716Z] dc5b74e6 [rank=0] || [2022-05-31T14:25:44.126717Z] dc5b74e6 [rank=0] || Thread 0x00007f34546e3240 (most recent call first): [2022-05-31T14:25:44.126837Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:25:44.126919Z] dc5b74e6 [rank=1] || Timeout (0:00:30)! [2022-05-31T14:25:44.126926Z] dc5b74e6 [rank=1] || Thread 0x00007f20ab5f5700 (most recent call first): [2022-05-31T14:25:44.126928Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:25:44.126930Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:25:44.127071Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:25:44.127111Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:25:44.127228Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:25:44.127230Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:25:44.127289Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:25:44.127378Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:25:44.127383Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:25:44.127385Z] dc5b74e6 [rank=1] || [2022-05-31T14:25:44.127387Z] dc5b74e6 [rank=1] || Thread 0x00007f2210724240 (most recent call first): [2022-05-31T14:25:44.127484Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 648 in iter [2022-05-31T14:25:44.127487Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:25:44.127622Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:25:44.127628Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 86 in iter [2022-05-31T14:25:44.127720Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:25:44.127723Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:25:44.127832Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:25:44.127905Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:25:44.127926Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:25:44.128074Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:25:44.128076Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:25:44.128211Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:25:44.128214Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:25:44.128339Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:25:44.128365Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:25:44.128417Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:25:44.128512Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:25:44.128517Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:25:44.128631Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:25:44.128746Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:25:44.128862Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:25:44.129002Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:25:44.129133Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:25:44.129265Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:25:44.129328Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:25:44.129400Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:26:14.129046Z] dc5b74e6 [rank=0] || Timeout (0:00:30)! [2022-05-31T14:26:14.129053Z] dc5b74e6 [rank=0] || Thread 0x00007f32e68e1700 (most recent call first): [2022-05-31T14:26:14.129055Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31T14:26:14.129057Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31T14:26:14.129059Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31T14:26:14.129061Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:26:14.129062Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:26:14.129063Z] dc5b74e6 [rank=0] || [2022-05-31T14:26:14.129064Z] dc5b74e6 [rank=0] || Thread 0x00007f32eedf4700 (most recent call first): [2022-05-31T14:26:14.129159Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31T14:26:14.129201Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31T14:26:14.129283Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31T14:26:14.129359Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31T14:26:14.129481Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31T14:26:14.129650Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31T14:26:14.129658Z] dc5b74e6 [rank=1] || Timeout (0:00:30)! [2022-05-31T14:26:14.129660Z] dc5b74e6 [rank=1] || Thread 0x00007f20ab5f5700 (most recent call first): [2022-05-31T14:26:14.129663Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:26:14.129756Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31T14:26:14.129870Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:26:14.129872Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31T14:26:14.129945Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:26:14.129947Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request [2022-05-31T14:26:14.130012Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:26:14.130105Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:26:14.130107Z] dc5b74e6 [rank=1] || [2022-05-31T14:26:14.130108Z] dc5b74e6 [rank=1] || Thread 0x00007f2210724240 (most recent call first): [2022-05-31T14:26:14.130147Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/requests.py", line 41 in request [2022-05-31T14:26:14.130255Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:26:14.130338Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/api/request.py", line 104 in do_request [2022-05-31T14:26:14.130485Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:26:14.130506Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 65 in _do_request [2022-05-31T14:26:14.130563Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:26:14.130688Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 86 in get [2022-05-31T14:26:14.130745Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:26:14.130867Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 57 in _get_preemption [2022-05-31T14:26:14.130917Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:26:14.131008Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 89 in run [2022-05-31T14:26:14.131055Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:26:14.131104Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:26:14.131178Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:26:14.131184Z] dc5b74e6 [rank=0] || [2022-05-31T14:26:14.131187Z] dc5b74e6 [rank=0] || Thread 0x00007f32ef5f5700 (most recent call first): [2022-05-31T14:26:14.131201Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:26:14.131242Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:26:14.131383Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:26:14.131392Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:26:14.131510Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:26:14.131531Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:26:14.131639Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:26:14.131642Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:26:14.131709Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:26:14.131712Z] dc5b74e6 [rank=0] || [2022-05-31T14:26:14.131713Z] dc5b74e6 [rank=0] || Thread 0x00007f34546e3240 (most recent call first): [2022-05-31T14:26:14.131824Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:26:14.131848Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:26:14.131982Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:26:14.131985Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:26:14.132123Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:26:14.132139Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:26:14.132196Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:26:14.132285Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:26:14.132330Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:26:14.132422Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:26:14.132534Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:26:14.132638Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:26:14.132750Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:26:14.132858Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:26:14.132972Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:26:14.133103Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:26:14.133221Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:26:14.133345Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:26:14.133411Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:26:14.133485Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:26:26.378800Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: http://10.6.5.8:8080 "GET /api/v1/allocations/34.29c616a1-c0bf-418c-a8bd-b855ef8ac23d.1/signals/preemption?timeout_seconds=60 HTTP/1.1" 200 17 [2022-05-31T14:26:26.381324Z] dc5b74e6 [rank=0] || DEBUG: [49] determined.core: (_PreemptionWatcher thread) _get_preemption(60) [2022-05-31T14:26:26.386681Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: Starting new HTTP connection (1): 10.6.5.8:8080 [2022-05-31T14:26:44.132713Z] dc5b74e6 [rank=1] || Timeout (0:00:30)! [2022-05-31T14:26:44.132726Z] dc5b74e6 [rank=1] || Thread 0x00007f20ab5f5700 (most recent call first): [2022-05-31T14:26:44.132734Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:26:44.132742Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:26:44.132900Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:26:44.132917Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:26:44.132924Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:26:44.132932Z] dc5b74e6 [rank=1] || [2022-05-31T14:26:44.132938Z] dc5b74e6 [rank=1] || Thread 0x00007f2210724240 (most recent call first): [2022-05-31T14:26:44.133031Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:26:44.133138Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:26:44.133307Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:26:44.133400Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:26:44.133529Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:26:44.133648Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:26:44.133812Z] dc5b74e6 [rank=0] || Timeout (0:00:30)! [2022-05-31T14:26:44.133814Z] dc5b74e6 [rank=0] || Thread 0x00007f32e68e1700 (most recent call first): [2022-05-31T14:26:44.133816Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31T14:26:44.133818Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31T14:26:44.133819Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:26:44.133909Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31T14:26:44.134045Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:26:44.134047Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:26:44.134074Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:26:44.134075Z] dc5b74e6 [rank=0] || [2022-05-31T14:26:44.134077Z] dc5b74e6 [rank=0] || Thread 0x00007f32eedf4700 (most recent call first): [2022-05-31T14:26:44.134081Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:26:44.134185Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31T14:26:44.134250Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:26:44.134257Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31T14:26:44.134379Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31T14:26:44.134492Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31T14:26:44.134505Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:26:44.134576Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31T14:26:44.134699Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:26:44.134702Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31T14:26:44.134783Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31T14:26:44.134785Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:26:44.134879Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:26:44.134882Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31T14:26:44.134950Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:26:44.134999Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request [2022-05-31T14:26:44.135138Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/requests.py", line 41 in request [2022-05-31T14:26:44.135263Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/api/request.py", line 104 in do_request [2022-05-31T14:26:44.135411Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 65 in _do_request [2022-05-31T14:26:44.135539Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 86 in get [2022-05-31T14:26:44.135670Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 57 in _get_preemption [2022-05-31T14:26:44.135781Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 89 in run [2022-05-31T14:26:44.135848Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:26:44.135918Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:26:44.135921Z] dc5b74e6 [rank=0] || [2022-05-31T14:26:44.135922Z] dc5b74e6 [rank=0] || Thread 0x00007f32ef5f5700 (most recent call first): [2022-05-31T14:26:44.135990Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:26:44.136128Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:26:44.136243Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:26:44.136303Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:26:44.136372Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:26:44.136374Z] dc5b74e6 [rank=0] || [2022-05-31T14:26:44.136375Z] dc5b74e6 [rank=0] || Thread 0x00007f34546e3240 (most recent call first): [2022-05-31T14:26:44.136493Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:26:44.136586Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:26:44.136751Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:26:44.136880Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:26:44.137014Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:26:44.137158Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:26:44.137306Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:26:44.137406Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:26:44.137519Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:26:44.137629Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:26:44.137762Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:26:44.137880Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:26:44.138001Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:26:44.138071Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:26:44.138144Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:27:14.135432Z] dc5b74e6 [rank=1] || Timeout (0:00:30)! [2022-05-31T14:27:14.135439Z] dc5b74e6 [rank=1] || Thread 0x00007f20ab5f5700 (most recent call first): [2022-05-31T14:27:14.135441Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:27:14.135449Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:27:14.135454Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:27:14.135527Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:27:14.135581Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:27:14.135583Z] dc5b74e6 [rank=1] ||

[kelvinwang139]

more logs because comments size limitations： [2022-05-31T14:27:14.135584Z] dc5b74e6 [rank=1] || Thread 0x00007f2210724240 (most recent call first): [2022-05-31T14:27:14.135715Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:27:14.135792Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:27:14.135954Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:27:14.136093Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:27:14.136206Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:27:14.136338Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:27:14.136448Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:27:14.136569Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:27:14.136684Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:27:14.136797Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:27:14.136937Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:27:14.137069Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:27:14.137197Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:27:14.137260Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:27:14.137339Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:27:14.138498Z] dc5b74e6 [rank=0] || Timeout (0:00:30)! [2022-05-31T14:27:14.138503Z] dc5b74e6 [rank=0] || Thread 0x00007f32e68e1700 (most recent call first): [2022-05-31T14:27:14.138509Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31T14:27:14.138514Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31T14:27:14.138522Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31T14:27:14.138600Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:27:14.138741Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:27:14.138743Z] dc5b74e6 [rank=0] || [2022-05-31T14:27:14.138744Z] dc5b74e6 [rank=0] || Thread 0x00007f32eedf4700 (most recent call first): [2022-05-31T14:27:14.138746Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31T14:27:14.138846Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31T14:27:14.138899Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31T14:27:14.138978Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31T14:27:14.139096Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31T14:27:14.139197Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31T14:27:14.139295Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31T14:27:14.139388Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31T14:27:14.139490Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request [2022-05-31T14:27:14.139635Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/requests.py", line 41 in request [2022-05-31T14:27:14.139794Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/api/request.py", line 104 in do_request [2022-05-31T14:27:14.139953Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 65 in _do_request [2022-05-31T14:27:14.140088Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 86 in get [2022-05-31T14:27:14.140232Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 57 in _get_preemption [2022-05-31T14:27:14.140356Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 89 in run [2022-05-31T14:27:14.140417Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:27:14.140486Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:27:14.140490Z] dc5b74e6 [rank=0] || [2022-05-31T14:27:14.140492Z] dc5b74e6 [rank=0] || Thread 0x00007f32ef5f5700 (most recent call first): [2022-05-31T14:27:14.140557Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:27:14.140687Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:27:14.140806Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:27:14.140914Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:27:14.140970Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:27:14.140973Z] dc5b74e6 [rank=0] || [2022-05-31T14:27:14.140975Z] dc5b74e6 [rank=0] || Thread 0x00007f34546e3240 (most recent call first): [2022-05-31T14:27:14.141062Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 630 in len [2022-05-31T14:27:14.141191Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:27:14.141281Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:27:14.141418Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:27:14.141550Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:27:14.141663Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:27:14.141783Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:27:14.141894Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:27:14.142008Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:27:14.142115Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:27:14.142224Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:27:14.142380Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:27:14.142493Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:27:14.142615Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:27:14.142680Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:27:14.142755Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:27:26.391524Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: http://10.6.5.8:8080 "GET /api/v1/allocations/34.29c616a1-c0bf-418c-a8bd-b855ef8ac23d.1/signals/preemption?timeout_seconds=60 HTTP/1.1" 200 17 [2022-05-31T14:27:26.393774Z] dc5b74e6 [rank=0] || DEBUG: [49] determined.core: (_PreemptionWatcher thread) _get_preemption(60) [2022-05-31T14:27:26.399156Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: Starting new HTTP connection (1): 10.6.5.8:8080 [2022-05-31T14:27:44.137787Z] dc5b74e6 [rank=1] || Timeout (0:00:30)! [2022-05-31T14:27:44.137794Z] dc5b74e6 [rank=1] || Thread 0x00007f20ab5f5700 (most recent call first): [2022-05-31T14:27:44.137798Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:27:44.137801Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:27:44.137914Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:27:44.137915Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:27:44.137986Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:27:44.137992Z] dc5b74e6 [rank=1] || [2022-05-31T14:27:44.137993Z] dc5b74e6 [rank=1] || Thread 0x00007f2210724240 (most recent call first): [2022-05-31T14:27:44.138074Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:27:44.138177Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:27:44.138349Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:27:44.138515Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:27:44.138607Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:27:44.138744Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:27:44.138853Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:27:44.138962Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:27:44.139094Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:27:44.139196Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:27:44.139341Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:27:44.139459Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:27:44.139593Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:27:44.139642Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:27:44.139726Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:27:44.143099Z] dc5b74e6 [rank=0] || Timeout (0:00:30)! [2022-05-31T14:27:44.143102Z] dc5b74e6 [rank=0] || Thread 0x00007f32e68e1700 (most recent call first): [2022-05-31T14:27:44.143105Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31T14:27:44.143108Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31T14:27:44.143110Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31T14:27:44.143241Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:27:44.143307Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:27:44.143310Z] dc5b74e6 [rank=0] || [2022-05-31T14:27:44.143311Z] dc5b74e6 [rank=0] || Thread 0x00007f32eedf4700 (most recent call first): [2022-05-31T14:27:44.143418Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31T14:27:44.143511Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31T14:27:44.143563Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31T14:27:44.143645Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31T14:27:44.143784Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31T14:27:44.143887Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31T14:27:44.143993Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31T14:27:44.144083Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31T14:27:44.144185Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request [2022-05-31T14:27:44.144337Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/requests.py", line 41 in request [2022-05-31T14:27:44.144476Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/api/request.py", line 104 in do_request [2022-05-31T14:27:44.144637Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 65 in _do_request [2022-05-31T14:27:44.144776Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 86 in get [2022-05-31T14:27:44.144928Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 57 in _get_preemption [2022-05-31T14:27:44.145043Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 89 in run [2022-05-31T14:27:44.145098Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:27:44.145171Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:27:44.145176Z] dc5b74e6 [rank=0] || [2022-05-31T14:27:44.145180Z] dc5b74e6 [rank=0] || Thread 0x00007f32ef5f5700 (most recent call first): [2022-05-31T14:27:44.145239Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:27:44.145408Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:27:44.145522Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:27:44.145588Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:27:44.145650Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:27:44.145661Z] dc5b74e6 [rank=0] || [2022-05-31T14:27:44.145665Z] dc5b74e6 [rank=0] || Thread 0x00007f34546e3240 (most recent call first): [2022-05-31T14:27:44.145784Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:27:44.145877Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:27:44.146016Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:27:44.146164Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:27:44.146254Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:27:44.146394Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:27:44.146498Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:27:44.146600Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:27:44.146706Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:27:44.146818Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:27:44.146946Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:27:44.147081Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:27:44.147196Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:27:44.147259Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:27:44.147333Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:28:14.140210Z] dc5b74e6 [rank=1] || Timeout (0:00:30)! [2022-05-31T14:28:14.140221Z] dc5b74e6 [rank=1] || Thread 0x00007f20ab5f5700 (most recent call first): [2022-05-31T14:28:14.140224Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:28:14.140227Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:28:14.140229Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:28:14.140315Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:28:14.140377Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:28:14.140382Z] dc5b74e6 [rank=1] || [2022-05-31T14:28:14.140385Z] dc5b74e6 [rank=1] || Thread 0x00007f2210724240 (most recent call first): [2022-05-31T14:28:14.140481Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 86 in iter [2022-05-31T14:28:14.140605Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:28:14.140746Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:28:14.140883Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:28:14.140997Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:28:14.141132Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:28:14.141247Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:28:14.141363Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:28:14.141485Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:28:14.141601Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:28:14.141745Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:28:14.141872Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:28:14.142004Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:28:14.142066Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:28:14.142160Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:28:14.147711Z] dc5b74e6 [rank=0] || Timeout (0:00:30)! [2022-05-31T14:28:14.147715Z] dc5b74e6 [rank=0] || Thread 0x00007f32e68e1700 (most recent call first): [2022-05-31T14:28:14.147720Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31T14:28:14.147725Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31T14:28:14.147729Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31T14:28:14.147798Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:28:14.147834Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:28:14.147837Z] dc5b74e6 [rank=0] || [2022-05-31T14:28:14.147839Z] dc5b74e6 [rank=0] || Thread 0x00007f32eedf4700 (most recent call first): [2022-05-31T14:28:14.147902Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31T14:28:14.148006Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31T14:28:14.148053Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31T14:28:14.148141Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31T14:28:14.148269Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31T14:28:14.148366Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31T14:28:14.148557Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31T14:28:14.148560Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31T14:28:14.148654Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request [2022-05-31T14:28:14.148803Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/requests.py", line 41 in request [2022-05-31T14:28:14.148933Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/api/request.py", line 104 in do_request [2022-05-31T14:28:14.149085Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 65 in _do_request [2022-05-31T14:28:14.149228Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 86 in get [2022-05-31T14:28:14.149368Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 57 in _get_preemption [2022-05-31T14:28:14.149493Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 89 in run [2022-05-31T14:28:14.149559Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:28:14.149631Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:28:14.149633Z] dc5b74e6 [rank=0] || [2022-05-31T14:28:14.149635Z] dc5b74e6 [rank=0] || Thread 0x00007f32ef5f5700 (most recent call first): [2022-05-31T14:28:14.149695Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:28:14.149834Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:28:14.149954Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:28:14.150016Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:28:14.150088Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:28:14.150090Z] dc5b74e6 [rank=0] || [2022-05-31T14:28:14.150091Z] dc5b74e6 [rank=0] || Thread 0x00007f34546e3240 (most recent call first): [2022-05-31T14:28:14.150203Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 63 in iter [2022-05-31T14:28:14.150306Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:28:14.150468Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:28:14.150590Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:28:14.150698Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:28:14.150813Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:28:14.150926Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:28:14.151026Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:28:14.151139Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:28:14.151250Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:28:14.151378Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:28:14.151497Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:28:14.151615Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:28:14.151676Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:28:14.151753Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:28:26.404643Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: http://10.6.5.8:8080 "GET /api/v1/allocations/34.29c616a1-c0bf-418c-a8bd-b855ef8ac23d.1/signals/preemption?timeout_seconds=60 HTTP/1.1" 200 17 [2022-05-31T14:28:26.407103Z] dc5b74e6 [rank=0] || DEBUG: [49] determined.core: (_PreemptionWatcher thread) _get_preemption(60) [2022-05-31T14:28:26.412551Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: Starting new HTTP connection (1): 10.6.5.8:8080 [2022-05-31T14:28:44.142639Z] dc5b74e6 [rank=1] || Timeout (0:00:30)! [2022-05-31T14:28:44.142649Z] dc5b74e6 [rank=1] || Thread 0x00007f20ab5f5700 (most recent call first): [2022-05-31T14:28:44.142654Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:28:44.142663Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:28:44.142727Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:28:44.142807Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:28:44.142836Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:28:44.142843Z] dc5b74e6 [rank=1] || [2022-05-31T14:28:44.142856Z] dc5b74e6 [rank=1] || Thread 0x00007f2210724240 (most recent call first): [2022-05-31T14:28:44.142952Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 86 in iter [2022-05-31T14:28:44.143065Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:28:44.152153Z] dc5b74e6 [rank=0] || Timeout (0:00:30)! [2022-05-31T14:28:44.152158Z] dc5b74e6 [rank=0] || Thread 0x00007f32e68e1700 (most recent call first): [2022-05-31T14:28:44.152159Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31T14:28:44.152161Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31T14:28:44.152162Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31T14:28:44.152246Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:28:44.152247Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:28:44.152248Z] dc5b74e6 [rank=0] || [2022-05-31T14:28:44.152253Z] dc5b74e6 [rank=0] || Thread 0x00007f32eedf4700 (most recent call first): [2022-05-31T14:28:44.152373Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31T14:28:44.152436Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31T14:28:44.152503Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31T14:28:44.152593Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31T14:28:44.152702Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31T14:28:44.152812Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31T14:28:44.152907Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31T14:28:44.153013Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31T14:28:44.153104Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request [2022-05-31T14:28:44.153255Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/requests.py", line 41 in request [2022-05-31T14:28:44.153384Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/api/request.py", line 104 in do_request [2022-05-31T14:28:44.153535Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 65 in _do_request [2022-05-31T14:28:44.153666Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 86 in get [2022-05-31T14:28:44.153817Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 57 in _get_preemption [2022-05-31T14:28:44.153944Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 89 in run [2022-05-31T14:28:44.154018Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:28:44.154082Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:28:44.154085Z] dc5b74e6 [rank=0] || [2022-05-31T14:28:44.154086Z] dc5b74e6 [rank=0] || Thread 0x00007f32ef5f5700 (most recent call first): [2022-05-31T14:28:44.154151Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:28:44.154280Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:28:44.154411Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:28:44.154479Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:28:44.154553Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:28:44.154559Z] dc5b74e6 [rank=0] || [2022-05-31T14:28:44.154562Z] dc5b74e6 [rank=0] || Thread 0x00007f34546e3240 (most recent call first): [2022-05-31T14:28:44.154662Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:28:44.154773Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:28:44.154931Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:28:44.155066Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:28:44.155179Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:28:44.155297Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:28:44.155409Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:28:44.155517Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:28:44.155623Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:28:44.155735Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:28:44.155870Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:28:44.156006Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:28:44.156126Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:28:44.156190Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:28:44.156263Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:29:14.143564Z] dc5b74e6 [rank=1] || Timeout (0:00:30)! [2022-05-31T14:29:14.143573Z] dc5b74e6 [rank=1] || Thread 0x00007f20ab5f5700 (most recent call first): [2022-05-31T14:29:14.143576Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:29:14.143578Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:29:14.143579Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:29:14.143679Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:29:14.143689Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:29:14.143694Z] dc5b74e6 [rank=1] || [2022-05-31T14:29:14.143699Z] dc5b74e6 [rank=1] || Thread 0x00007f2210724240 (most recent call first): [2022-05-31T14:29:14.143790Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:29:14.143893Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:29:14.144041Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:29:14.144176Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:29:14.144279Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:29:14.144412Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:29:14.144515Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:29:14.144629Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:29:14.144752Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:29:14.144866Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:29:14.145012Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:29:14.145147Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:29:14.145272Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:29:14.145342Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:29:14.145409Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:29:14.156683Z] dc5b74e6 [rank=0] || Timeout (0:00:30)! [2022-05-31T14:29:14.156687Z] dc5b74e6 [rank=0] || Thread 0x00007f32e68e1700 (most recent call first): [2022-05-31T14:29:14.156688Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31T14:29:14.156690Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31T14:29:14.156691Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31T14:29:14.156781Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:29:14.156786Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:29:14.156791Z] dc5b74e6 [rank=0] || [2022-05-31T14:29:14.156795Z] dc5b74e6 [rank=0] || Thread 0x00007f32eedf4700 (most recent call first): [2022-05-31T14:29:14.156883Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31T14:29:14.156935Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31T14:29:14.157026Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31T14:29:14.157087Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31T14:29:14.157215Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31T14:29:14.157306Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31T14:29:14.157406Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31T14:29:14.157495Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31T14:29:14.157590Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request [2022-05-31T14:29:14.157737Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/requests.py", line 41 in request [2022-05-31T14:29:14.157886Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/api/request.py", line 104 in do_request [2022-05-31T14:29:14.158042Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 65 in _do_request [2022-05-31T14:29:14.158181Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 86 in get [2022-05-31T14:29:14.158340Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 57 in _get_preemption [2022-05-31T14:29:14.158468Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 89 in run [2022-05-31T14:29:14.158520Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:29:14.158590Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:29:14.158593Z] dc5b74e6 [rank=0] || [2022-05-31T14:29:14.158594Z] dc5b74e6 [rank=0] || Thread 0x00007f32ef5f5700 (most recent call first): [2022-05-31T14:29:14.158650Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:29:14.158806Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:29:14.158910Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:29:14.158975Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:29:14.159045Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:29:14.159047Z] dc5b74e6 [rank=0] || [2022-05-31T14:29:14.159049Z] dc5b74e6 [rank=0] || Thread 0x00007f34546e3240 (most recent call first): [2022-05-31T14:29:14.159151Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:29:14.159258Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:29:14.159399Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:29:14.159527Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:29:14.159637Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:29:14.159754Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:29:14.159865Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:29:14.159966Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:29:14.160074Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:29:14.160186Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:29:14.160323Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:29:14.160442Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:29:14.160558Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:29:14.160621Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:29:14.160698Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:29:26.417270Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: http://10.6.5.8:8080 "GET /api/v1/allocations/34.29c616a1-c0bf-418c-a8bd-b855ef8ac23d.1/signals/preemption?timeout_seconds=60 HTTP/1.1" 200 17 [2022-05-31T14:29:26.419357Z] dc5b74e6 [rank=0] || DEBUG: [49] determined.core: (_PreemptionWatcher thread) _get_preemption(60) [2022-05-31T14:29:26.424531Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: Starting new HTTP connection (1): 10.6.5.8:8080 [2022-05-31T14:29:44.145900Z] dc5b74e6 [rank=1] || Timeout (0:00:30)! [2022-05-31T14:29:44.145909Z] dc5b74e6 [rank=1] || Thread 0x00007f20ab5f5700 (most recent call first): [2022-05-31T14:29:44.145911Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:29:44.145916Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:29:44.145919Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:29:44.145991Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:29:44.146063Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:29:44.146068Z] dc5b74e6 [rank=1] || [2022-05-31T14:29:44.146071Z] dc5b74e6 [rank=1] || Thread 0x00007f2210724240 (most recent call first): [2022-05-31T14:29:44.146145Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 86 in iter [2022-05-31T14:29:44.146256Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:29:44.146457Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:29:44.146590Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:29:44.146700Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:29:44.146842Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:29:44.146954Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:29:44.147072Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:29:44.147193Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:29:44.147304Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:29:44.147450Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:29:44.147583Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:29:44.147704Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:29:44.147776Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:29:44.147841Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:29:44.161198Z] dc5b74e6 [rank=0] || Timeout (0:00:30)! [2022-05-31T14:29:44.161204Z] dc5b74e6 [rank=0] || Thread 0x00007f32e68e1700 (most recent call first): [2022-05-31T14:29:44.161206Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31T14:29:44.161208Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31T14:29:44.161210Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31T14:29:44.161213Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:29:44.161219Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:29:44.161224Z] dc5b74e6 [rank=0] || [2022-05-31T14:29:44.161229Z] dc5b74e6 [rank=0] || Thread 0x00007f32eedf4700 (most recent call first): [2022-05-31T14:29:44.161339Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31T14:29:44.161344Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31T14:29:44.161426Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31T14:29:44.161499Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31T14:29:44.161614Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31T14:29:44.161716Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31T14:29:44.161837Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31T14:29:44.161928Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31T14:29:44.162019Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request [2022-05-31T14:29:44.162169Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/requests.py", line 41 in request [2022-05-31T14:29:44.162307Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/api/request.py", line 104 in do_request [2022-05-31T14:29:44.162472Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 65 in _do_request [2022-05-31T14:29:44.162599Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 86 in get [2022-05-31T14:29:44.162743Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 57 in _get_preemption [2022-05-31T14:29:44.162868Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 89 in run [2022-05-31T14:29:44.162929Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:29:44.163000Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:29:44.163003Z] dc5b74e6 [rank=0] || [2022-05-31T14:29:44.163005Z] dc5b74e6 [rank=0] || Thread 0x00007f32ef5f5700 (most recent call first): [2022-05-31T14:29:44.163072Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:29:44.163203Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:29:44.163324Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:29:44.163388Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:29:44.163460Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:29:44.163464Z] dc5b74e6 [rank=0] || [2022-05-31T14:29:44.163465Z] dc5b74e6 [rank=0] || Thread 0x00007f34546e3240 (most recent call first): [2022-05-31T14:29:44.163595Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 63 in iter [2022-05-31T14:29:44.163661Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:29:44.163814Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:29:44.163945Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:29:44.164058Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:29:44.164170Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:29:44.164279Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:29:44.164383Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:29:44.164489Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:29:44.164599Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:29:44.164729Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:29:44.164843Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:29:44.164962Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:29:44.165026Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:29:44.165101Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:30:14.148294Z] dc5b74e6 [rank=1] || Timeout (0:00:30)! [2022-05-31T14:30:14.148302Z] dc5b74e6 [rank=1] || Thread 0x00007f20ab5f5700 (most recent call first): [2022-05-31T14:30:14.148306Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:30:14.148309Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:30:14.148311Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:30:14.148434Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:30:14.148440Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:30:14.148444Z] dc5b74e6 [rank=1] || [2022-05-31T14:30:14.148448Z] dc5b74e6 [rank=1] || Thread 0x00007f2210724240 (most recent call first): [2022-05-31T14:30:14.148554Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 630 in len [2022-05-31T14:30:14.148662Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:30:14.148769Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:30:14.148949Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:30:14.149100Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:30:14.149201Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:30:14.149333Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:30:14.149437Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:30:14.149561Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:30:14.149677Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:30:14.149797Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:30:14.149938Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:30:14.150079Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:30:14.150184Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:30:14.150266Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:30:14.150348Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:30:14.165541Z] dc5b74e6 [rank=0] || Timeout (0:00:30)! [2022-05-31T14:30:14.165546Z] dc5b74e6 [rank=0] || Thread 0x00007f32e68e1700 (most recent call first): [2022-05-31T14:30:14.165549Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31T14:30:14.165550Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31T14:30:14.165552Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31T14:30:14.165554Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:30:14.165642Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:30:14.165646Z] dc5b74e6 [rank=0] || [2022-05-31T14:30:14.165648Z] dc5b74e6 [rank=0] || Thread 0x00007f32eedf4700 (most recent call first): [2022-05-31T14:30:14.165652Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31T14:30:14.165737Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31T14:30:14.165823Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31T14:30:14.165871Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31T14:30:14.165991Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31T14:30:14.166096Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31T14:30:14.166193Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31T14:30:14.166280Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31T14:30:14.166421Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request [2022-05-31T14:30:14.166548Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/requests.py", line 41 in request [2022-05-31T14:30:14.166694Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/api/request.py", line 104 in do_request [2022-05-31T14:30:14.166846Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 65 in _do_request [2022-05-31T14:30:14.166977Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 86 in get [2022-05-31T14:30:14.167124Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 57 in _get_preemption [2022-05-31T14:30:14.167244Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 89 in run [2022-05-31T14:30:14.167310Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:30:14.167377Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:30:14.167381Z] dc5b74e6 [rank=0] || [2022-05-31T14:30:14.167382Z] dc5b74e6 [rank=0] || Thread 0x00007f32ef5f5700 (most recent call first): [2022-05-31T14:30:14.167474Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:30:14.167593Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:30:14.167719Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:30:14.167778Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:30:14.167850Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:30:14.167853Z] dc5b74e6 [rank=0] || [2022-05-31T14:30:14.167855Z] dc5b74e6 [rank=0] || Thread 0x00007f34546e3240 (most recent call first): [2022-05-31T14:30:14.167964Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:30:14.168060Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:30:14.168210Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:30:14.168333Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:30:14.168442Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:30:14.168560Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:30:14.168671Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:30:14.168776Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:30:14.168886Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:30:14.169000Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:30:14.169127Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:30:14.169248Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:30:14.169362Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:30:14.169427Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:30:14.169509Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:30:26.429389Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: http://10.6.5.8:8080 "GET /api/v1/allocations/34.29c616a1-c0bf-418c-a8bd-b855ef8ac23d.1/signals/preemption?timeout_seconds=60 HTTP/1.1" 200 17 [2022-05-31T14:30:26.431497Z] dc5b74e6 [rank=0] || DEBUG: [49] determined.core: (_PreemptionWatcher thread) _get_preemption(60) [2022-05-31T14:30:26.436711Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: Starting new HTTP connection (1): 10.6.5.8:8080 [2022-05-31T14:30:44.150814Z] dc5b74e6 [rank=1] || Timeout (0:00:30)! [2022-05-31T14:30:44.150825Z] dc5b74e6 [rank=1] || Thread 0x00007f20ab5f5700 (most recent call first): [2022-05-31T14:30:44.150830Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:30:44.150838Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:30:44.150846Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:30:44.150928Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:30:44.150943Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:30:44.150950Z] dc5b74e6 [rank=1] || [2022-05-31T14:30:44.150957Z] dc5b74e6 [rank=1] || Thread 0x00007f2210724240 (most recent call first): [2022-05-31T14:30:44.151068Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 630 in len [2022-05-31T14:30:44.151182Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 63 in iter [2022-05-31T14:30:44.151288Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:30:44.151444Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:30:44.151584Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:30:44.151699Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:30:44.151837Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:30:44.151942Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:30:44.152059Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:30:44.152168Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:30:44.152291Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:30:44.152433Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:30:44.152572Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:30:44.152686Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:30:44.152762Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:30:44.152815Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:30:44.169901Z] dc5b74e6 [rank=0] || Timeout (0:00:30)! [2022-05-31T14:30:44.169909Z] dc5b74e6 [rank=0] || Thread 0x00007f32e68e1700 (most recent call first): [2022-05-31T14:30:44.169914Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31T14:30:44.169920Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31T14:30:44.169926Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31T14:30:44.169997Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:30:44.170128Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:30:44.170131Z] dc5b74e6 [rank=0] || [2022-05-31T14:30:44.170135Z] dc5b74e6 [rank=0] || Thread 0x00007f32eedf4700 (most recent call first): [2022-05-31T14:30:44.170180Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31T14:30:44.170269Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31T14:30:44.170361Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31T14:30:44.170475Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31T14:30:44.170548Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31T14:30:44.170661Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31T14:30:44.170781Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31T14:30:44.170852Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31T14:30:44.170936Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request [2022-05-31T14:30:44.171066Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/requests.py", line 41 in request [2022-05-31T14:30:44.171197Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/api/request.py", line 104 in do_request [2022-05-31T14:30:44.171348Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 65 in _do_request [2022-05-31T14:30:44.171481Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 86 in get [2022-05-31T14:30:44.171639Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 57 in _get_preemption [2022-05-31T14:30:44.171749Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 89 in run [2022-05-31T14:30:44.171803Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:30:44.171888Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:30:44.171892Z] dc5b74e6 [rank=0] || [2022-05-31T14:30:44.171895Z] dc5b74e6 [rank=0] || Thread 0x00007f32ef5f5700 (most recent call first): [2022-05-31T14:30:44.171933Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:30:44.172093Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:30:44.172187Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:30:44.172246Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:30:44.172308Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:30:44.172311Z] dc5b74e6 [rank=0] || [2022-05-31T14:30:44.172312Z] dc5b74e6 [rank=0] || Thread 0x00007f34546e3240 (most recent call first): [2022-05-31T14:30:44.172413Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:30:44.172518Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:30:44.172655Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:30:44.172777Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:30:44.172889Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:30:44.173007Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:30:44.173111Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:30:44.173216Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:30:44.173320Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:30:44.173428Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:30:44.173562Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:30:44.173685Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:30:44.173798Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:30:44.173860Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:30:44.173960Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:31:14.153335Z] dc5b74e6 [rank=1] || Timeout (0:00:30)! [2022-05-31T14:31:14.153343Z] dc5b74e6 [rank=1] || Thread 0x00007f20ab5f5700 (most recent call first): [2022-05-31T14:31:14.153345Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:31:14.153348Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:31:14.153349Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:31:14.153408Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:31:14.153460Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:31:14.153462Z] dc5b74e6 [rank=1] || [2022-05-31T14:31:14.153463Z] dc5b74e6 [rank=1] || Thread 0x00007f2210724240 (most recent call first): [2022-05-31T14:31:14.153603Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/_tensor.py", line 648 in iter [2022-05-31T14:31:14.153677Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:31:14.153793Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:31:14.153945Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:31:14.154087Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:31:14.154205Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:31:14.154351Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:31:14.154501Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:31:14.154599Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:31:14.154718Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:31:14.154832Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:31:14.154974Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:31:14.155095Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:31:14.155230Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:31:14.155292Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:31:14.155382Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:31:14.174360Z] dc5b74e6 [rank=0] || Timeout (0:00:30)! [2022-05-31T14:31:14.174366Z] dc5b74e6 [rank=0] || Thread 0x00007f32e68e1700 (most recent call first): [2022-05-31T14:31:14.174369Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 306 in wait [2022-05-31T14:31:14.174372Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 558 in wait [2022-05-31T14:31:14.174374Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tqdm/_monitor.py", line 60 in run [2022-05-31T14:31:14.174481Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:31:14.174486Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:31:14.174488Z] dc5b74e6 [rank=0] || [2022-05-31T14:31:14.174490Z] dc5b74e6 [rank=0] || Thread 0x00007f32eedf4700 (most recent call first): [2022-05-31T14:31:14.174549Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/socket.py", line 669 in readinto [2022-05-31T14:31:14.174631Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 277 in _read_status [2022-05-31T14:31:14.174681Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 316 in begin [2022-05-31T14:31:14.174766Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/http/client.py", line 1348 in getresponse [2022-05-31T14:31:14.174868Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444 in _make_request [2022-05-31T14:31:14.174963Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703 in urlopen [2022-05-31T14:31:14.175061Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/adapters.py", line 440 in send [2022-05-31T14:31:14.175141Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 645 in send [2022-05-31T14:31:14.175246Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/requests/sessions.py", line 529 in request [2022-05-31T14:31:14.175367Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/requests.py", line 41 in request [2022-05-31T14:31:14.175491Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/api/request.py", line 104 in do_request [2022-05-31T14:31:14.175644Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 65 in _do_request [2022-05-31T14:31:14.175778Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/experimental/session.py", line 86 in get [2022-05-31T14:31:14.175920Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 57 in _get_preemption [2022-05-31T14:31:14.176042Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_preempt.py", line 89 in run [2022-05-31T14:31:14.176111Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:31:14.176173Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:31:14.176175Z] dc5b74e6 [rank=0] || [2022-05-31T14:31:14.176177Z] dc5b74e6 [rank=0] || Thread 0x00007f32ef5f5700 (most recent call first): [2022-05-31T14:31:14.176245Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:31:14.176361Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:31:14.176483Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:31:14.176547Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:31:14.176617Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:31:14.176620Z] dc5b74e6 [rank=0] || [2022-05-31T14:31:14.176621Z] dc5b74e6 [rank=0] || Thread 0x00007f34546e3240 (most recent call first): [2022-05-31T14:31:14.176719Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:31:14.176826Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 228 in iter [2022-05-31T14:31:14.176955Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:31:14.177081Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:31:14.177181Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:31:14.177292Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:31:14.177392Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:31:14.177490Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:31:14.177586Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:31:14.177687Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:31:14.177819Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:31:14.177923Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:31:14.178038Z] dc5b74e6 [rank=0] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:31:14.178101Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:31:14.178169Z] dc5b74e6 [rank=0] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:31:26.443101Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: http://10.6.5.8:8080 "GET /api/v1/allocations/34.29c616a1-c0bf-418c-a8bd-b855ef8ac23d.1/signals/preemption?timeout_seconds=60 HTTP/1.1" 200 17 [2022-05-31T14:31:26.445234Z] dc5b74e6 [rank=0] || DEBUG: [49] determined.core: (_PreemptionWatcher thread) _get_preemption(60) [2022-05-31T14:31:26.450530Z] dc5b74e6 [rank=0] || DEBUG: [49] urllib3.connectionpool: Starting new HTTP connection (1): 10.6.5.8:8080 [2022-05-31T14:31:44.155851Z] dc5b74e6 [rank=1] || Timeout (0:00:30)! [2022-05-31T14:31:44.155860Z] dc5b74e6 [rank=1] || Thread 0x00007f20ab5f5700 (most recent call first): [2022-05-31T14:31:44.155863Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 302 in wait [2022-05-31T14:31:44.155865Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 261 in get [2022-05-31T14:31:44.155868Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/summary/writer/event_file_writer.py", line 204 in run [2022-05-31T14:31:44.155955Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 932 in _bootstrap_inner [2022-05-31T14:31:44.155995Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/threading.py", line 890 in _bootstrap [2022-05-31T14:31:44.155997Z] dc5b74e6 [rank=1] || [2022-05-31T14:31:44.155998Z] dc5b74e6 [rank=1] || Thread 0x00007f2210724240 (most recent call first): [2022-05-31T14:31:44.156126Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/model_hub/mmdetection/_data.py", line 82 in iter [2022-05-31T14:31:44.156219Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/sampler.py", line 229 in iter [2022-05-31T14:31:44.156361Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 42 in iter [2022-05-31T14:31:44.156502Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/samplers.py", line 116 in iter [2022-05-31T14:31:44.156618Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 512 in _next_index [2022-05-31T14:31:44.156748Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1209 in _try_put_index [2022-05-31T14:31:44.156858Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 975 in _reset [2022-05-31T14:31:44.156981Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 944 in init [2022-05-31T14:31:44.157093Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 305 in _get_iterator [2022-05-31T14:31:44.157209Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 359 in iter [2022-05-31T14:31:44.157359Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py", line 224 in run [2022-05-31T14:31:44.157499Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 121 in main [2022-05-31T14:31:44.157614Z] dc5b74e6 [rank=1] || File "/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py", line 130 in [2022-05-31T14:31:44.157684Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 87 in _run_code [2022-05-31T14:31:44.157772Z] dc5b74e6 [rank=1] || File "/opt/conda/lib/python3.8/runpy.py", line 194 in _run_module_as_main [2022-05-31T14:31:44.178576Z] dc5b74e6 [rank=0] || Timeout (0:00:30)!