search

fangwei123456 / Spike-Element-Wise-ResNet

Deep Residual Learning in Spiking Neural Networks
Mozilla Public License 2.0
140 stars 21 forks source link

The training time on Imagenet. #10

Open iminfine opened 2 years ago

iminfine commented 2 years ago

Thanks for your great work!

Could you please tell me how long does it take to train a model on Imagenet using 8 GPUs?

fangwei123456 commented 2 years ago

It takes about 10 minutes for one epoch with resnet18. You can also download tensorboard logs in this repo and check the used time with tensorboard.

fangwei123456 commented 2 years ago

https://github.com/fangwei123456/Spike-Element-Wise-ResNet/tree/main/origin_logs/imagenet

iminfine commented 2 years ago

Thanks for your reply. That's really fast, however it takes about 4 hours for me to train one epoch using a 8 Tesla V100 cluster node. My pytorch vision is 1.11.0 and spikingjelly is 0.0.0.0.11.

The python command is python -m torch.distributed.launch --nproc_per_node=8 --use_env train.py --cos_lr_T 320 --model sew_resnet18 -b 32 --output-dir ./logs --tb --print-freq 1 --amp --cache-dataset --connect_f ADD --T 4 --lr 0.1 --epoch 320 --data-path /jmain02/home/J2AD019/exk01/bxg73-exk01/traindata/imagenet

The output is:

Python3 anaconda is now loaded in your environment.

CUDA-10.2 loaded

GCC 9.1.0 environment now loaded

/jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/distributed/launch.py:186: FutureWarning: The module torch.distributed.launch is deprecated and will be removed in future. Use torchrun. Note that --use_env is set by default in torchrun. If your script expects --local_rank argument to be set, please change it to read from os.environ['LOCAL_RANK'] instead. See https://pytorch.org/docs/stable/distributed.html#launch-utility for further instructions

FutureWarning, WARNING:torch.distributed.run:

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

spikingjelly.clock_driven.spike_op: try to use torch.utils.cpp_extension.load_inline to load cudnn functions. If it is hanging, pleast try to delete torch_extensions cache directory. (In most cases, the directory is /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/) spikingjelly.clock_driven.spike_op: try to use torch.utils.cpp_extension.load_inline to load cudnn functions. If it is hanging, pleast try to delete torch_extensions cache directory. (In most cases, the directory is /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/) spikingjelly.clock_driven.spike_op: try to use torch.utils.cpp_extension.load_inline to load cudnn functions. If it is hanging, pleast try to delete torch_extensions cache directory. (In most cases, the directory is /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/) spikingjelly.clock_driven.spike_op: try to use torch.utils.cpp_extension.load_inline to load cudnn functions. If it is hanging, pleast try to delete torch_extensions cache directory. (In most cases, the directory is /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/) spikingjelly.clock_driven.spike_op: try to use torch.utils.cpp_extension.load_inline to load cudnn functions. If it is hanging, pleast try to delete torch_extensions cache directory. (In most cases, the directory is /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/) spikingjelly.clock_driven.spike_op: try to use torch.utils.cpp_extension.load_inline to load cudnn functions. If it is hanging, pleast try to delete torch_extensions cache directory. (In most cases, the directory is /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/) spikingjelly.clock_driven.spike_op: try to use torch.utils.cpp_extension.load_inline to load cudnn functions. If it is hanging, pleast try to delete torch_extensions cache directory. (In most cases, the directory is /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/) spikingjelly.clock_driven.spike_op: try to use torch.utils.cpp_extension.load_inline to load cudnn functions. If it is hanging, pleast try to delete torch_extensions cache directory. (In most cases, the directory is /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/) spikingjelly.clock_driven.spike_op: Error building extension 'cpp_wrapper': [1/2] /jmain02/apps/gcc9/9.1.0/bin/g++ -MMD -MF main.o.d -DTORCH_EXTENSION_NAME=cpp_wrapper -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/include -isystem /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/include/TH -isystem /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/include/THC -isystem /jmain02/apps/cuda/10.2/include -isystem /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/cpp_wrapper/main.cpp -o main.o FAILED: main.o /jmain02/apps/gcc9/9.1.0/bin/g++ -MMD -MF main.o.d -DTORCH_EXTENSION_NAME=cpp_wrapper -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/include -isystem /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/include/TH -isystem /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/include/THC -isystem /jmain02/apps/cuda/10.2/include -isystem /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/cpp_wrapper/main.cpp -o main.o /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/cpp_wrapper/main.cpp: In function ‘void pybind11_init_cppwrapper(pybind11::module&)’: /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/cpp_wrapper/main.cpp:4:65: error: ‘cudnn_convolution_backward’ was not declared in this scope 4 | m.def("cudnn_convolution_backward", torch::wrap_pybind_function(cudnn_convolution_backward), "cudnn_convolution_backward"); | ^~~~~~ /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/cpp_wrapper/main.cpp:5:71: error: ‘cudnn_convolution_backward_input’ was not declared in this scope 5 | m.def("cudnn_convolution_backward_input", torch::wrap_pybind_function(cudnn_convolution_backward_input), "cudnn_convolution_backward_input"); | ^~~~~~~~ /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/cpp_wrapper/main.cpp:6:72: error: ‘cudnn_convolution_backward_weight’ was not declared in this scope 6 | m.def("cudnn_convolution_backward_weight", torch::wrap_pybind_function(cudnn_convolution_backward_weight), "cudnn_convolution_backward_weight"); | ^~~~~~~~~ ninja: build stopped: subcommand failed.

spikingjelly.clock_driven.spike_op: /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/cpp_wrapper/cpp_wrapper.so: cannot open shared object file: No such file or directory spikingjelly.clock_driven.spike_op: /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/cpp_wrapper/cpp_wrapper.so: cannot open shared object file: No such file or directory spikingjelly.clock_driven.spike_op: /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/cpp_wrapper/cpp_wrapper.so: cannot open shared object file: No such file or directory spikingjelly.clock_driven.spike_op: /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/cpp_wrapper/cpp_wrapper.so: cannot open shared object file: No such file or directory spikingjelly.clock_driven.spike_op: /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/cpp_wrapper/cpp_wrapper.so: cannot open shared object file: No such file or directory spikingjelly.clock_driven.spike_op: /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/cpp_wrapper/cpp_wrapper.so: cannot open shared object file: No such file or directory | distributed init (rank 0): env:// spikingjelly.clock_driven.spike_op: /jmain02/home/J2AD019/exk01/bxg73-exk01/.cache/torch_extensions/py37_cu102/cpp_wrapper/cpp_wrapper.so: cannot open shared object file: No such file or directory | distributed init (rank 2): env:// | distributed init (rank 5): env:// | distributed init (rank 7): env:// | distributed init (rank 1): env:// | distributed init (rank 4): env:// | distributed init (rank 6): env:// | distributed init (rank 3): env:// Namespace(T=4, adam=False, amp=True, batch_size=32, cache_dataset=True, connect_f='ADD', cos_lr_T=320, data_path='/jmain02/home/J2AD019/exk01/bxg73-exk01/traindata/imagenet', device='cuda', dist_backend='nccl', dist_url='env://', distributed=True, epochs=320, gpu=0, lr=0.1, model='sew_resnet18', momentum=0.9, output_dir='./logs', print_freq=1, rank=0, resume='', start_epoch=0, sync_bn=False, tb=True, test_only=False, weight_decay=0, workers=16, world_size=8, zero_init_residual=False) Loading data Loading training data Loading dataset_train from /jmain02/home/J2AD019/exk01/bxg73-exk01/.torch/vision/datasets/imagefolder/2074a9ecc8.pt Took 15.483056783676147 Loading validation data Loading dataset_test from /jmain02/home/J2AD019/exk01/bxg73-exk01/.torch/vision/datasets/imagefolder/9d382b0ab2.pt Creating data loaders dataset_train:1281167, dataset_test:50000 /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/utils/data/dataloader.py:490: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary. cpuset_checked)) Creating model /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/utils/data/dataloader.py:490: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary. cpuset_checked)) /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/utils/data/dataloader.py:490: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary. cpuset_checked)) /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/utils/data/dataloader.py:490: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary. cpuset_checked)) /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/utils/data/dataloader.py:490: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary. cpuset_checked)) /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/utils/data/dataloader.py:490: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary. cpuset_checked)) /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/utils/data/dataloader.py:490: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary. cpuset_checked)) /jmain02/home/J2AD019/exk01/bxg73-exk01/.conda/envs/snn/lib/python3.7/site-packages/torch/utils/data/dataloader.py:490: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary. cpuset_checked)) SEWResNet( (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False) (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (sn1): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) (maxpool): SeqToANNContainer( (0): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False) ) (layer1): Sequential( (0): BasicBlock( (conv1): SeqToANNContainer( (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (sn1): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) (conv2): SeqToANNContainer( (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (sn2): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) ) (1): BasicBlock( (conv1): SeqToANNContainer( (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (sn1): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) (conv2): SeqToANNContainer( (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (sn2): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) ) ) (layer2): Sequential( (0): BasicBlock( (conv1): SeqToANNContainer( (0): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (sn1): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) (conv2): SeqToANNContainer( (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (downsample): Sequential( (0): SeqToANNContainer( (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (1): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) ) (sn2): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) ) (1): BasicBlock( (conv1): SeqToANNContainer( (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (sn1): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) (conv2): SeqToANNContainer( (0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (sn2): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) ) ) (layer3): Sequential( (0): BasicBlock( (conv1): SeqToANNContainer( (0): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (sn1): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) (conv2): SeqToANNContainer( (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (downsample): Sequential( (0): SeqToANNContainer( (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (1): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) ) (sn2): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) ) (1): BasicBlock( (conv1): SeqToANNContainer( (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (sn1): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) (conv2): SeqToANNContainer( (0): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (sn2): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) ) ) (layer4): Sequential( (0): BasicBlock( (conv1): SeqToANNContainer( (0): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (sn1): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) (conv2): SeqToANNContainer( (0): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (downsample): Sequential( (0): SeqToANNContainer( (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (1): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) ) (sn2): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) ) (1): BasicBlock( (conv1): SeqToANNContainer( (0): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (sn1): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) (conv2): SeqToANNContainer( (0): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) ) (sn2): MultiStepIFNode( v_threshold=1.0, v_reset=0.0, detach_reset=True, backend=torch (surrogate_function): Sigmoid(alpha=4.0, spiking=True) ) ) ) (avgpool): SeqToANNContainer( (0): AdaptiveAvgPool2d(output_size=(1, 1)) ) (fc): Linear(in_features=512, out_features=1000, bias=True) ) purge_step_train=0, purge_step_te=0 Start training Epoch: [0] [ 0/5005] eta: 6 days, 4:04:26 lr: 0.1 img/s: 0.7308301416419292 loss: 7.0216 (7.0216) acc1: 0.0000 (0.0000) acc5: 0.0000 (0.0000) time: 106.5069 data: 62.7210 max mem: 3437 Epoch: [0] [ 1/5005] eta: 3 days, 2:58:16 lr: 0.1 img/s: 23.49479489232508 loss: 7.0216 (7.0388) acc1: 0.0000 (0.0000) acc5: 0.0000 (0.0000) time: 53.9361 data: 31.3621 max mem: 3524 Epoch: [0] [ 2/5005] eta: 2 days, 2:34:21 lr: 0.1 img/s: 24.62836232747285 loss: 7.0216 (7.0147) acc1: 0.0000 (0.0000) acc5: 0.0000 (0.0000) time: 36.3905 data: 20.9081 max mem: 3524 Epoch: [0] [ 3/5005] eta: 1 day, 14:25:50 lr: 0.1 img/s: 21.858091001681647 loss: 6.9934 (7.0094) acc1: 0.0000 (0.0000) acc5: 0.0000 (0.0000) time: 27.6590 data: 15.6812 max mem: 3524 Epoch: [0] [ 4/5005] eta: 1 day, 7:34:30 lr: 0.1 img/s: 10.625388394919328 loss: 6.9934 (6.9903) acc1: 0.0000 (0.0000) acc5: 0.0000 (0.0000) time: 22.7296 data: 12.5450 max mem: 3524 Epoch: [0] [ 5/5005] eta: 1 day, 2:43:45 lr: 0.1 img/s: 17.564296266614356 loss: 6.9934 (6.9998) acc1: 0.0000 (0.0000) acc5: 0.0000 (0.0000) time: 19.2450 data: 10.4542 max mem: 3524 Epoch: [0] [ 6/5005] eta: 23:24:00 lr: 0.1 img/s: 12.853481808788054 loss: 6.9934 (6.9983) acc1: 0.0000 (0.0000) acc5: 0.0000 (0.0000) time: 16.8514 data: 8.9608 max mem: 3524 Epoch: [0] [ 7/5005] eta: 20:48:42 lr: 0.1 img/s: 16.305018585509014 loss: 6.9918 (6.9975) acc1: 0.0000 (0.0000) acc5: 0.0000 (0.0000) time: 14.9904 data: 7.8408 max mem: 3524 Epoch: [0] [ 8/5005] eta: 18:54:14 lr: 0.1 img/s: 12.083899079818286 loss: 6.9918 (6.9939) acc1: 0.0000 (0.0000) acc5: 0.0000 (0.0000) time: 13.6191 data: 6.9696 max mem: 3524 Epoch: [0] [ 9/5005] eta: 17:19:34 lr: 0.1 img/s: 14.090440279646748 loss: 6.9894 (6.9834) acc1: 0.0000 (0.0000) acc5: 0.0000 (0.3125) time: 12.4848 data: 6.2732 max mem: 3524 Epoch: [0] [ 10/5005] eta: 16:04:31 lr: 0.1 img/s: 12.32699510677423 loss: 6.9894 (6.9822) acc1: 0.0000 (0.0000) acc5: 0.0000 (0.2841) time: 11.5858 data: 5.7029 max mem: 3524 Epoch: [0] [ 11/5005] eta: 15:18:06 lr: 0.1 img/s: 6.50177249204728 loss: 6.9692 (6.9790) acc1: 0.0000 (0.2604) acc5: 0.0000 (0.5208) time: 11.0305 data: 5.2277 max mem: 3524 Epoch: [0] [ 12/5005] eta: 14:18:37 lr: 0.1 img/s: 18.12335051358464 loss: 6.9692 (6.9768) acc1: 0.0000 (0.2404) acc5: 0.0000 (0.4808) time: 10.3179 data: 4.8256 max mem: 3524 Epoch: [0] [ 13/5005] eta: 13:27:36 lr: 0.1 img/s: 18.149013740856038 loss: 6.9692 (6.9813) acc1: 0.0000 (0.2232) acc5: 0.0000 (0.4464) time: 9.7069 data: 4.4810 max mem: 3524 Epoch: [0] [ 14/5005] eta: 12:43:13 lr: 0.1 img/s: 18.471869146157733 loss: 6.9692 (6.9771) acc1: 0.0000 (0.2083) acc5: 0.0000 (0.4167) time: 9.1752 data: 4.1822 max mem: 3524 Epoch: [0] [ 15/5005] eta: 12:06:43 lr: 0.1 img/s: 14.666900227875166 loss: 6.9666 (6.9703) acc1: 0.0000 (0.1953) acc5: 0.0000 (0.3906) time: 8.7382 data: 3.9209 max mem: 3524 Epoch: [0] [ 16/5005] eta: 11:32:30 lr: 0.1 img/s: 18.041901419005722 loss: 6.9692 (6.9775) acc1: 0.0000 (0.1838) acc5: 0.0000 (0.3676) time: 8.3285 data: 3.6902 max mem: 3524 Epoch: [0] [ 17/5005] eta: 11:16:35 lr: 0.1 img/s: 6.51747352966013 loss: 6.9666 (6.9750) acc1: 0.0000 (0.1736) acc5: 0.0000 (0.3472) time: 8.1386 data: 3.4853 max mem: 3524 Epoch: [0] [ 18/5005] eta: 10:53:12 lr: 0.1 img/s: 11.325595367481371 loss: 6.9692 (6.9761) acc1: 0.0000 (0.1645) acc5: 0.0000 (0.3289) time: 7.8590 data: 3.3018 max mem: 3524 Epoch: [0] [ 19/5005] eta: 10:30:05 lr: 0.1 img/s: 13.766327456646925 loss: 6.9666 (6.9738) acc1: 0.0000 (0.1562) acc5: 0.0000 (0.3125) time: 7.5823 data: 3.1368 max mem: 3524 Epoch: [0] [ 20/5005] eta: 10:07:59 lr: 0.1 img/s: 15.786012288410692 loss: 6.9666 (6.9786) acc1: 0.0000 (0.1488) acc5: 0.0000 (0.2976) time: 2.3583 data: 0.0008 max mem: 3524 Epoch: [0] [ 21/5005] eta: 9:48:50 lr: 0.1 img/s: 14.043265961066577 loss: 6.9645 (6.9779) acc1: 0.0000 (0.1420) acc5: 0.0000 (0.2841) time: 2.4040 data: 0.0006 max mem: 3524 Epoch: [0] [ 22/5005] eta: 9:31:15 lr: 0.1 img/s: 14.203568455649554 loss: 6.9645 (6.9795) acc1: 0.0000 (0.1359) acc5: 0.0000 (0.2717) time: 2.4517 data: 0.0006 max mem: 3524 Epoch: [0] [ 23/5005] eta: 9:15:05 lr: 0.1 img/s: 14.298295489046843 loss: 6.9645 (6.9797) acc1: 0.0000 (0.1302) acc5: 0.0000 (0.2604) time: 2.4904 data: 0.0006 max mem: 3524 Epoch: [0] [ 24/5005] eta: 9:11:22 lr: 0.1 img/s: 5.718321583280166 loss: 6.9645 (6.9789) acc1: 0.0000 (0.1250) acc5: 0.0000 (0.2500) time: 2.6197 data: 0.0006 max mem: 3524 Epoch: [0] [ 25/5005] eta: 8:54:16 lr: 0.1 img/s: 24.56634379552959 loss: 6.9640 (6.9782) acc1: 0.0000 (0.1202) acc5: 0.0000 (0.3606) time: 2.5946 data: 0.0015 max mem: 3524 Epoch: [0] [ 26/5005] eta: 8:37:44 lr: 0.1 img/s: 29.3260355672145 loss: 6.9640 (6.9778) acc1: 0.0000 (0.1157) acc5: 0.0000 (0.3472) time: 2.5247 data: 0.0016 max mem: 3524 Epoch: [0] [ 27/5005] eta: 8:23:35 lr: 0.1 img/s: 21.380297933808205 loss: 6.9628 (6.9769) acc1: 0.0000 (0.1116) acc5: 0.0000 (0.3348) time: 2.5014 data: 0.0016 max mem: 3524 Epoch: [0] [ 28/5005] eta: 8:13:12 lr: 0.1 img/s: 12.914981189149076 loss: 6.9628 (6.9814) acc1: 0.0000 (0.1078) acc5: 0.0000 (0.3233) time: 2.4929 data: 0.0016 max mem: 3524 Epoch: [0] [ 29/5005] eta: 8:02:42 lr: 0.1 img/s: 14.646432815199637 loss: 6.9628 (6.9806) acc1: 0.0000 (0.1042) acc5: 0.0000 (0.4167) time: 2.4884 data: 0.0014 max mem: 3524 Epoch: [0] [ 30/5005] eta: 7:53:57 lr: 0.1 img/s: 12.386998839264479 loss: 6.9623 (6.9800) acc1: 0.0000 (0.1008) acc5: 0.0000 (0.4032) time: 2.4878 data: 0.0014 max mem: 3524 Epoch: [0] [ 31/5005] eta: 7:44:55 lr: 0.1 img/s: 14.150348788491614 loss: 6.9628 (6.9810) acc1: 0.0000 (0.0977) acc5: 0.0000 (0.3906) time: 2.3547 data: 0.0014 max mem: 3524 Epoch: [0] [ 32/5005] eta: 7:36:43 lr: 0.1 img/s: 13.409478989662277 loss: 6.9640 (6.9816) acc1: 0.0000 (0.0947) acc5: 0.0000 (0.3788) time: 2.3858 data: 0.0014 max mem: 3524 Epoch: [0] [ 33/5005] eta: 7:29:16 lr: 0.1 img/s: 12.856338190062846 loss: 6.9640 (6.9852) acc1: 0.0000 (0.0919) acc5: 0.0000 (0.3676) time: 2.4221 data: 0.0014 max mem: 3524 Epoch: [0] [ 34/5005] eta: 7:21:27 lr: 0.1 img/s: 15.752324481312899 loss: 6.9640 (6.9838) acc1: 0.0000 (0.0893) acc5: 0.0000 (0.4464) time: 2.4434 data: 0.0077 max mem: 3524 Epoch: [0] [ 35/5005] eta: 7:13:29 lr: 0.1 img/s: 16.8396130813212 loss: 6.9640 (6.9800) acc1: 0.0000 (0.0868) acc5: 0.0000 (0.4340) time: 2.4293 data: 0.0078 max mem: 3524 Epoch: [0] [ 36/5005] eta: 7:07:12 lr: 0.1 img/s: 12.983464037556143 loss: 6.9628 (6.9781) acc1: 0.0000 (0.0845) acc5: 0.0000 (0.4223) time: 2.4639 data: 0.0077 max mem: 3524 Epoch: [0] [ 37/5005] eta: 7:00:54 lr: 0.1 img/s: 13.873551118626969 loss: 6.9628 (6.9775) acc1: 0.0000 (0.0822) acc5: 0.0000 (0.4934) time: 2.3337 data: 0.0078 max mem: 3524 Epoch: [0] [ 38/5005] eta: 6:54:23 lr: 0.1 img/s: 15.569952985416755 loss: 6.9623 (6.9742) acc1: 0.0000 (0.0801) acc5: 0.0000 (0.4808) time: 2.2952 data: 0.0077 max mem: 3524 Epoch: [0] [ 39/5005] eta: 6:55:38 lr: 0.1 img/s: 5.668362093937356 loss: 6.9623 (6.9738) acc1: 0.0000 (0.1562) acc5: 0.0000 (0.5469) time: 2.4612 data: 0.0077 max mem: 3524 Epoch: [0] [ 40/5005] eta: 6:48:04 lr: 0.1 img/s: 24.98817835479234 loss: 6.9623 (6.9762) acc1: 0.0000 (0.1524) acc5: 0.0000 (0.5335) time: 2.4257 data: 0.0095 max mem: 3524 Epoch: [0] [ 41/5005] eta: 6:41:15 lr: 0.1 img/s: 21.103622079334734 loss: 6.9578 (6.9744) acc1: 0.0000 (0.1488) acc5: 0.0000 (0.5208) time: 2.3876 data: 0.0095 max mem: 3524 Epoch: [0] [ 42/5005] eta: 6:35:08 lr: 0.1 img/s: 18.793300163292486 loss: 6.9578 (6.9743) acc1: 0.0000 (0.1453) acc5: 0.0000 (0.5087) time: 2.3603 data: 0.0098 max mem: 3524 Epoch: [0] [ 43/5005] eta: 6:29:16 lr: 0.1 img/s: 18.880180145456652 loss: 6.9557 (6.9694) acc1: 0.0000 (0.1420) acc5: 0.0000 (0.5682) time: 2.3332 data: 0.0098 max mem: 3524 Epoch: [0] [ 44/5005] eta: 6:24:16 lr: 0.1 img/s: 15.740176442601271 loss: 6.9552 (6.9689) acc1: 0.0000 (0.1389) acc5: 0.0000 (0.5556) time: 2.1550 data: 0.0098 max mem: 3524 Epoch: [0] [ 45/5005] eta: 6:25:49 lr: 0.1 img/s: 5.760671425302737 loss: 6.9552 (6.9691) acc1: 0.0000 (0.1359) acc5: 0.0000 (0.5435) time: 2.3668 data: 0.0089 max mem: 3524 Epoch: [0] [ 46/5005] eta: 6:23:22 lr: 0.1 img/s: 9.647872429131288 loss: 6.9552 (6.9707) acc1: 0.0000 (0.1330) acc5: 0.0000 (0.5319) time: 2.4780 data: 0.0088 max mem: 3524 Epoch: [0] [ 47/5005] eta: 6:18:43 lr: 0.1 img/s: 16.174458624885034 loss: 6.9557 (6.9720) acc1: 0.0000 (0.1302) acc5: 0.0000 (0.5208) time: 2.5020 data: 0.0088 max mem: 3524 Epoch: [0] [ 48/5005] eta: 6:15:19 lr: 0.1 img/s: 12.231416261953218 loss: 6.9557 (6.9726) acc1: 0.0000 (0.1276) acc5: 0.0000 (0.5102) time: 2.5089 data: 0.0088 max mem: 3524 Epoch: [0] [ 49/5005] eta: 6:10:24 lr: 0.1 img/s: 19.869536858155872 loss: 6.9623 (6.9760) acc1: 0.0000 (0.1250) acc5: 0.0000 (0.5000) time: 2.4802 data: 0.0088 max mem: 3524 Epoch: [0] [ 50/5005] eta: 6:07:11 lr: 0.1 img/s: 12.60857636316922 loss: 6.9557 (6.9751) acc1: 0.0000 (0.1225) acc5: 0.0000 (0.4902) time: 2.4779 data: 0.0088 max mem: 3524 Epoch: [0] [ 51/5005] eta: 6:03:18 lr: 0.1 img/s: 15.568065740059938 loss: 6.9557 (6.9758) acc1: 0.0000 (0.1202) acc5: 0.0000 (0.5409) time: 2.4676 data: 0.0088 max mem: 3524 Epoch: [0] [ 52/5005] eta: 6:02:45 lr: 0.1 img/s: 7.816214406199035 loss: 6.9557 (6.9767) acc1: 0.0000 (0.1179) acc5: 0.0000 (0.5307) time: 2.5530 data: 0.0087 max mem: 3524 Epoch: [0] [ 53/5005] eta: 6:00:22 lr: 0.1 img/s: 11.134163213900502 loss: 6.9557 (6.9774) acc1: 0.0000 (0.1157) acc5: 0.0000 (0.5208) time: 2.5722 data: 0.0087 max mem: 3524 Epoch: [0] [ 54/5005] eta: 5:58:51 lr: 0.1 img/s: 9.389494026286272 loss: 6.9718 (6.9794) acc1: 0.0000 (0.1136) acc5: 0.0000 (0.5114) time: 2.6347 data: 0.0024 max mem: 3524 Epoch: [0] [ 55/5005] eta: 5:54:28 lr: 0.1 img/s: 23.205246113418553 loss: 6.9718 (6.9793) acc1: 0.0000 (0.1116) acc5: 0.0000 (0.5022) time: 2.6108 data: 0.0046 max mem: 3524 Epoch: [0] [ 56/5005] eta: 5:51:02 lr: 0.1 img/s: 16.234486954554885 loss: 6.9718 (6.9788) acc1: 0.0000 (0.1096) acc5: 0.0000 (0.5482) time: 2.5862 data: 0.0046 max mem: 3524 Epoch: [0] [ 57/5005] eta: 5:47:18 lr: 0.1 img/s: 19.029912858197527 loss: 6.9718 (6.9778) acc1: 0.0000 (0.1616) acc5: 0.0000 (0.5927) time: 2.5549 data: 0.0046 max mem: 3524 Epoch: [0] [ 58/5005] eta: 5:43:44 lr: 0.1 img/s: 18.707737596610194 loss: 6.9741 (6.9790) acc1: 0.0000 (0.1589) acc5: 0.0000 (0.5826) time: 2.5377 data: 0.0046 max mem: 3524 Epoch: [0] [ 59/5005] eta: 5:41:10 lr: 0.1 img/s: 13.627180746260686 loss: 6.9741 (6.9785) acc1: 0.0000 (0.1562) acc5: 0.0000 (0.5729) time: 2.3728 data: 0.0046 max mem: 3524 Epoch: [0] [ 60/5005] eta: 5:38:10 lr: 0.1 img/s: 16.272849209288847 loss: 6.9741 (6.9795) acc1: 0.0000 (0.1537) acc5: 0.0000 (0.5635) time: 2.4053 data: 0.0028 max mem: 3524 Epoch: [0] [ 61/5005] eta: 5:41:51 lr: 0.1 img/s: 4.6151750845923445 loss: 6.9962 (6.9797) acc1: 0.0000 (0.1512) acc5: 0.0000 (0.5544) time: 2.6762 data: 0.0028 max mem: 3524 Epoch: [0] [ 62/5005] eta: 5:41:30 lr: 0.1 img/s: 8.1324544898701 loss: 6.9962 (6.9784) acc1: 0.0000 (0.1488) acc5: 0.0000 (0.5456) time: 2.7875 data: 0.0025 max mem: 3524 Epoch: [0] [ 63/5005] eta: 5:37:52 lr: 0.1 img/s: 23.559456336419707 loss: 7.0024 (6.9796) acc1: 0.0000 (0.1465) acc5: 0.0000 (0.5371) time: 2.7711 data: 0.0029 max mem: 3524 Epoch: [0] [ 64/5005] eta: 5:34:14 lr: 0.1 img/s: 24.684927667266113 loss: 7.0024 (6.9796) acc1: 0.0000 (0.1442) acc5: 0.0000 (0.5769) time: 2.7343 data: 0.0029 max mem: 3524 Epoch: [0] [ 65/5005] eta: 5:31:22 lr: 0.1 img/s: 17.659367636049296 loss: 7.0024 (6.9771) acc1: 0.0000 (0.1420) acc5: 0.0000 (0.5682) time: 2.5471 data: 0.0029 max mem: 3524 Epoch: [0] [ 66/5005] eta: 5:29:10 lr: 0.1 img/s: 13.959540361196481 loss: 6.9962 (6.9774) acc1: 0.0000 (0.1399) acc5: 0.0000 (0.5597) time: 2.4959 data: 0.0029 max mem: 3524 Epoch: [0] [ 67/5005] eta: 5:26:53 lr: 0.1 img/s: 14.787660170687875 loss: 6.9913 (6.9727) acc1: 0.0000 (0.1379) acc5: 0.0000 (0.5974) time: 2.5052 data: 0.0030 max mem: 3524 Epoch: [0] [ 68/5005] eta: 5:27:38 lr: 0.1 img/s: 6.886882401490695 loss: 6.9781 (6.9710) acc1: 0.0000 (0.1812) acc5: 0.0000 (0.6341) time: 2.6068 data: 0.0030 max mem: 3524 Epoch: [0] [ 69/5005] eta: 5:25:14 lr: 0.1 img/s: 15.960058984962545 loss: 6.9781 (6.9713) acc1: 0.0000 (0.1786) acc5: 0.0000 (0.6250) time: 2.6265 data: 0.0030 max mem: 3524 Epoch: [0] [ 70/5005] eta: 5:25:50 lr: 0.1 img/s: 7.079491449504786 loss: 6.9913 (6.9728) acc1: 0.0000 (0.1761) acc5: 0.0000 (0.6602) time: 2.7256 data: 0.0030 max mem: 3524 Epoch: [0] [ 71/5005] eta: 5:22:56 lr: 0.1 img/s: 21.659937104106792 loss: 6.9781 (6.9704) acc1: 0.0000 (0.1736) acc5: 0.0000 (0.6510) time: 2.6967 data: 0.0030 max mem: 3524 Epoch: [0] [ 72/5005] eta: 5:20:32 lr: 0.1 img/s: 17.257318926875406 loss: 6.9733 (6.9704) acc1: 0.0000 (0.1712) acc5: 0.0000 (0.6421) time: 2.5848 data: 0.0031 max mem: 3524 Epoch: [0] [ 73/5005] eta: 5:18:27 lr: 0.1 img/s: 15.530416677418112 loss: 6.9733 (6.9713) acc1: 0.0000 (0.1689) acc5: 0.0000 (0.6334) time: 2.5450 data: 0.0040 max mem: 3524 Epoch: [0] [ 74/5005] eta: 5:15:51 lr: 0.1 img/s: 20.423674089737258 loss: 6.9733 (6.9726) acc1: 0.0000 (0.1667) acc5: 0.0000 (0.6250) time: 2.4530 data: 0.0040 max mem: 3524 Epoch: [0] [ 75/5005] eta: 5:13:52 lr: 0.1 img/s: 15.51822618885476 loss: 6.9733 (6.9714) acc1: 0.0000 (0.1645) acc5: 0.0000 (0.6579) time: 2.4849 data: 0.0018 max mem: 3524 Epoch: [0] [ 76/5005] eta: 5:14:18 lr: 0.1 img/s: 7.450328112519955 loss: 6.9733 (6.9683) acc1: 0.0000 (0.1623) acc5: 0.0000 (0.7305) time: 2.6011 data: 0.0018 max mem: 3524 Epoch: [0] [ 77/5005] eta: 5:13:18 lr: 0.1 img/s: 10.915775017693093 loss: 6.9781 (6.9685) acc1: 0.0000 (0.1603) acc5: 0.0000 (0.7212) time: 2.6636 data: 0.0018 max mem: 3524 Epoch: [0] [ 78/5005] eta: 5:11:14 lr: 0.1 img/s: 16.880040768340344 loss: 6.9733 (6.9650) acc1: 0.0000 (0.1582) acc5: 0.0000 (0.7911) time: 2.6729 data: 0.0018 max mem: 3524 Epoch: [0] [ 79/5005] eta: 5:09:32 lr: 0.1 img/s: 14.67835121989203 loss: 6.9733 (6.9646) acc1: 0.0000 (0.1562) acc5: 0.0000 (0.8203) time: 2.6645 data: 0.0018 max mem: 3524 Epoch: [0] [ 80/5005] eta: 5:07:28 lr: 0.1 img/s: 17.848187571110714 loss: 6.9327 (6.9637) acc1: 0.0000 (0.1543) acc5: 0.0000 (0.8488) time: 2.6558 data: 0.0018 max mem: 3524 Epoch: [0] [ 81/5005] eta: 5:05:46 lr: 0.1 img/s: 15.090654294714156 loss: 6.9327 (6.9653) acc1: 0.0000 (0.1524) acc5: 0.0000 (0.8384) time: 2.4151 data: 0.0018 max mem: 3524 Epoch: [0] [ 82/5005] eta: 5:03:58 lr: 0.1 img/s: 16.306375521701685 loss: 6.9327 (6.9647) acc1: 0.0000 (0.1506) acc5: 0.0000 (0.8283) time: 2.3166 data: 0.0019 max mem: 3524 Epoch: [0] [ 83/5005] eta: 5:02:50 lr: 0.1 img/s: 12.258777659894724 loss: 6.9242 (6.9642) acc1: 0.0000 (0.1488) acc5: 0.0000 (0.8185) time: 2.3788 data: 0.0015 max mem: 3524 Epoch: [0] [ 84/5005] eta: 5:01:30 lr: 0.1 img/s: 13.508237678556673 loss: 6.9127 (6.9610) acc1: 0.0000 (0.1471) acc5: 0.0000 (0.8088) time: 2.4325 data: 0.0015 max mem: 3524 Epoch: [0] [ 85/5005] eta: 5:01:28 lr: 0.1 img/s: 10.424231786274623 loss: 6.9242 (6.9619) acc1: 0.0000 (0.1453) acc5: 0.0000 (0.8358) time: 2.5268 data: 0.0313 max mem: 3524 Epoch: [0] [ 86/5005] eta: 4:59:54 lr: 0.1 img/s: 15.34704238551641 loss: 6.9127 (6.9572) acc1: 0.0000 (0.1796) acc5: 0.0000 (0.8980) time: 2.5164 data: 0.0313 max mem: 3524 Epoch: [0] [ 87/5005] eta: 4:59:07 lr: 0.1 img/s: 11.220261081553836 loss: 6.9127 (6.9556) acc1: 0.0000 (0.1776) acc5: 0.0000 (0.9233) time: 2.5527 data: 0.0332 max mem: 3524 Epoch: [0] [ 88/5005] eta: 4:57:35 lr: 0.1 img/s: 15.612441961259272 loss: 6.9127 (6.9535) acc1: 0.0000 (0.1756) acc5: 0.0000 (0.9129) time: 2.4229 data: 0.0332 max mem: 3524 Epoch: [0] [ 89/5005] eta: 4:56:09 lr: 0.1 img/s: 15.2014627719819 loss: 6.8898 (6.9525) acc1: 0.0000 (0.1736) acc5: 0.0000 (0.9375) time: 2.4280 data: 0.0333 max mem: 3524 Epoch: [0] [ 90/5005] eta: 5:00:36 lr: 0.1 img/s: 3.70862829658896 loss: 6.8898 (6.9519) acc1: 0.0000 (0.1717) acc5: 0.0000 (0.9615) time: 2.6334 data: 0.0333 max mem: 3524 Epoch: [0] [ 91/5005] eta: 4:58:11 lr: 0.1 img/s: 31.430338306882568 loss: 6.8989 (6.9518) acc1: 0.0000 (0.1698) acc5: 0.0000 (0.9511) time: 2.6105 data: 0.0333 max mem: 3524 Epoch: [0] [ 92/5005] eta: 4:55:57 lr: 0.1 img/s: 27.055443836124528 loss: 6.8898 (6.9497) acc1: 0.0000 (0.2016) acc5: 3.1250 (0.9745) time: 2.5768 data: 0.0332 max mem: 3524 Epoch: [0] [ 93/5005] eta: 4:55:20 lr: 0.1 img/s: 10.765443704286495 loss: 6.8854 (6.9467) acc1: 0.0000 (0.2327) acc5: 3.1250 (0.9973) time: 2.6215 data: 0.0323 max mem: 3524 Epoch: [0] [ 94/5005] eta: 4:53:33 lr: 0.1 img/s: 19.85491964923269 loss: 6.8854 (6.9468) acc1: 0.0000 (0.2303) acc5: 3.1250 (0.9868) time: 2.6237 data: 0.0323 max mem: 3524 Epoch: [0] [ 95/5005] eta: 4:52:01 lr: 0.1 img/s: 17.172281508381833 loss: 6.8593 (6.9457) acc1: 0.0000 (0.2279) acc5: 3.1250 (0.9766) time: 2.6138 data: 0.0323 max mem: 3524 Epoch: [0] [ 96/5005] eta: 4:50:46 lr: 0.1 img/s: 14.887674245695596 loss: 6.8898 (6.9470) acc1: 0.0000 (0.2255) acc5: 0.0000 (0.9665) time: 2.5065 data: 0.0323 max mem: 3524 Epoch: [0] [ 97/5005] eta: 4:49:31 lr: 0.1 img/s: 14.993944899726994 loss: 6.8898 (6.9475) acc1: 0.0000 (0.2232) acc5: 0.0000 (0.9566) time: 2.4666 data: 0.0323 max mem: 3524 Epoch: [0] [ 98/5005] eta: 4:48:38 lr: 0.1 img/s: 12.664206310602891 loss: 6.8989 (6.9483) acc1: 0.0000 (0.2210) acc5: 0.0000 (0.9470) time: 2.4982 data: 0.0323 max mem: 3524 Epoch: [0] [ 99/5005] eta: 4:47:30 lr: 0.1 img/s: 14.551470382562051 loss: 6.8898 (6.9474) acc1: 0.0000 (0.2188) acc5: 0.0000 (1.0000) time: 2.5002 data: 0.0333 max mem: 3524 Epoch: [0] [ 100/5005] eta: 4:46:36 lr: 0.1 img/s: 12.95119376479493 loss: 6.8989 (6.9486) acc1: 0.0000 (0.2166) acc5: 0.0000 (0.9901) time: 2.5341 data: 0.0333 max mem: 3524 Epoch: [0] [ 101/5005] eta: 4:45:33 lr: 0.1 img/s: 14.079115436434792 loss: 6.8593 (6.9476) acc1: 0.0000 (0.2145) acc5: 0.0000 (0.9804) time: 2.5421 data: 0.0337 max mem: 3524 Epoch: [0] [ 102/5005] eta: 4:44:28 lr: 0.1 img/s: 14.620388827010379 loss: 6.8593 (6.9475) acc1: 0.0000 (0.2124) acc5: 0.0000 (0.9709) time: 2.5533 data: 0.0336 max mem: 3524 Epoch: [0] [ 103/5005] eta: 4:43:31 lr: 0.1 img/s: 13.62830983555788 loss: 6.8536 (6.9458) acc1: 0.0000 (0.2103) acc5: 0.0000 (0.9916) time: 2.5401 data: 0.0336 max mem: 3524 Epoch: [0] [ 104/5005] eta: 4:47:43 lr: 0.1 img/s: 3.5788954758177014 loss: 6.8593 (6.9466) acc1: 0.0000 (0.2083) acc5: 0.0000 (0.9821) time: 2.8689 data: 0.0337 max mem: 3524 Epoch: [0] [ 105/5005] eta: 4:45:43 lr: 0.1 img/s: 32.16833850427397 loss: 6.8593 (6.9471) acc1: 0.0000 (0.2064) acc5: 0.0000 (0.9729) time: 2.7337 data: 0.0039 max mem: 3524 Epoch: [0] [ 106/5005] eta: 4:44:06 lr: 0.1 img/s: 21.965399409794685 loss: 6.8593 (6.9458) acc1: 0.0000 (0.2044) acc5: 0.0000 (0.9638) time: 2.7023 data: 0.0039 max mem: 3524 Epoch: [0] [ 107/5005] eta: 4:42:24 lr: 0.1 img/s: 24.185731978558632 loss: 6.8593 (6.9428) acc1: 0.0000 (0.2025) acc5: 0.0000 (1.0417) time: 2.6240 data: 0.0021 max mem: 3524 Epoch: [0] [ 108/5005] eta: 4:41:14 lr: 0.1 img/s: 16.274224475577217 loss: 6.8989 (6.9431) acc1: 0.0000 (0.2007) acc5: 0.0000 (1.0321) time: 2.6198 data: 0.0021 max mem: 3524 Epoch: [0] [ 109/5005] eta: 4:40:12 lr: 0.1 img/s: 14.936840423055163 loss: 6.8989 (6.9427) acc1: 0.0000 (0.1989) acc5: 0.0000 (1.0227) time: 2.6215 data: 0.0019 max mem: 3524 Epoch: [0] [ 110/5005] eta: 4:38:57 lr: 0.1 img/s: 17.650403525406254 loss: 6.9363 (6.9445) acc1: 0.0000 (0.1971) acc5: 0.0000 (1.0135) time: 2.2808 data: 0.0019 max mem: 3524 Epoch: [0] [ 111/5005] eta: 4:39:52 lr: 0.1 img/s: 6.737601664359351 loss: 6.8937 (6.9439) acc1: 0.0000 (0.1953) acc5: 0.0000 (1.0324) time: 2.4674 data: 0.0020 max mem: 3524 Epoch: [0] [ 112/5005] eta: 4:38:58 lr: 0.1 img/s: 14.113799190530155 loss: 6.8937 (6.9416) acc1: 0.0000 (0.1936) acc5: 0.0000 (1.0232) time: 2.5216 data: 0.0020 max mem: 3524 Epoch: [0] [ 113/5005] eta: 4:38:03 lr: 0.1 img/s: 14.510838824362144 loss: 6.8937 (6.9409) acc1: 0.0000 (0.2193) acc5: 0.0000 (1.0417) time: 2.4832 data: 0.0020 max mem: 3524 Epoch: [0] [ 114/5005] eta: 4:37:13 lr: 0.1 img/s: 13.728385303035045 loss: 6.8937 (6.9413) acc1: 0.0000 (0.2174) acc5: 0.0000 (1.0598) time: 2.5192 data: 0.0020 max mem: 3524 Epoch: [0] [ 115/5005] eta: 4:36:22 lr: 0.1 img/s: 14.116843840157852 loss: 6.8937 (6.9385) acc1: 0.0000 (0.2155) acc5: 0.0000 (1.0506) time: 2.5394 data: 0.0020 max mem: 3524 Epoch: [0] [ 116/5005] eta: 4:37:07 lr: 0.1 img/s: 7.041334835566152 loss: 6.8937 (6.9382) acc1: 0.0000 (0.2137) acc5: 0.0000 (1.0417) time: 2.6591 data: 0.0020 max mem: 3524 Epoch: [0] [ 117/5005] eta: 4:35:58 lr: 0.1 img/s: 17.55354110301456 loss: 6.8794 (6.9367) acc1: 0.0000 (0.2119) acc5: 0.0000 (1.0593) time: 2.6435 data: 0.0020 max mem: 3524 Epoch: [0] [ 118/5005] eta: 4:34:52 lr: 0.1 img/s: 17.21154945038035 loss: 6.8643 (6.9358) acc1: 0.0000 (0.2101) acc5: 0.0000 (1.0504) time: 2.6101 data: 0.0019 max mem: 3524 Epoch: [0] [ 119/5005] eta: 4:33:57 lr: 0.1 img/s: 15.2663163081414 loss: 6.8643 (6.9345) acc1: 0.0000 (0.2083) acc5: 0.0000 (1.0417) time: 2.6040 data: 0.0009 max mem: 3524 Epoch: [0] [ 120/5005] eta: 4:32:47 lr: 0.1 img/s: 18.683396538070326 loss: 6.8468 (6.9314) acc1: 0.0000 (0.2324) acc5: 0.0000 (1.0589) time: 2.5660 data: 0.0009 max mem: 3524 Epoch: [0] [ 121/5005] eta: 4:32:29 lr: 0.1 img/s: 10.711076703285144 loss: 6.8265 (6.9286) acc1: 0.0000 (0.2561) acc5: 0.0000 (1.0758) time: 2.6014 data: 0.0005 max mem: 3524 Epoch: [0] [ 122/5005] eta: 4:32:35 lr: 0.1 img/s: 8.89404430875566 loss: 6.8130 (6.9268) acc1: 0.0000 (0.2795) acc5: 0.0000 (1.0925) time: 2.6719 data: 0.0005 max mem: 3524 Epoch: [0] [ 123/5005] eta: 4:31:40 lr: 0.1 img/s: 15.78177279036155 loss: 6.8130 (6.9245) acc1: 0.0000 (0.2772) acc5: 0.0000 (1.0837) time: 2.6558 data: 0.0005 max mem: 3524 Epoch: [0] [ 124/5005] eta: 4:30:31 lr: 0.1 img/s: 19.30677520762111 loss: 6.7854 (6.9232) acc1: 0.0000 (0.2750) acc5: 0.0000 (1.0750) time: 2.2915 data: 0.0004 max mem: 3524 Epoch: [0] [ 125/5005] eta: 4:29:44 lr: 0.1 img/s: 14.552835156480322 loss: 6.7662 (6.9211) acc1: 0.0000 (0.3224) acc5: 0.0000 (1.1161) time: 2.3517 data: 0.0004 max mem: 3524 Epoch: [0] [ 126/5005] eta: 4:28:53 lr: 0.1 img/s: 15.611428661821327 loss: 6.7595 (6.9196) acc1: 0.0000 (0.3199) acc5: 0.0000 (1.1319) time: 2.3822 data: 0.0012 max mem: 3524 Epoch: [0] [ 127/5005] eta: 4:29:26 lr: 0.1 img/s: 7.488247765552618 loss: 6.7662 (6.9185) acc1: 0.0000 (0.3174) acc5: 0.0000 (1.1230) time: 2.5297 data: 0.0012 max mem: 3524 Epoch: [0] [ 128/5005] eta: 4:29:09 lr: 0.1 img/s: 10.861623072154703 loss: 6.7595 (6.9170) acc1: 0.0000 (0.3149) acc5: 0.0000 (1.1143) time: 2.5787 data: 0.0012 max mem: 3524 Epoch: [0] [ 129/5005] eta: 4:28:22 lr: 0.1 img/s: 15.00557861001404 loss: 6.7595 (6.9171) acc1: 0.0000 (0.3125) acc5: 0.0000 (1.1298) time: 2.5782 data: 0.0012 max mem: 3524 Epoch: [0] [ 130/5005] eta: 4:27:34 lr: 0.1 img/s: 15.156624263837683 loss: 6.7595 (6.9175) acc1: 0.0000 (0.3101) acc5: 0.0000 (1.1212) time: 2.5931 data: 0.0012 max mem: 3524 Epoch: [0] [ 131/5005] eta: 4:26:44 lr: 0.1 img/s: 15.763428532100706 loss: 6.7595 (6.9173) acc1: 0.0000 (0.3078) acc5: 0.0000 (1.1127) time: 2.4571 data: 0.0012 max mem: 3524 Epoch: [0] [ 132/5005] eta: 4:26:08 lr: 0.1 img/s: 13.369753762062567 loss: 6.7595 (6.9158) acc1: 0.0000 (0.3055) acc5: 0.0000 (1.1513) time: 2.4634 data: 0.0012 max mem: 3524 Epoch: [0] [ 133/5005] eta: 4:25:25 lr: 0.1 img/s: 14.761155793208346 loss: 6.7313 (6.9124) acc1: 0.0000 (0.3032) acc5: 0.0000 (1.1660) time: 2.4615 data: 0.0012 max mem: 3524 Epoch: [0] [ 134/5005] eta: 4:26:15 lr: 0.1 img/s: 6.728650373418343 loss: 6.7313 (6.9128) acc1: 0.0000 (0.3009) acc5: 0.0000 (1.1806) time: 2.5828 data: 0.0012 max mem: 3524 Epoch: [0] [ 135/5005] eta: 4:25:55 lr: 0.1 img/s: 11.427109185313864 loss: 6.7595 (6.9122) acc1: 0.0000 (0.2987) acc5: 0.0000 (1.1719) time: 2.6094 data: 0.0011 max mem: 3524 Epoch: [0] [ 136/5005] eta: 4:25:10 lr: 0.1 img/s: 15.11706382950925 loss: 6.7313 (6.9096) acc1: 0.0000 (0.3193) acc5: 3.1250 (1.2089) time: 2.4881 data: 0.0011 max mem: 3524 Epoch: [0] [ 137/5005] eta: 4:24:34 lr: 0.1 img/s: 13.72624984672504 loss: 6.7313 (6.9093) acc1: 0.0000 (0.3170) acc5: 0.0000 (1.2002) time: 2.5135 data: 0.0011 max mem: 3524 Epoch: [0] [ 138/5005] eta: 4:23:38 lr: 0.1 img/s: 18.19213179014312 loss: 6.7274 (6.9064) acc1: 0.0000 (0.3147) acc5: 3.1250 (1.2365) time: 2.5085 data: 0.0011 max mem: 3524 Epoch: [0] [ 139/5005] eta: 4:23:03 lr: 0.1 img/s: 13.617405900123535 loss: 6.7274 (6.9065) acc1: 0.0000 (0.3125) acc5: 3.1250 (1.2277) time: 2.5212 data: 0.0011 max mem: 3524 Epoch: [0] [ 140/5005] eta: 4:22:36 lr: 0.1 img/s: 12.655278043673235 loss: 6.7274 (6.9046) acc1: 0.0000 (0.3324) acc5: 3.1250 (1.2411) time: 2.5619 data: 0.0011 max mem: 3524 Epoch: [0] [ 141/5005] eta: 4:22:02 lr: 0.1 img/s: 13.634999651550883 loss: 6.7313 (6.9046) acc1: 0.0000 (0.3301) acc5: 0.0000 (1.2324) time: 2.5299 data: 0.0012 max mem: 3524 Epoch: [0] [ 142/5005] eta: 4:21:40 lr: 0.1 img/s: 11.942600516364719 loss: 6.7662 (6.9037) acc1: 0.0000 (0.3278) acc5: 0.0000 (1.2238) time: 2.4840 data: 0.0012 max mem: 3524 Epoch: [0] [ 143/5005] eta: 4:20:50 lr: 0.1 img/s: 17.172048620928862 loss: 6.7662 (6.9027) acc1: 0.0000 (0.3472) acc5: 0.0000 (1.2370) time: 2.4758 data: 0.0012 max mem: 3524 Epoch: [0] [ 144/5005] eta: 4:20:24 lr: 0.1 img/s: 12.601742153212438 loss: 6.7595 (6.9007) acc1: 0.0000 (0.3448) acc5: 3.1250 (1.2500) time: 2.5199 data: 0.0012 max mem: 3524 Epoch: [0] [ 145/5005] eta: 4:21:34 lr: 0.1 img/s: 5.942613448190975 loss: 6.7595 (6.8996) acc1: 0.0000 (0.3425) acc5: 0.0000 (1.2414) time: 2.6792 data: 0.0012 max mem: 3524 Epoch: [0] [ 146/5005] eta: 4:20:56 lr: 0.1 img/s: 14.657661367781058 loss: 6.7720 (6.8995) acc1: 0.0000 (0.3401) acc5: 0.0000 (1.2330) time: 2.6850 data: 0.0003 max mem: 3524 Epoch: [0] [ 147/5005] eta: 4:21:12 lr: 0.1 img/s: 8.421684692987 loss: 6.7720 (6.8987) acc1: 0.0000 (0.3378) acc5: 0.0000 (1.2247) time: 2.6613 data: 0.0003 max mem: 3524 Epoch: [0] [ 148/5005] eta: 4:20:42 lr: 0.1 img/s: 13.33972415861759 loss: 6.7720 (6.8975) acc1: 0.0000 (0.3565) acc5: 0.0000 (1.2374) time: 2.6350 data: 0.0013 max mem: 3524 Epoch: [0] [ 149/5005] eta: 4:20:05 lr: 0.1 img/s: 14.596599050128345 loss: 6.7720 (6.8967) acc1: 0.0000 (0.3542) acc5: 0.0000 (1.2500) time: 2.6379 data: 0.0013 max mem: 3524 Epoch: [0] [ 150/5005] eta: 4:19:08 lr: 0.1 img/s: 21.06738648383888 loss: 6.7595 (6.8957) acc1: 0.0000 (0.3518) acc5: 0.0000 (1.2417) time: 2.6083 data: 0.0013 max mem: 3524 Epoch: [0] [ 151/5005] eta: 4:18:18 lr: 0.1 img/s: 18.27867179818218 loss: 6.7419 (6.8944) acc1: 0.0000 (0.3701) acc5: 3.1250 (1.2541) time: 2.5944 data: 0.0013 max mem: 3524 Epoch: [0] [ 152/5005] eta: 4:18:06 lr: 0.1 img/s: 11.016886689377717 loss: 6.7595 (6.8946) acc1: 0.0000 (0.3676) acc5: 0.0000 (1.2459) time: 2.6207 data: 0.0021 max mem: 3524 Epoch: [0] [ 153/5005] eta: 4:17:21 lr: 0.1 img/s: 17.37321050119752 loss: 6.7595 (6.8936) acc1: 0.0000 (0.3653) acc5: 0.0000 (1.2581) time: 2.6044 data: 0.0021 max mem: 3524 Epoch: [0] [ 154/5005] eta: 4:18:26 lr: 0.1 img/s: 5.9690475054300265 loss: 6.7595 (6.8945) acc1: 0.0000 (0.3629) acc5: 0.0000 (1.2500) time: 2.6347 data: 0.0021 max mem: 3524 Epoch: [0] [ 155/5005] eta: 4:17:50 lr: 0.1 img/s: 14.933431837943598 loss: 6.7595 (6.8944) acc1: 0.0000 (0.3606) acc5: 0.0000 (1.2420) time: 2.6018 data: 0.0021 max mem: 3524 Epoch: [0] [ 156/5005] eta: 4:17:07 lr: 0.1 img/s: 16.909946396754727 loss: 6.7595 (6.8934) acc1: 0.0000 (0.3583) acc5: 0.0000 (1.2540) time: 2.5906 data: 0.0021 max mem: 3524 Epoch: [0] [ 157/5005] eta: 4:16:23 lr: 0.1 img/s: 17.000873111325177 loss: 6.7595 (6.8927) acc1: 0.0000 (0.3560) acc5: 0.0000 (1.2856) time: 2.5682 data: 0.0021 max mem: 3524 Epoch: [0] [ 158/5005] eta: 4:15:50 lr: 0.1 img/s: 14.701519489427367 loss: 6.7595 (6.8913) acc1: 0.0000 (0.3734) acc5: 0.0000 (1.2972) time: 2.5890 data: 0.0021 max mem: 3524 Epoch: [0] [ 159/5005] eta: 4:15:09 lr: 0.1 img/s: 16.606601848434416 loss: 6.7595 (6.8911) acc1: 0.0000 (0.3711) acc5: 3.1250 (1.3086) time: 2.5679 data: 0.0021 max mem: 3524 Epoch: [0] [ 160/5005] eta: 4:16:34 lr: 0.1 img/s: 5.2511359754639235 loss: 6.7595 (6.8898) acc1: 0.0000 (0.3688) acc5: 0.0000 (1.3005) time: 2.7462 data: 0.0022 max mem: 3524 Epoch: [0] [ 161/5005] eta: 4:15:45 lr: 0.1 img/s: 19.67864818799277 loss: 6.7419 (6.8883) acc1: 0.0000 (0.3665) acc5: 0.0000 (1.2924) time: 2.7102 data: 0.0022 max mem: 3524 Epoch: [0] [ 162/5005] eta: 4:14:56 lr: 0.1 img/s: 19.951552968241383 loss: 6.7407 (6.8874) acc1: 0.0000 (0.3643) acc5: 0.0000 (1.2845) time: 2.6570 data: 0.0028 max mem: 3524 Epoch: [0] [ 163/5005] eta: 4:14:00 lr: 0.1 img/s: 23.298667342967846 loss: 6.7407 (6.8870) acc1: 0.0000 (0.3620) acc5: 0.0000 (1.2957) time: 2.6326 data: 0.0029 max mem: 3524 Epoch: [0] [ 164/5005] eta: 4:13:52 lr: 0.1 img/s: 10.765931594509427 loss: 6.7407 (6.8860) acc1: 0.0000 (0.3598) acc5: 0.0000 (1.2879) time: 2.6543 data: 0.0030 max mem: 3524 Epoch: [0] [ 165/5005] eta: 4:13:08 lr: 0.1 img/s: 18.20622733508788 loss: 6.7419 (6.8855) acc1: 0.0000 (0.3577) acc5: 0.0000 (1.2801) time: 2.4730 data: 0.0030 max mem: 3524 Epoch: [0] [ 166/5005] eta: 4:12:44 lr: 0.1 img/s: 13.203945982728332 loss: 6.7419 (6.8852) acc1: 0.0000 (0.3555) acc5: 0.0000 (1.2725) time: 2.4850 data: 0.0030 max mem: 3524 Epoch: [0] [ 167/5005] eta: 4:12:22 lr: 0.1 img/s: 13.01977551323103 loss: 6.7407 (6.8834) acc1: 0.0000 (0.3534) acc5: 0.0000 (1.2649) time: 2.4180 data: 0.0031 max mem: 3524 Epoch: [0] [ 168/5005] eta: 4:11:44 lr: 0.1 img/s: 16.730500260647208 loss: 6.7419 (6.8833) acc1: 0.0000 (0.3513) acc5: 0.0000 (1.2574) time: 2.3927 data: 0.0021 max mem: 3524 Epoch: [0] [ 169/5005] eta: 4:11:11 lr: 0.1 img/s: 15.345571963798838 loss: 6.7407 (6.8809) acc1: 0.0000 (0.3493) acc5: 0.0000 (1.2500) time: 2.3874 data: 0.0021 max mem: 3524 Epoch: [0] [ 170/5005] eta: 4:10:42 lr: 0.1 img/s: 14.6418759285645 loss: 6.7363 (6.8799) acc1: 0.0000 (0.3472) acc5: 0.0000 (1.2427) time: 2.4207 data: 0.0021 max mem: 3524 Epoch: [0] [ 171/5005] eta: 4:10:20 lr: 0.1 img/s: 13.035825038823077 loss: 6.7407 (6.8805) acc1: 0.0000 (0.3452) acc5: 0.0000 (1.2355) time: 2.4559 data: 0.0021 max mem: 3524 Epoch: [0] [ 172/5005] eta: 4:09:49 lr: 0.1 img/s: 15.15493170849963 loss: 6.7363 (6.8793) acc1: 0.0000 (0.3432) acc5: 0.0000 (1.2645) time: 2.4159 data: 0.0018 max mem: 3524 Epoch: [0] [ 173/5005] eta: 4:09:33 lr: 0.1 img/s: 12.117617817659719 loss: 6.7238 (6.8768) acc1: 0.0000 (0.3412) acc5: 0.0000 (1.2572) time: 2.4559 data: 0.0018 max mem: 3524 Epoch: [0] [ 174/5005] eta: 4:09:15 lr: 0.1 img/s: 12.633309033270065 loss: 6.7238 (6.8767) acc1: 0.0000 (0.3393) acc5: 0.0000 (1.2679) time: 2.3145 data: 0.0018 max mem: 3524 Epoch: [0] [ 175/5005] eta: 4:08:57 lr: 0.1 img/s: 12.53977165868335 loss: 6.7238 (6.8765) acc1: 0.0000 (0.3374) acc5: 0.0000 (1.2607) time: 2.3349 data: 0.0018 max mem: 3524 Epoch: [0] [ 176/5005] eta: 4:08:25 lr: 0.1 img/s: 15.80938706197321 loss: 6.7158 (6.8748) acc1: 0.0000 (0.3355) acc5: 0.0000 (1.2535) time: 2.3415 data: 0.0018 max mem: 3524 Epoch: [0] [ 177/5005] eta: 4:09:03 lr: 0.1 img/s: 6.902848401481643 loss: 6.7141 (6.8728) acc1: 0.0000 (0.3336) acc5: 0.0000 (1.2640) time: 2.4792 data: 0.0018 max mem: 3524 Epoch: [0] [ 178/5005] eta: 4:08:27 lr: 0.1 img/s: 17.201036642083455 loss: 6.7158 (6.8720) acc1: 0.0000 (0.3317) acc5: 0.0000 (1.2744) time: 2.4634 data: 0.0018 max mem: 3524 Epoch: [0] [ 179/5005] eta: 4:08:04 lr: 0.1 img/s: 13.579570120716753 loss: 6.7141 (6.8704) acc1: 0.0000 (0.3299) acc5: 0.0000 (1.2674) time: 2.4848 data: 0.0018 max mem: 3524 Epoch: [0] [ 180/5005] eta: 4:07:32 lr: 0.1 img/s: 15.990265097159005 loss: 6.7141 (6.8681) acc1: 0.0000 (0.3280) acc5: 0.0000 (1.3122) time: 2.2801 data: 0.0017 max mem: 3524 Epoch: [0] [ 181/5005] eta: 4:07:05 lr: 0.1 img/s: 14.942883640148963 loss: 6.7158 (6.8673) acc1: 0.0000 (0.3262) acc5: 0.0000 (1.3049) time: 2.3059 data: 0.0017 max mem: 3524 Epoch: [0] [ 182/5005] eta: 4:06:58 lr: 0.1 img/s: 10.90749133529718 loss: 6.7158 (6.8670) acc1: 0.0000 (0.3245) acc5: 0.0000 (1.3149) time: 2.3718 data: 0.0011 max mem: 3524 Epoch: [0] [ 183/5005] eta: 4:06:40 lr: 0.1 img/s: 12.677312817563939 loss: 6.7158 (6.8667) acc1: 0.0000 (0.3227) acc5: 0.0000 (1.3077) time: 2.4292 data: 0.0010 max mem: 3524 Epoch: [0] [ 184/5005] eta: 4:06:05 lr: 0.1 img/s: 17.52452070825749 loss: 6.7180 (6.8669) acc1: 0.0000 (0.3209) acc5: 0.0000 (1.3176) time: 2.3719 data: 0.0010 max mem: 3524 Epoch: [0] [ 185/5005] eta: 4:05:33 lr: 0.1 img/s: 16.324643818265276 loss: 6.7180 (6.8666) acc1: 0.0000 (0.3192) acc5: 0.0000 (1.3441) time: 2.3820 data: 0.0010 max mem: 3524 Epoch: [0] [ 186/5005] eta: 4:06:11 lr: 0.1 img/s: 6.934142919330818 loss: 6.7180 (6.8659) acc1: 0.0000 (0.3175) acc5: 0.0000 (1.3369) time: 2.4920 data: 0.0014 max mem: 3524 Epoch: [0] [ 187/5005] eta: 4:06:58 lr: 0.1 img/s: 6.493881027105324 loss: 6.7180 (6.8645) acc1: 0.0000 (0.3324) acc5: 0.0000 (1.3630) time: 2.6207 data: 0.0066 max mem: 3524 Epoch: [0] [ 188/5005] eta: 4:06:12 lr: 0.1 img/s: 23.21157478552882 loss: 6.7158 (6.8631) acc1: 0.0000 (0.3307) acc5: 0.0000 (1.3558) time: 2.5940 data: 0.0066 max mem: 3524 Epoch: [0] [ 189/5005] eta: 4:05:33 lr: 0.1 img/s: 19.13696392388829 loss: 6.7158 (6.8613) acc1: 0.0000 (0.3289) acc5: 0.0000 (1.3816) time: 2.5733 data: 0.0066 max mem: 3524 Epoch: [0] [ 190/5005] eta: 4:05:01 lr: 0.1 img/s: 16.920159623519446 loss: 6.7180 (6.8617) acc1: 0.0000 (0.3272) acc5: 3.1250 (1.3907) time: 2.5586 data: 0.0066 max mem: 3524 Epoch: [0] [ 191/5005] eta: 4:04:26 lr: 0.1 img/s: 17.78135583609822 loss: 6.7180 (6.8612) acc1: 0.0000 (0.3255) acc5: 3.1250 (1.3835) time: 2.5258 data: 0.0066 max mem: 3524 Epoch: [0] [ 192/5005] eta: 4:04:04 lr: 0.1 img/s: 14.131949918857313 loss: 6.7180 (6.8604) acc1: 0.0000 (0.3238) acc5: 3.1250 (1.3925) time: 2.5335 data: 0.0066 max mem: 3524 Epoch: [0] [ 193/5005] eta: 4:03:39 lr: 0.1 img/s: 14.805268920609015 loss: 6.7310 (6.8599) acc1: 0.0000 (0.3222) acc5: 3.1250 (1.4014) time: 2.5096 data: 0.0067 max mem: 3524 Epoch: [0] [ 194/5005] eta: 4:03:12 lr: 0.1 img/s: 16.495734842612514 loss: 6.7209 (6.8591) acc1: 0.0000 (0.3205) acc5: 3.1250 (1.3942) time: 2.4858 data: 0.0125 max mem: 3524 Epoch: [0] [ 195/5005] eta: 4:05:16 lr: 0.1 img/s: 3.898160842604837 loss: 6.7180 (6.8575) acc1: 0.0000 (0.3189) acc5: 3.1250 (1.3871) time: 2.7686 data: 0.0125 max mem: 3524 Epoch: [0] [ 196/5005] eta: 4:04:28 lr: 0.1 img/s: 26.413704816046547 loss: 6.7209 (6.8585) acc1: 0.0000 (0.3331) acc5: 3.1250 (1.3959) time: 2.7295 data: 0.0140 max mem: 3524 Epoch: [0] [ 197/5005] eta: 4:03:39 lr: 0.1 img/s: 28.183318561093554 loss: 6.7310 (6.8584) acc1: 0.0000 (0.3314) acc5: 3.1250 (1.3889) time: 2.5545 data: 0.0140 max mem: 3524 Epoch: [0] [ 198/5005] eta: 4:03:05 lr: 0.1 img/s: 17.967086289523706 loss: 6.7428 (6.8585) acc1: 0.0000 (0.3298) acc5: 0.0000 (1.3819) time: 2.5505 data: 0.0140 max mem: 3524 Epoch: [0] [ 199/5005] eta: 4:02:37 lr: 0.1 img/s: 16.349010881497264 loss: 6.7617 (6.8585) acc1: 0.0000 (0.3281) acc5: 3.1250 (1.3906) time: 2.5305 data: 0.0140 max mem: 3524 Epoch: [0] [ 200/5005] eta: 4:02:14 lr: 0.1 img/s: 14.747191371725172 loss: 6.7617 (6.8572) acc1: 0.0000 (0.3265) acc5: 3.1250 (1.3993) time: 2.5402 data: 0.0152 max mem: 3524 Epoch: [0] [ 201/5005] eta: 4:01:46 lr: 0.1 img/s: 15.943896301861008 loss: 6.7617 (6.8567) acc1: 0.0000 (0.3249) acc5: 3.1250 (1.3923) time: 2.5335 data: 0.0152 max mem: 3524 Epoch: [0] [ 202/5005] eta: 4:01:25 lr: 0.1 img/s: 14.331107357848747 loss: 6.7617 (6.8564) acc1: 0.0000 (0.3233) acc5: 0.0000 (1.3855) time: 2.4986 data: 0.0154 max mem: 3524 Epoch: [0] [ 203/5005] eta: 4:01:00 lr: 0.1 img/s: 15.289921388059465 loss: 6.7515 (6.8558) acc1: 0.0000 (0.3217) acc5: 0.0000 (1.3787) time: 2.4771 data: 0.0154 max mem: 3524 Epoch: [0] [ 204/5005] eta: 4:00:41 lr: 0.1 img/s: 13.701160004226178 loss: 6.7515 (6.8558) acc1: 0.0000 (0.3354) acc5: 0.0000 (1.4024) time: 2.5025 data: 0.0154 max mem: 3524 Epoch: [0] [ 205/5005] eta: 4:00:31 lr: 0.1 img/s: 11.964821296637028 loss: 6.7428 (6.8552) acc1: 0.0000 (0.3337) acc5: 0.0000 (1.4108) time: 2.5382 data: 0.0153 max mem: 3524

I tried to use a set of small number of num_worker i.e. 4,8 and the the problem was remained. I also reproduced the sew_resnet18 on cifar10dvs and dvsgesture using one V100 GPU , that's really fast. Do you think what cost the low training effectiveness on imagenet?

fangwei123456 commented 2 years ago

Hi, I notice that your backend=torch. You can use cupy backend to get the fastest speed. By the way, is your imagenet dataset stored in a SSD?

fangwei123456 commented 2 years ago

--print-freq 1 is too small. I suggest you can set it to 1024. Too much printing will slow down the training.

iminfine commented 2 years ago

Of course fast SSDs are used in the cluster. I tried to set --print-freq to 4096, however the problem was remained. I set it to 1 to see the training speed. Maybe I should use the lower version of pytorch and spikingjelly. What is your version of pytorch by the way?

fangwei123456 commented 2 years ago

I always use the latest version of SJ. My pytorch is 1.10.

fangwei123456 commented 2 years ago

(surrogate_function): Sigmoid(alpha=4.0, spiking=True)

The surrogate function is ATan in the origin paper.

iminfine commented 2 years ago

Thanks, I will try. But I don't change the code. It seems the surrogate_function is set to Sigmoid as default for MultiStepIFNode in SJ 0.0.0.0.11.

fangwei123456 commented 2 years ago

You can refer to this tutorial: https://spikingjelly.readthedocs.io/zh_CN/latest/clock_driven_en/16_train_large_scale_snn.html

And you can change parameters like this:

from spikingjelly.clock_driven import neuron, surrogate, functional
from spikingjelly.clock_driven.model import sew_resnet

net = sew_resnet.multi_step_sew_resnet18(pretrained=False, progress=True, T=4, cnf='ADD', multi_step_neuron=neuron.MultiStepIFNode, v_threshold=1., surrogate_function=surrogate.ATan(), detach_reset=True, backend='cupy')
print(net)
iminfine commented 2 years ago

Thanks!

iminfine commented 2 years ago

I tested the new code on one and 8 GPUs:

Test on 1 GPU:

Epoch: [0] [ 2028/40037] eta: 3:37:09 lr: 0.1 img/s: 129.64251403711614 loss: 6.6957 (6.7950) acc1: 0.0000 (0.3943) acc5: 3.1250 (1.7327) time: 0.2519 data: 0.0028 max mem: 3917 Epoch: [0] [ 2029/40037] eta: 3:37:06 lr: 0.1 img/s: 166.240256040586 loss: 6.6957 (6.7950) acc1: 0.0000 (0.3941) acc5: 3.1250 (1.7318) time: 0.2523 data: 0.0028 max mem: 3917 Epoch: [0] [ 2030/40037] eta: 3:37:03 lr: 0.1 img/s: 186.46245750620648 loss: 6.6437 (6.7949) acc1: 0.0000 (0.3954) acc5: 3.1250 (1.7341) time: 0.2555 data: 0.0028 max mem: 3917 Epoch: [0] [ 2031/40037] eta: 3:37:03 lr: 0.1 img/s: 92.10039950648392 loss: 6.6437 (6.7947) acc1: 0.0000 (0.3952) acc5: 3.1250 (1.7347) time: 0.2581 data: 0.0023 max mem: 3917 Epoch: [0] [ 2032/40037] eta: 3:37:00 lr: 0.1 img/s: 146.32829683059012 loss: 6.6437 (6.7946) acc1: 0.0000 (0.3950) acc5: 3.1250 (1.7354) time: 0.2524 data: 0.0023 max mem: 3917 Epoch: [0] [ 2033/40037] eta: 3:36:59 lr: 0.1 img/s: 97.93375969996242 loss: 6.6957 (6.7946) acc1: 0.0000 (0.3949) acc5: 3.1250 (1.7346) time: 0.2539 data: 0.0023 max mem: 3917 Epoch: [0] [ 2034/40037] eta: 3:36:57 lr: 0.1 img/s: 130.6604959964954 loss: 6.6957 (6.7946) acc1: 0.0000 (0.3947) acc5: 3.1250 (1.7337) time: 0.2526 data: 0.0023 max mem: 3917 Epoch: [0] [ 2035/40037] eta: 3:36:53 lr: 0.1 img/s: 218.68219507068758 loss: 6.6437 (6.7945) acc1: 0.0000 (0.3945) acc5: 3.1250 (1.7329) time: 0.2431 data: 0.0023 max mem: 3917 Epoch: [0] [ 2036/40037] eta: 3:36:51 lr: 0.1 img/s: 129.54403819017486 loss: 6.6195 (6.7944) acc1: 0.0000 (0.3958) acc5: 3.1250 (1.7351) time: 0.2428 data: 0.0023 max mem: 3917 Epoch: [0] [ 2037/40037] eta: 3:36:50 lr: 0.1 img/s: 107.71793186416492 loss: 6.6195 (6.7942) acc1: 0.0000 (0.3956) acc5: 3.1250 (1.7342) time: 0.2416 data: 0.0020 max mem: 3917

Test on 8 GPUs:

Epoch: [0] [ 0/5005] eta: 6 days, 21:25:21 lr: 0.1 img/s: 0.7162064827826533 loss: 6.9572 (6.9572) acc1: 0.0000 (0.0000) acc5: 0.0000 (0.0000) time: 118.9853 data: 74.3054 max mem: 3828 Epoch: [0] [ 100/5005] eta: 4:38:45 lr: 0.1 img/s: 24.51477441075965 loss: 6.9122 (6.9480) acc1: 0.0000 (0.2475) acc5: 0.0000 (0.8973) time: 2.5794 data: 0.0025 max mem: 3917 Epoch: [0] [ 200/5005] eta: 3:52:50 lr: 0.1 img/s: 20.61706434270436 loss: 6.7006 (6.8448) acc1: 0.0000 (0.3109) acc5: 0.0000 (1.6014) time: 2.4168 data: 0.0322 max mem: 3917 Epoch: [0] [ 300/5005] eta: 3:33:51 lr: 0.1 img/s: 26.37344423126652 loss: 6.5842 (6.7693) acc1: 0.0000 (0.4672) acc5: 3.1250 (2.1595) time: 2.4160 data: 0.0009 max mem: 3917 Epoch: [0] [ 400/5005] eta: 3:22:46 lr: 0.1 img/s: 15.70835842300954 loss: 6.3666 (6.6991) acc1: 0.0000 (0.5455) acc5: 3.1250 (2.5171) time: 2.1200 data: 0.0013 max mem: 3917 Epoch: [0] [ 500/5005] eta: 3:16:07 lr: 0.1 img/s: 16.109199706568038 loss: 6.3906 (6.6299) acc1: 0.0000 (0.6924) acc5: 6.2500 (2.9628) time: 2.4670 data: 0.0072 max mem: 3917 Epoch: [0] [ 600/5005] eta: 3:13:14 lr: 0.1 img/s: 14.379072492342976 loss: 6.1937 (6.5670) acc1: 0.0000 (0.8215) acc5: 3.1250 (3.3590) time: 2.6843 data: 0.0056 max mem: 3917 Epoch: [0] [ 700/5005] eta: 3:06:40 lr: 0.1 img/s: 33.17576383162582 loss: 6.0886 (6.5084) acc1: 0.0000 (0.9718) acc5: 3.1250 (3.7268) time: 2.1434 data: 0.0015 max mem: 3917 Epoch: [0] [ 800/5005] eta: 2:59:55 lr: 0.1 img/s: 15.578086917725486 loss: 6.0426 (6.4529) acc1: 3.1250 (1.1314) acc5: 6.2500 (4.2213) time: 2.3102 data: 0.0031 max mem: 3917 Epoch: [0] [ 900/5005] eta: 2:54:35 lr: 0.1 img/s: 23.108754857448833 loss: 6.0389 (6.4068) acc1: 0.0000 (1.2555) acc5: 6.2500 (4.5817) time: 2.3846 data: 0.0021 max mem: 3917 Epoch: [0] [1000/5005] eta: 2:49:01 lr: 0.1 img/s: 23.056101381225055 loss: 5.7338 (6.3530) acc1: 3.1250 (1.3736) acc5: 9.3750 (5.0387) time: 2.5856 data: 0.0029 max mem: 3917

See the difference of FPS.

iminfine commented 2 years ago

The num_worker of these two case are 16the training speed of them are almost same. The ddp seems not working. Don't know what happens here. My pytorch version is 1.11 and I will test the older version.

fangwei123456 commented 2 years ago

This function disables printing when not on master: https://github.com/fangwei123456/Spike-Element-Wise-ResNet/blob/1e2e9a5685d1a6e21fe809603bd4eb2576c1c5dc/imagenet/utils.py#L187

You can comment these codes, enable printing on all sub-processes and check the error.

It the error is caused by cupy (https://github.com/fangwei123456/spikingjelly/blob/master/bugs.md), you can install the latest SJ from source codes from github to avoid this problem.