search

lhoyer / DAFormer

[CVPR22] Official Implementation of DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation
Other
470 stars 93 forks source link

GPU Out of memory errors with RTX 3080 #4

Closed DJayalath closed 2 years ago

DJayalath commented 2 years ago

Hi, thanks for your excellent work.

I see that your experiments were run with an RTX 2080 Ti (11GB?). I am having the following error with RTX 3080 (10GB), I wonder if this is expected or not. And if you have any tips on reducing the GPU memory usage?

Full terminal output:

Run job sim2forest_uda_warm_fdthings_rcs_croppl_a999_daformer_mitb5_s0
2021-12-23 22:42:52,516 - mmseg - INFO - Environment info:
------------------------------------------------------------
sys.platform: linux
Python: 3.8.12 (default, Oct 12 2021, 13:49:34) [GCC 7.5.0]
CUDA available: True
GPU 0: NVIDIA GeForce RTX 3080
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.2.r11.2/compiler.29618528_0
GCC: gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
PyTorch: 1.9.0+cu111
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

TorchVision: 0.10.0+cu111
OpenCV: 4.4.0
MMCV: 1.3.7
MMCV Compiler: GCC 9.3
MMCV CUDA Compiler: 11.2
MMSegmentation: 0.16.0+21c5499
------------------------------------------------------------

2021-12-23 22:42:52,516 - mmseg - INFO - Distributed training: False
2021-12-23 22:42:52,984 - mmseg - INFO - Config:
log_config = dict(
    interval=50, hooks=[dict(type='TextLoggerHook', by_epoch=False)])
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]
cudnn_benchmark = True
norm_cfg = dict(type='BN', requires_grad=True)
find_unused_parameters = True
model = dict(
    type='EncoderDecoder',
    pretrained='pretrained/mit_b5.pth',
    backbone=dict(type='mit_b5', style='pytorch'),
    decode_head=dict(
        type='DAFormerHead',
        in_channels=[64, 128, 320, 512],
        in_index=[0, 1, 2, 3],
        channels=256,
        dropout_ratio=0.1,
        num_classes=19,
        norm_cfg=dict(type='BN', requires_grad=True),
        align_corners=False,
        decoder_params=dict(
            embed_dims=256,
            embed_cfg=dict(type='mlp', act_cfg=None, norm_cfg=None),
            embed_neck_cfg=dict(type='mlp', act_cfg=None, norm_cfg=None),
            fusion_cfg=dict(
                type='aspp',
                sep=True,
                dilations=(1, 6, 12, 18),
                pool=False,
                act_cfg=dict(type='ReLU'),
                norm_cfg=dict(type='BN', requires_grad=True))),
        loss_decode=dict(
            type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0)),
    train_cfg=dict(
        work_dir=
        'work_dirs/local-basic/211223_2242_sim2forest_uda_warm_fdthings_rcs_croppl_a999_daformer_mitb5_s0_dfad9'
    ),
    test_cfg=dict(mode='whole'))
dataset_type = 'ForestRealDataset'
data_root = 'data/forest/'
img_norm_cfg = dict(
    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
crop_size = (512, 512)
sim_train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='LoadAnnotations'),
    dict(type='Resize', img_scale=(640, 480)),
    dict(type='RandomCrop', crop_size=(512, 512), cat_max_ratio=0.75),
    dict(type='RandomFlip', prob=0.5),
    dict(
        type='Normalize',
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        to_rgb=True),
    dict(type='Pad', size=(512, 512), pad_val=0, seg_pad_val=255),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_semantic_seg'])
]
forest_train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='LoadAnnotations'),
    dict(type='Resize', img_scale=(640, 480)),
    dict(type='RandomCrop', crop_size=(512, 512), cat_max_ratio=0.75),
    dict(type='RandomFlip', prob=0.5),
    dict(
        type='Normalize',
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        to_rgb=True),
    dict(type='Pad', size=(512, 512), pad_val=0, seg_pad_val=255),
    dict(type='DefaultFormatBundle'),
    dict(type='Collect', keys=['img', 'gt_semantic_seg'])
]
test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(
        type='MultiScaleFlipAug',
        img_scale=(640, 480),
        flip=False,
        transforms=[
            dict(type='Resize', keep_ratio=True),
            dict(type='RandomFlip'),
            dict(
                type='Normalize',
                mean=[123.675, 116.28, 103.53],
                std=[58.395, 57.12, 57.375],
                to_rgb=True),
            dict(type='ImageToTensor', keys=['img']),
            dict(type='Collect', keys=['img'])
        ])
]
data = dict(
    samples_per_gpu=2,
    workers_per_gpu=4,
    train=dict(
        type='UDADataset',
        source=dict(
            type='ForestSimDataset',
            data_root='data/sim/',
            img_dir='images',
            ann_dir='labels',
            pipeline=[
                dict(type='LoadImageFromFile'),
                dict(type='LoadAnnotations'),
                dict(type='Resize', img_scale=(640, 480)),
                dict(
                    type='RandomCrop',
                    crop_size=(512, 512),
                    cat_max_ratio=0.75),
                dict(type='RandomFlip', prob=0.5),
                dict(
                    type='Normalize',
                    mean=[123.675, 116.28, 103.53],
                    std=[58.395, 57.12, 57.375],
                    to_rgb=True),
                dict(type='Pad', size=(512, 512), pad_val=0, seg_pad_val=255),
                dict(type='DefaultFormatBundle'),
                dict(type='Collect', keys=['img', 'gt_semantic_seg'])
            ]),
        target=dict(
            type='ForestRealDataset',
            data_root='data/forest/',
            img_dir='images',
            ann_dir='labels',
            pipeline=[
                dict(type='LoadImageFromFile'),
                dict(type='LoadAnnotations'),
                dict(type='Resize', img_scale=(640, 480)),
                dict(
                    type='RandomCrop',
                    crop_size=(512, 512),
                    cat_max_ratio=0.75),
                dict(type='RandomFlip', prob=0.5),
                dict(
                    type='Normalize',
                    mean=[123.675, 116.28, 103.53],
                    std=[58.395, 57.12, 57.375],
                    to_rgb=True),
                dict(type='Pad', size=(512, 512), pad_val=0, seg_pad_val=255),
                dict(type='DefaultFormatBundle'),
                dict(type='Collect', keys=['img', 'gt_semantic_seg'])
            ]),
        rare_class_sampling=dict(
            min_pixels=3000, class_temp=0.01, min_crop_ratio=0.5)),
    val=dict(
        type='ForestRealDataset',
        data_root='data/forest/',
        img_dir='images',
        ann_dir='labels',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(
                type='MultiScaleFlipAug',
                img_scale=(640, 480),
                flip=False,
                transforms=[
                    dict(type='Resize', keep_ratio=True),
                    dict(type='RandomFlip'),
                    dict(
                        type='Normalize',
                        mean=[123.675, 116.28, 103.53],
                        std=[58.395, 57.12, 57.375],
                        to_rgb=True),
                    dict(type='ImageToTensor', keys=['img']),
                    dict(type='Collect', keys=['img'])
                ])
        ]),
    test=dict(
        type='ForestRealDataset',
        data_root='data/forest/',
        img_dir='images',
        ann_dir='labels',
        pipeline=[
            dict(type='LoadImageFromFile'),
            dict(
                type='MultiScaleFlipAug',
                img_scale=(640, 480),
                flip=False,
                transforms=[
                    dict(type='Resize', keep_ratio=True),
                    dict(type='RandomFlip'),
                    dict(
                        type='Normalize',
                        mean=[123.675, 116.28, 103.53],
                        std=[58.395, 57.12, 57.375],
                        to_rgb=True),
                    dict(type='ImageToTensor', keys=['img']),
                    dict(type='Collect', keys=['img'])
                ])
        ]))
uda = dict(
    type='DACS',
    alpha=0.999,
    pseudo_threshold=0.968,
    pseudo_weight_ignore_top=15,
    pseudo_weight_ignore_bottom=120,
    imnet_feature_dist_lambda=0.005,
    imnet_feature_dist_classes=[6, 7, 11, 12, 13, 14, 15, 16, 17, 18],
    imnet_feature_dist_scale_min_ratio=0.75,
    mix='class',
    blur=True,
    color_jitter_strength=0.2,
    color_jitter_probability=0.2,
    debug_img_interval=1000,
    print_grad_magnitude=False)
use_ddp_wrapper = True
optimizer = dict(
    type='AdamW',
    lr=6e-05,
    betas=(0.9, 0.999),
    weight_decay=0.01,
    paramwise_cfg=dict(
        custom_keys=dict(
            head=dict(lr_mult=10.0),
            pos_block=dict(decay_mult=0.0),
            norm=dict(decay_mult=0.0))))
optimizer_config = None
lr_config = dict(
    policy='poly',
    warmup='linear',
    warmup_iters=1500,
    warmup_ratio=1e-06,
    power=1.0,
    min_lr=0.0,
    by_epoch=False)
seed = 0
n_gpus = 1
runner = dict(type='IterBasedRunner', max_iters=40000)
checkpoint_config = dict(by_epoch=False, interval=40000, max_keep_ckpts=1)
evaluation = dict(interval=4000, metric='mIoU')
name = '211223_2242_sim2forest_uda_warm_fdthings_rcs_croppl_a999_daformer_mitb5_s0_dfad9'
exp = 'basic'
name_dataset = 'sim2forest'
name_architecture = 'daformer_sepaspp_mitb5'
name_encoder = 'mitb5'
name_decoder = 'daformer_sepaspp'
name_uda = 'dacs_a999_fd_things_rcs0.01_cpl'
name_opt = 'adamw_6e-05_pmTrue_poly10warm_1x2_40k'
work_dir = 'work_dirs/local-basic/211223_2242_sim2forest_uda_warm_fdthings_rcs_croppl_a999_daformer_mitb5_s0_dfad9'
git_rev = '21c5499f0ee1ea0ecd991003ba4598782d42ec04'
gpu_ids = range(0, 1)

2021-12-23 22:42:52,984 - mmseg - INFO - Set random seed to 0, deterministic: False
/home/hans/Documents/part3project/Models/DAFormer/mmseg/models/backbones/mix_transformer.py:214: UserWarning: DeprecationWarning: pretrained is a deprecated, please use "init_cfg" instead
  warnings.warn('DeprecationWarning: pretrained is a deprecated, '
2021-12-23 22:42:54,161 - mmseg - INFO - Load mit checkpoint.
2021-12-23 22:42:54,161 - mmseg - INFO - Use load_from_local loader
/home/hans/Documents/part3project/Models/DAFormer/env/lib/python3.8/site-packages/mmcv/cnn/utils/weight_init.py:118: UserWarning: init_cfg without layer key, if you do not define override key either, this init_cfg will do nothing
  warnings.warn(
2021-12-23 22:42:54,332 - mmseg - INFO - Load mit checkpoint.
2021-12-23 22:42:54,332 - mmseg - INFO - Use load_from_local loader
2021-12-23 22:42:54,469 - mmseg - INFO - Load mit checkpoint.
2021-12-23 22:42:54,470 - mmseg - INFO - Use load_from_local loader
2021-12-23 22:42:54,606 - mmseg - INFO - DACS(
  (model): EncoderDecoder(
    (backbone): mit_b5(
      (patch_embed1): OverlapPatchEmbed(
        (proj): Conv2d(3, 64, kernel_size=(7, 7), stride=(4, 4), padding=(3, 3))
        (norm): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
      )
      (patch_embed2): OverlapPatchEmbed(
        (proj): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
        (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
      )
      (patch_embed3): OverlapPatchEmbed(
        (proj): Conv2d(128, 320, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
        (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
      )
      (patch_embed4): OverlapPatchEmbed(
        (proj): Conv2d(320, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
        (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
      )
      (block1): ModuleList(
        (0): Block(
          (norm1): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=64, out_features=64, bias=True)
            (kv): Linear(in_features=64, out_features=128, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=64, out_features=64, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(64, 64, kernel_size=(8, 8), stride=(8, 8))
            (norm): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): Identity()
          (norm2): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=64, out_features=256, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256)
            )
            (act): GELU()
            (fc2): Linear(in_features=256, out_features=64, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (1): Block(
          (norm1): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=64, out_features=64, bias=True)
            (kv): Linear(in_features=64, out_features=128, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=64, out_features=64, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(64, 64, kernel_size=(8, 8), stride=(8, 8))
            (norm): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=64, out_features=256, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256)
            )
            (act): GELU()
            (fc2): Linear(in_features=256, out_features=64, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (2): Block(
          (norm1): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=64, out_features=64, bias=True)
            (kv): Linear(in_features=64, out_features=128, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=64, out_features=64, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(64, 64, kernel_size=(8, 8), stride=(8, 8))
            (norm): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=64, out_features=256, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256)
            )
            (act): GELU()
            (fc2): Linear(in_features=256, out_features=64, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
      )
      (norm1): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
      (block2): ModuleList(
        (0): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (1): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (2): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (3): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (4): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (5): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
      )
      (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
      (block3): ModuleList(
        (0): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (1): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (2): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (3): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (4): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (5): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (6): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (7): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (8): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (9): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (10): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (11): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (12): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (13): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (14): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (15): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (16): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (17): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (18): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (19): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (20): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (21): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (22): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (23): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (24): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (25): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (26): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (27): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (28): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (29): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (30): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (31): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (32): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (33): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (34): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (35): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (36): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (37): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (38): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (39): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
      )
      (norm3): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
      (block4): ModuleList(
        (0): Block(
          (norm1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=512, out_features=512, bias=True)
            (kv): Linear(in_features=512, out_features=1024, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=512, out_features=512, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=512, out_features=2048, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(2048, 2048, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2048)
            )
            (act): GELU()
            (fc2): Linear(in_features=2048, out_features=512, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (1): Block(
          (norm1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=512, out_features=512, bias=True)
            (kv): Linear(in_features=512, out_features=1024, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=512, out_features=512, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=512, out_features=2048, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(2048, 2048, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2048)
            )
            (act): GELU()
            (fc2): Linear(in_features=2048, out_features=512, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (2): Block(
          (norm1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=512, out_features=512, bias=True)
            (kv): Linear(in_features=512, out_features=1024, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=512, out_features=512, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=512, out_features=2048, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(2048, 2048, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2048)
            )
            (act): GELU()
            (fc2): Linear(in_features=2048, out_features=512, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
      )
      (norm4): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
    )
    (decode_head): DAFormerHead(
      input_transform=multiple_select, ignore_index=255, align_corners=False
      (loss_decode): CrossEntropyLoss()
      (conv_seg): Conv2d(256, 19, kernel_size=(1, 1), stride=(1, 1))
      (dropout): Dropout2d(p=0.1, inplace=False)
      (embed_layers): ModuleDict(
        (0): MLP(
          (proj): Linear(in_features=64, out_features=256, bias=True)
        )
        (1): MLP(
          (proj): Linear(in_features=128, out_features=256, bias=True)
        )
        (2): MLP(
          (proj): Linear(in_features=320, out_features=256, bias=True)
        )
        (3): MLP(
          (proj): Linear(in_features=512, out_features=256, bias=True)
        )
      )
      (fuse_layer): ASPPWrapper(
        (aspp_modules): DepthwiseSeparableASPPModule(
          (0): ConvModule(
            (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (activate): ReLU(inplace=True)
          )
          (1): DepthwiseSeparableConvModule(
            (depthwise_conv): ConvModule(
              (conv): Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(6, 6), dilation=(6, 6), groups=1024, bias=False)
              (bn): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
            (pointwise_conv): ConvModule(
              (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
          )
          (2): DepthwiseSeparableConvModule(
            (depthwise_conv): ConvModule(
              (conv): Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(12, 12), dilation=(12, 12), groups=1024, bias=False)
              (bn): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
            (pointwise_conv): ConvModule(
              (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
          )
          (3): DepthwiseSeparableConvModule(
            (depthwise_conv): ConvModule(
              (conv): Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(18, 18), dilation=(18, 18), groups=1024, bias=False)
              (bn): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
            (pointwise_conv): ConvModule(
              (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
          )
        )
        (bottleneck): ConvModule(
          (conv): Conv2d(1024, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activate): ReLU(inplace=True)
        )
      )
    )
    init_cfg={'type': 'Normal', 'std': 0.01, 'override': {'name': 'conv_seg'}}
  )
  (ema_model): EncoderDecoder(
    (backbone): mit_b5(
      (patch_embed1): OverlapPatchEmbed(
        (proj): Conv2d(3, 64, kernel_size=(7, 7), stride=(4, 4), padding=(3, 3))
        (norm): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
      )
      (patch_embed2): OverlapPatchEmbed(
        (proj): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
        (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
      )
      (patch_embed3): OverlapPatchEmbed(
        (proj): Conv2d(128, 320, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
        (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
      )
      (patch_embed4): OverlapPatchEmbed(
        (proj): Conv2d(320, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
        (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
      )
      (block1): ModuleList(
        (0): Block(
          (norm1): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=64, out_features=64, bias=True)
            (kv): Linear(in_features=64, out_features=128, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=64, out_features=64, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(64, 64, kernel_size=(8, 8), stride=(8, 8))
            (norm): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): Identity()
          (norm2): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=64, out_features=256, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256)
            )
            (act): GELU()
            (fc2): Linear(in_features=256, out_features=64, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (1): Block(
          (norm1): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=64, out_features=64, bias=True)
            (kv): Linear(in_features=64, out_features=128, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=64, out_features=64, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(64, 64, kernel_size=(8, 8), stride=(8, 8))
            (norm): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=64, out_features=256, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256)
            )
            (act): GELU()
            (fc2): Linear(in_features=256, out_features=64, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (2): Block(
          (norm1): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=64, out_features=64, bias=True)
            (kv): Linear(in_features=64, out_features=128, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=64, out_features=64, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(64, 64, kernel_size=(8, 8), stride=(8, 8))
            (norm): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=64, out_features=256, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256)
            )
            (act): GELU()
            (fc2): Linear(in_features=256, out_features=64, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
      )
      (norm1): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
      (block2): ModuleList(
        (0): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (1): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (2): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (3): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (4): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (5): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
      )
      (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
      (block3): ModuleList(
        (0): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (1): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (2): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (3): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (4): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (5): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (6): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (7): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (8): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (9): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (10): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (11): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (12): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (13): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (14): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (15): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (16): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (17): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (18): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (19): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (20): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (21): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (22): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (23): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (24): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (25): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (26): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (27): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (28): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (29): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (30): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (31): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (32): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (33): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (34): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (35): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (36): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (37): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (38): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (39): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
      )
      (norm3): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
      (block4): ModuleList(
        (0): Block(
          (norm1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=512, out_features=512, bias=True)
            (kv): Linear(in_features=512, out_features=1024, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=512, out_features=512, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=512, out_features=2048, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(2048, 2048, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2048)
            )
            (act): GELU()
            (fc2): Linear(in_features=2048, out_features=512, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (1): Block(
          (norm1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=512, out_features=512, bias=True)
            (kv): Linear(in_features=512, out_features=1024, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=512, out_features=512, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=512, out_features=2048, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(2048, 2048, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2048)
            )
            (act): GELU()
            (fc2): Linear(in_features=2048, out_features=512, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (2): Block(
          (norm1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=512, out_features=512, bias=True)
            (kv): Linear(in_features=512, out_features=1024, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=512, out_features=512, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=512, out_features=2048, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(2048, 2048, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2048)
            )
            (act): GELU()
            (fc2): Linear(in_features=2048, out_features=512, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
      )
      (norm4): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
    )
    (decode_head): DAFormerHead(
      input_transform=multiple_select, ignore_index=255, align_corners=False
      (loss_decode): CrossEntropyLoss()
      (conv_seg): Conv2d(256, 19, kernel_size=(1, 1), stride=(1, 1))
      (dropout): Dropout2d(p=0.1, inplace=False)
      (embed_layers): ModuleDict(
        (0): MLP(
          (proj): Linear(in_features=64, out_features=256, bias=True)
        )
        (1): MLP(
          (proj): Linear(in_features=128, out_features=256, bias=True)
        )
        (2): MLP(
          (proj): Linear(in_features=320, out_features=256, bias=True)
        )
        (3): MLP(
          (proj): Linear(in_features=512, out_features=256, bias=True)
        )
      )
      (fuse_layer): ASPPWrapper(
        (aspp_modules): DepthwiseSeparableASPPModule(
          (0): ConvModule(
            (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (activate): ReLU(inplace=True)
          )
          (1): DepthwiseSeparableConvModule(
            (depthwise_conv): ConvModule(
              (conv): Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(6, 6), dilation=(6, 6), groups=1024, bias=False)
              (bn): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
            (pointwise_conv): ConvModule(
              (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
          )
          (2): DepthwiseSeparableConvModule(
            (depthwise_conv): ConvModule(
              (conv): Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(12, 12), dilation=(12, 12), groups=1024, bias=False)
              (bn): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
            (pointwise_conv): ConvModule(
              (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
          )
          (3): DepthwiseSeparableConvModule(
            (depthwise_conv): ConvModule(
              (conv): Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(18, 18), dilation=(18, 18), groups=1024, bias=False)
              (bn): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
            (pointwise_conv): ConvModule(
              (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
          )
        )
        (bottleneck): ConvModule(
          (conv): Conv2d(1024, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activate): ReLU(inplace=True)
        )
      )
    )
    init_cfg={'type': 'Normal', 'std': 0.01, 'override': {'name': 'conv_seg'}}
  )
  (imnet_model): EncoderDecoder(
    (backbone): mit_b5(
      (patch_embed1): OverlapPatchEmbed(
        (proj): Conv2d(3, 64, kernel_size=(7, 7), stride=(4, 4), padding=(3, 3))
        (norm): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
      )
      (patch_embed2): OverlapPatchEmbed(
        (proj): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
        (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
      )
      (patch_embed3): OverlapPatchEmbed(
        (proj): Conv2d(128, 320, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
        (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
      )
      (patch_embed4): OverlapPatchEmbed(
        (proj): Conv2d(320, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
        (norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
      )
      (block1): ModuleList(
        (0): Block(
          (norm1): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=64, out_features=64, bias=True)
            (kv): Linear(in_features=64, out_features=128, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=64, out_features=64, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(64, 64, kernel_size=(8, 8), stride=(8, 8))
            (norm): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): Identity()
          (norm2): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=64, out_features=256, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256)
            )
            (act): GELU()
            (fc2): Linear(in_features=256, out_features=64, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (1): Block(
          (norm1): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=64, out_features=64, bias=True)
            (kv): Linear(in_features=64, out_features=128, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=64, out_features=64, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(64, 64, kernel_size=(8, 8), stride=(8, 8))
            (norm): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=64, out_features=256, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256)
            )
            (act): GELU()
            (fc2): Linear(in_features=256, out_features=64, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (2): Block(
          (norm1): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=64, out_features=64, bias=True)
            (kv): Linear(in_features=64, out_features=128, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=64, out_features=64, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(64, 64, kernel_size=(8, 8), stride=(8, 8))
            (norm): LayerNorm((64,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=64, out_features=256, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256)
            )
            (act): GELU()
            (fc2): Linear(in_features=256, out_features=64, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
      )
      (norm1): LayerNorm((64,), eps=1e-06, elementwise_affine=True)
      (block2): ModuleList(
        (0): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (1): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (2): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (3): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (4): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (5): Block(
          (norm1): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=128, out_features=128, bias=True)
            (kv): Linear(in_features=128, out_features=256, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=128, out_features=128, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(128, 128, kernel_size=(4, 4), stride=(4, 4))
            (norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=128, out_features=512, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=512)
            )
            (act): GELU()
            (fc2): Linear(in_features=512, out_features=128, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
      )
      (norm2): LayerNorm((128,), eps=1e-06, elementwise_affine=True)
      (block3): ModuleList(
        (0): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (1): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (2): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (3): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (4): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (5): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (6): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (7): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (8): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (9): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (10): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (11): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (12): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (13): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (14): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (15): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (16): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (17): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (18): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (19): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (20): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (21): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (22): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (23): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (24): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (25): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (26): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (27): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (28): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (29): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (30): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (31): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (32): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (33): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (34): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (35): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (36): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (37): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (38): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (39): Block(
          (norm1): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=320, out_features=320, bias=True)
            (kv): Linear(in_features=320, out_features=640, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=320, out_features=320, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
            (sr): Conv2d(320, 320, kernel_size=(2, 2), stride=(2, 2))
            (norm): LayerNorm((320,), eps=1e-05, elementwise_affine=True)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=320, out_features=1280, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(1280, 1280, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1280)
            )
            (act): GELU()
            (fc2): Linear(in_features=1280, out_features=320, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
      )
      (norm3): LayerNorm((320,), eps=1e-06, elementwise_affine=True)
      (block4): ModuleList(
        (0): Block(
          (norm1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=512, out_features=512, bias=True)
            (kv): Linear(in_features=512, out_features=1024, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=512, out_features=512, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=512, out_features=2048, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(2048, 2048, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2048)
            )
            (act): GELU()
            (fc2): Linear(in_features=2048, out_features=512, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (1): Block(
          (norm1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=512, out_features=512, bias=True)
            (kv): Linear(in_features=512, out_features=1024, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=512, out_features=512, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=512, out_features=2048, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(2048, 2048, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2048)
            )
            (act): GELU()
            (fc2): Linear(in_features=2048, out_features=512, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
        (2): Block(
          (norm1): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (attn): Attention(
            (q): Linear(in_features=512, out_features=512, bias=True)
            (kv): Linear(in_features=512, out_features=1024, bias=True)
            (attn_drop): Dropout(p=0.0, inplace=False)
            (proj): Linear(in_features=512, out_features=512, bias=True)
            (proj_drop): Dropout(p=0.0, inplace=False)
          )
          (drop_path): DropPath()
          (norm2): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
          (mlp): Mlp(
            (fc1): Linear(in_features=512, out_features=2048, bias=True)
            (dwconv): DWConv(
              (dwconv): Conv2d(2048, 2048, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=2048)
            )
            (act): GELU()
            (fc2): Linear(in_features=2048, out_features=512, bias=True)
            (drop): Dropout(p=0.0, inplace=False)
          )
        )
      )
      (norm4): LayerNorm((512,), eps=1e-06, elementwise_affine=True)
    )
    (decode_head): DAFormerHead(
      input_transform=multiple_select, ignore_index=255, align_corners=False
      (loss_decode): CrossEntropyLoss()
      (conv_seg): Conv2d(256, 19, kernel_size=(1, 1), stride=(1, 1))
      (dropout): Dropout2d(p=0.1, inplace=False)
      (embed_layers): ModuleDict(
        (0): MLP(
          (proj): Linear(in_features=64, out_features=256, bias=True)
        )
        (1): MLP(
          (proj): Linear(in_features=128, out_features=256, bias=True)
        )
        (2): MLP(
          (proj): Linear(in_features=320, out_features=256, bias=True)
        )
        (3): MLP(
          (proj): Linear(in_features=512, out_features=256, bias=True)
        )
      )
      (fuse_layer): ASPPWrapper(
        (aspp_modules): DepthwiseSeparableASPPModule(
          (0): ConvModule(
            (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (activate): ReLU(inplace=True)
          )
          (1): DepthwiseSeparableConvModule(
            (depthwise_conv): ConvModule(
              (conv): Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(6, 6), dilation=(6, 6), groups=1024, bias=False)
              (bn): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
            (pointwise_conv): ConvModule(
              (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
          )
          (2): DepthwiseSeparableConvModule(
            (depthwise_conv): ConvModule(
              (conv): Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(12, 12), dilation=(12, 12), groups=1024, bias=False)
              (bn): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
            (pointwise_conv): ConvModule(
              (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
          )
          (3): DepthwiseSeparableConvModule(
            (depthwise_conv): ConvModule(
              (conv): Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(18, 18), dilation=(18, 18), groups=1024, bias=False)
              (bn): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
            (pointwise_conv): ConvModule(
              (conv): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
              (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
              (activate): ReLU(inplace=True)
            )
          )
        )
        (bottleneck): ConvModule(
          (conv): Conv2d(1024, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
          (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          (activate): ReLU(inplace=True)
        )
      )
    )
    init_cfg={'type': 'Normal', 'std': 0.01, 'override': {'name': 'conv_seg'}}
  )
)
2021-12-23 22:42:54,654 - mmseg - INFO - Loaded 1001 images from data/sim/images
2021-12-23 22:42:54,664 - mmseg - INFO - Loaded 1001 images from data/forest/images
2021-12-23 22:42:54,666 - mmseg - INFO - RCS Classes: [6, 5, 1, 0, 2, 4, 3]
2021-12-23 22:42:54,666 - mmseg - INFO - RCS ClassProb: [3.9898804e-01 3.4685844e-01 2.4818756e-01 5.9657078e-03 1.8850398e-07
 4.6506953e-16 2.8859502e-18]
2021-12-23 22:42:58,769 - mmseg - INFO - Loaded 1001 images from data/forest/images
2021-12-23 22:42:58,769 - mmseg - INFO - Start running, host: hans@hans-3080-desktop, work_dir: /home/hans/Documents/part3project/Models/DAFormer/work_dirs/local-basic/211223_2242_sim2forest_uda_warm_fdthings_rcs_croppl_a999_daformer_mitb5_s0_dfad9
2021-12-23 22:42:58,769 - mmseg - INFO - workflow: [('train', 1)], max: 40000 iters
Traceback (most recent call last):
  File "run_experiments.py", line 101, in <module>
    train.main([config_files[i]])
  File "/home/hans/Documents/part3project/Models/DAFormer/tools/train.py", line 166, in main
    train_segmentor(
  File "/home/hans/Documents/part3project/Models/DAFormer/mmseg/apis/train.py", line 131, in train_segmentor
    runner.run(data_loaders, cfg.workflow)
  File "/home/hans/Documents/part3project/Models/DAFormer/env/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 131, in run
    iter_runner(iter_loaders[i], **kwargs)
  File "/home/hans/Documents/part3project/Models/DAFormer/env/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 60, in train
    outputs = self.model.train_step(data_batch, self.optimizer, **kwargs)
  File "/home/hans/Documents/part3project/Models/DAFormer/env/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py", line 67, in train_step
    return self.module.train_step(*inputs[0], **kwargs[0])
  File "/home/hans/Documents/part3project/Models/DAFormer/mmseg/models/uda/dacs.py", line 138, in train_step
    log_vars = self(**data_batch)
  File "/home/hans/Documents/part3project/Models/DAFormer/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/hans/Documents/part3project/Models/DAFormer/env/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 97, in new_func
    return old_func(*args, **kwargs)
  File "/home/hans/Documents/part3project/Models/DAFormer/mmseg/models/segmentors/base.py", line 109, in forward
    return self.forward_train(img, img_metas, **kwargs)
  File "/home/hans/Documents/part3project/Models/DAFormer/mmseg/models/uda/dacs.py", line 232, in forward_train
    clean_loss.backward(retain_graph=self.enable_fdist)
  File "/home/hans/Documents/part3project/Models/DAFormer/env/lib/python3.8/site-packages/torch/_tensor.py", line 255, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/hans/Documents/part3project/Models/DAFormer/env/lib/python3.8/site-packages/torch/autograd/__init__.py", line 147, in backward
    Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 624.00 MiB (GPU 0; 9.78 GiB total capacity; 6.61 GiB already allocated; 624.69 MiB free; 6.79 GiB reserved in total by PyTorch)
lhoyer commented 2 years ago

Yes, DAFormer just fits into the 11 GB memory of the RTX2080 Ti. To save GPU memory, you could try to run our method with the simplified SegFormer decoder. You can find more details on the configuration here: experiments.py#L327. This corresponds to row 8 of Table 5 in the paper.

It might also be possible to save GPU memory by using pre-computed ImageNet features for the FD over here: mmseg/models/uda/dacs.py#L159. However, I don't know if that would be sufficient to fit the original DAFormer into an RTX 3080.

DJayalath commented 2 years ago

Thanks for the swift response. I have tried the simplified SegFormer decoder with the configuration you have highlighted. Unfortunately, this was still not sufficient.

lhoyer commented 2 years ago

Further possibilities to reduce memory consumption are:

Even though these changes will most probably reduce the mIoU (as shown in the ablations in the paper), they might be a starting point to get DAFormer running your GPU. Further, switching to a less demanding window manager (e.g. xfce) can sometimes give the few extra MB of GPU memory that are missing.

DJayalath commented 2 years ago

Many thanks for your suggestions. I am (just) able to fit DAFormer in 10GiB of GPU memory by disabling FD and reducing the crop size from 512x512 to 480x480. I have not experimented with using a smaller encoder yet, but this may not be necessary.