there exist some bugs:

[adienxy]

base your advice: import torch from taylor_shift import TaylorShiftTransformer

max_seq_len = 4096 input_dim = 256 model = TaylorShiftTransformer(max_seq_len, input_dim, num_classes=1000, num_heads=8, depth=6, embed_dim=256)

bs = 32 seq_len = 1024 # or any other sequence length <= max_seq_len x = torch.randn(bs, seq_len, input_dim) pred = model(x) # (bs, num_classes)

Traceback (most recent call last): File "f:\1aA_Wxy_WorkStation\5.27\TaylorShift-main\test.py", line 6, in model = TaylorShiftTransformer(max_seq_len, input_dim, num_classes=1000, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "f:\1aA_Wxy_WorkStation\5.27\TaylorShift-main\taylor_shift.py", line 155, in init super().init(normalize_input=normalize_input, **kwargs) File "f:\1aA_Wxy_WorkStation\5.27\TaylorShift-main\taylor_shift.py", line 116, in init super().init(img_size=img_size, patch_size=patch_size, in_chans=in_chans, embed_dim=embed_dim, File "f:\1aA_Wxy_WorkStation\5.27\TaylorShift-main\vit.py", line 91, in init super(TimmViT, self).init(img_size=img_size, patch_size=patch_size, in_chans=in_chans, TypeError: VisionTransformer.init() got an unexpected keyword argument 'proj_drop_rate'

[tobna]

This looks like a mismatch with the timm version. Try updating timm