search

lessw2020 / Ranger21

Ranger deep learning optimizer rewrite to use newest components
Apache License 2.0
321 stars 45 forks source link

adaptive gclipping: unable to process len of 5 - currently must be <= 4 #11

Closed HungYu-Wu closed 3 years ago

HungYu-Wu commented 3 years ago

Can Ranger21 be used in 3D?

Because using 3D, the length will be 5 not 4.

Thanks.

saruarlive commented 3 years ago

Hi, I also got the same error. Did you solve the problem? Could you kindly share if you had done that!

Alxaline commented 3 years ago

Hi,

Same error encountered here, If I'm not wrong, the unit_norm function must take a condition of ndim == 5 and an elif condition can be added: 

        elif xlen == 4:  # conv2d kernels
            dim = (1, 2, 3)
        elif xlen == 5:  # conv3d kernels
            dim = (1, 2, 3, 4)

I let this be confirmed by @lessw2020 which can update the code to take into account the 3D convolution.

saruarlive commented 3 years ago

@Alxaline Thanks for looking into it. I saw that code. But my question is: are 3d kernels (dim=1,2,3,4) supported in the remaining part of the script!

Alxaline commented 3 years ago

@saruarlive For my part I run in 3D and it does not pose any problem except for the modification mentioned above.

nestordemeure commented 3 years ago

@saruarlive 3D kernels (and arbitrary dimenssions in general) are supported in the remaining of the code. unit_norm was an oversight and it will be fixed shortly.

lessw2020 commented 3 years ago

Hi @HungYu-Wu, @saruarlive and @Alxaline, Thanks for pointing out the limitation with the agc implementation. I've extended it to handle any dimension. If you pull the newest version it will handle 3D, 4D etc. now. Please let me know if any problems and thanks for the feedback on this issue!