Open hewiew opened 2 years ago
ActNN is a lossy algorithm, so it is possible that it does not work with 2 bits for some models.
Please try using more warmup iterations, and
actnn.set_optimization_level("L2")
ActNN is a lossy algorithm, so it is possible that it does not work with 2 bits for some models.
Please try using more warmup iterations, and
actnn.set_optimization_level("L2")
I've tried set_optimization_level "L0", "L1", "L2", "L3", but the performance seems no much difference, so I'm confused
as for warmup iterations, i used 4 epoch warmup, is that enough for training?
ActNN L0 does exactly the same thing with full precision training. Is the 2% accuracy loss within random error?
Could you print(model) before the training loop, and check if the model is correctly converted?
ActNN converts nn.Modules with its own modules, and I noticed there are additional model converters after actnn.QModules. If these converters are looking for the original nn.Modules (e.g., nn.BatchNorm), they may not found the corresponding module. You can try moving actnn.QModule after these converters.
if opts.separable_conv and 'plus' in opts.model: network.convert_to_separable_conv(model.model.classifier) utils.set_bn_momentum(model.model.backbone, momentum=0.01)
ActNN L0 does exactly the same thing with full precision training. Is the 2% accuracy loss within random error?
I'm afraid not, when I don't use the actnn.module wrapper, I can always get a model with 2% of iou higher than that use it. Incidentally, my model is a 2-class segmentation model, Does this have an impact?
Could you print(model) before the training loop, and check if the model is correctly converted?
ActNN converts nn.Modules with its own modules, and I noticed there are additional model converters after actnn.QModules. If these converters are looking for the original nn.Modules (e.g., nn.BatchNorm), they may not found the corresponding module. You can try moving actnn.QModule after these converters.
if opts.separable_conv and 'plus' in opts.model: network.convert_to_separable_conv(model.model.classifier) utils.set_bn_momentum(model.model.backbone, momentum=0.01)
the codes you list are always skiped during my training, and I print my model, seem all layers are converted correctly:
DataParallel( (module): QModule( (model): DeepLabV3( (backbone): IntermediateLayerGetter( (conv1): QConv2d(4, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False) (bn1): QBatchNorm2d(64, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() (maxpool): QMaxPool2d(kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), dilation=(1, 1), ceil_mode=False) (layer1): Sequential( (0): Bottleneck( (conv1): QConv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(64, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): QBatchNorm2d(64, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() (downsample): Sequential( (0): QConv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) ) ) (1): Bottleneck( (conv1): QConv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(64, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): QBatchNorm2d(64, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() ) (2): Bottleneck( (conv1): QConv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(64, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): QBatchNorm2d(64, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() ) ) (layer2): Sequential( (0): Bottleneck( (conv1): QConv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(128, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn2): QBatchNorm2d(128, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(512, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() (downsample): Sequential( (0): QConv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): QBatchNorm2d(512, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) ) ) (1): Bottleneck( (conv1): QConv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(128, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): QBatchNorm2d(128, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(512, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() ) (2): Bottleneck( (conv1): QConv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(128, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): QBatchNorm2d(128, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(512, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() ) (3): Bottleneck( (conv1): QConv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(128, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): QBatchNorm2d(128, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(512, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() ) ) (layer3): Sequential( (0): Bottleneck( (conv1): QConv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (bn2): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(1024, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() (downsample): Sequential( (0): QConv2d(512, 1024, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): QBatchNorm2d(1024, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) ) ) (1): Bottleneck( (conv1): QConv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(1024, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() ) (2): Bottleneck( (conv1): QConv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(1024, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() ) (3): Bottleneck( (conv1): QConv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(1024, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() ) (4): Bottleneck( (conv1): QConv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(1024, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() ) (5): Bottleneck( (conv1): QConv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): QBatchNorm2d(256, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(1024, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() ) ) (layer4): Sequential( (0): Bottleneck( (conv1): QConv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(512, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (bn2): QBatchNorm2d(512, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(2048, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() (downsample): Sequential( (0): QConv2d(1024, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): QBatchNorm2d(2048, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) ) ) (1): Bottleneck( (conv1): QConv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(512, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False) (bn2): QBatchNorm2d(512, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(2048, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() ) (2): Bottleneck( (conv1): QConv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn1): QBatchNorm2d(512, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv2): QConv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False) (bn2): QBatchNorm2d(512, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (conv3): QConv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False) (bn3): QBatchNorm2d(2048, eps=1e-05, momentum=0.01, affine=True, track_running_stats=True) (relu): QReLU() ) ) ) (classifier): DeepLabHeadV3Plus( (project): Sequential( (0): QConv2d(256, 48, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): QBatchNorm2d(48, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): QReLU() ) (aspp): ASPP( (convs): ModuleList( (0): Sequential( (0): QConv2d(2048, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): QBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): QReLU() ) (1): ASPPConv( (0): QConv2d(2048, 256, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False) (1): QBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): QReLU() ) (2): ASPPConv( (0): QConv2d(2048, 256, kernel_size=(3, 3), stride=(1, 1), padding=(5, 5), dilation=(5, 5), bias=False) (1): QBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): QReLU() ) (3): ASPPConv( (0): QConv2d(2048, 256, kernel_size=(3, 3), stride=(1, 1), padding=(8, 8), dilation=(8, 8), bias=False) (1): QBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): QReLU() ) (4): ASPPPooling( (0): AdaptiveAvgPool2d(output_size=1) (1): QConv2d(2048, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (2): QBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (3): QReLU() ) ) (project): Sequential( (0): QConv2d(1280, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): QBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): QReLU() (3): QDropout(p=0.1, inplace=False) ) ) (classifier): Sequential( (0): QConv2d(304, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): QBatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): QReLU() (3): QConv2d(256, 2, kernel_size=(1, 1), stride=(1, 1)) ) ) ) ) )
That's strange. ActNN L0 and full precision training should have identical behavior.
Could you try to debug by the following:
If you spot a bug in our implementation, please create a PR for us.
That's strange. ActNN L0 and full precision training should have identical behavior.
Could you try to debug by the following:
1. prepare a model checkpoint, and a batch of (data, label) 2. compute the gradient with full-precision training 3. compute the gradient with actnn 4. check if the gradients are identical, if not, replace actnn layers with nn.module layers one by one, and observe which layer is causing the gradient discrepancy.
If you spot a bug in our implementation, please create a PR for us.
Thanks for your advise last week. I've followed the steps you list but it seems we are getting into a trouble:
I restored my model (only a nn.Conv2D layer) from a fixed checkpoint, and I found that the gradient with actnn(set level: "L0") and full-precision training are totally different.
I've sent an email to your Gmail mailbox with my experimental code and printed results with details. You may check that during your free time.
Here is my codes (part):
import actnn from actnn import QModule, QScheme
actnn.set_optimization_level("L3")
def main():
after_scheduler=scheduler)
if name == 'main': main()