Training time

[Mshz2]

Hey there :)

Do you have any suggestion how we could reduce training time? Is there any way like we reduce size of the network for less complex images than human portrait, or change the arguments below?

parser.add_argument('--dataset', help='Training dataset path', required=True)
parser.add_argument('--resolution', help='Specifies resolution', default=256, type=int)
parser.add_argument('--result-dir', help='Root directory for run results (default: %(default)s)', default='results', metavar='DIR')
parser.add_argument('--DiffAugment', help='Comma-separated list of DiffAugment policy', default='color,translation,cutout')
parser.add_argument('--num-gpus', help='Number of GPUs (default: %(default)s)', default=1, type=int, metavar='N')
parser.add_argument('--batch-size', help='Batch size', default=16, type=int, metavar='N')
parser.add_argument('--total-kimg', help='Training length in thousands of images (default: %(default)s)', metavar='KIMG', default=300, type=int)
parser.add_argument('--ema-kimg', help='Half-life of exponential moving average in thousands of images', metavar='KIMG', default=None, type=int)
parser.add_argument('--num-samples', help='Number of samples', default=None, type=int)
parser.add_argument('--gamma', help='R1 regularization weight', default=None, type=float)
parser.add_argument('--fmap-base', help='Number of feature maps', default=None, type=int)
parser.add_argument('--fmap-max', help='Maximum number of feature maps', default=None, type=int)
parser.add_argument('--latent-size', help='Latent size', default=None, type=int)
parser.add_argument('--mirror-augment', help='Mirror augment (default: %(default)s)', default=True, metavar='BOOL', type=_str_to_bool)
parser.add_argument('--impl', help='Custom op implementation (default: %(default)s)', default='cuda')
parser.add_argument('--metrics', help='Comma-separated list of metrics or "none" (default: %(default)s)', default='fid5k-train', type=_parse_comma_sep)
parser.add_argument('--resume', help='Resume checkpoint path', default=None)
parser.add_argument('--resume-kimg', help='Resume training length', default=0, type=int)
parser.add_argument('--num-repeats', help='Repeats of evaluation runs (default: %(default)s)', default=1, type=int, metavar='N')
parser.add_argument('--eval', help='Evalulate mode?', action='store_true')

I know that if we have more powerful GPU and if we reduce resolution, the timing get less but I'm not sure about other arguments.

Best

[zsyzzsoft]

Yes, you may reduce '--fmap-base' by 2-4x, which defaults to 16384.

[Mshz2]

Yes, you may reduce '--fmap-base' by 2-4x, which defaults to 16384.

Thank you so much for your reply. Do you think is it good idea if we reduce the number of cnn layers?

Best

[zsyzzsoft]

Currently we only have 2 layers at each level, so I don't think reducing the number of layers is practical.

[Mshz2]

Currently we only have 2 layers at each level, so I don't think reducing the number of layers is practical.

May I ask what you mean by level? Btw, the number of feature maps by --fmap-base, is the overall feature maps after all layers, right?

[zsyzzsoft]

"level" means at each resolution, before upsampling/downsampling. fmap-base is a multiplication factor of feature maps for all layers.