Autoaugment Error

[SiBensberg]

So I wanted to train with the V1 AA policy but sadly I am getting various different errors:

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: output dimensions must be positive
     [[{{node parser/cond_39/then/_352/cond_1/then/_1229/cond/else/_2474/while/body/_3784/cond/then/_4653/PartitionedCall/translate/transform/ImageProjectiveTransformV3}}]]
     [[IteratorGetNext]]
     [[IteratorGetNext/_19805]]
  (1) Invalid argument: output dimensions must be positive
     [[{{node parser/cond_39/then/_352/cond_1/then/_1229/cond/else/_2474/while/body/_3784/cond/then/_4653/PartitionedCall/translate/transform/ImageProjectiveTransformV3}}]]
     [[IteratorGetNext]]

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: Incompatible shapes: [3024,4032,3] vs. [3581,4032,3]
     [[{{node parser/cond_33/then/_287/cond_1/then/_1103/cond/else/_2242/while/body/_3550/cond/then/_4597/mul_5}}]]
     [[IteratorGetNext]]
  (1) Invalid argument: Incompatible shapes: [3024,4032,3] vs. [3581,4032,3]
     [[{{node parser/cond_33/then/_287/cond_1/then/_1103/cond/else/_2242/while/body/_3550/cond/then/_4597/mul_5}}]]
     [[IteratorGetNext]]
     [[IteratorGetNext/_18540]]

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: Need minval < maxval, got 0 >= 0
     [[{{node parser/cond_33/then/_287/cond/then/_1093/cond/else/_2232/while/body/_3502/cond/then/_4579/random_uniform}}]]
     [[IteratorGetNext]]
  (1) Invalid argument: Need minval < maxval, got 0 >= 0
     [[{{node parser/cond_33/then/_287/cond/then/_1093/cond/else/_2232/while/body/_3502/cond/then/_4579/random_uniform}}]]
     [[IteratorGetNext]]
     [[IteratorGetNext/_19811]]

The training works well with the same environment without augmentation. I inspected my dataset with inspect.py and wrote some other tools to check for errors in bounding boxes. I checked for boxes exceeding the image boundaries and for boxes with area smaller than zero. But (sadly) I found nothing. At this point I have no idea anymore what the cause of this problem could be.

There are similiar issues to this but nothing helped: #904 #494

[Steve-2040]

Hi @SiBensberg I had errors when training using autoaugment v0 on TPU (tf-version=2.6.0): Paddings must be non-negative: -1 721 Commenting out the 'Cutout' in (efficientdet/aug/autoaugment.py) solved the problem for me and training continue with the rest of the augmentation options:

 policy = [
      [('TranslateX_BBox', 0.6, 4), ('Equalize', 0.8, 10)],
      #[('TranslateY_Only_BBoxes', 0.2, 2), ('Cutout', 0.8, 8)],
      [('Sharpness', 0.0, 8), ('ShearX_BBox', 0.4, 0)],
      [('ShearY_BBox', 1.0, 2), ('TranslateY_Only_BBoxes', 0.6, 6)],
      [('Rotate_BBox', 0.6, 10), ('Color', 1.0, 6)],
  ]

My config file:

num_classes: 5
label_map: {1: name1, 2: name2, 3: name3, 4: name4}
max_instances_per_image: 20
regenerate_source_id: True
image_size: 1280x1024
aspect_ratios: [0.1, 0.26, 0.5, 1.33] # ratio w/h: e.g. w=1.4, h=0.7 is 2.0
jitter_min: 0.8
jitter_max: 1.4
autoaugment_policy: v0

I have not investigated it further. It looks like in my case, 'Cutout' wrongly calculate the bbox coordinates? Hope this helps.

[SiBensberg]

@Steve-2040 Thank you for your answer and the idea. Sadly still one of the errors above.

[fsx950223]

Try to change https://github.com/google/automl/blob/2dbfb7984c20c24d856b96b54498799ed1b270cb/efficientdet/aug/autoaugment.py#L316 to

return image_ops.rotate(mask, angle, interpolation=self.interpolation, fill_mode='reflect')

[SiBensberg]

Thank you @fsx950223 for your answer. I think I did not fully understand your proposal. There is no mask in rotate and rotate is not a function of a class as far as I understand. So i changed it to:

return image_ops.rotate(image, radians, interpolation='nearest', fill_mode='reflect')

Which sadly still throws me this error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: Incompatible shapes: [3024,4032,3] vs. [3245,4032,3]
     [[{{node parser/cond_26/then/_212/cond_1/then/_958/cond/else/_2029/while/body/_3255/cond/then/_4504/mul_5}}]]
     [[IteratorGetNext]]
     [[IteratorGetNext/_19830]]
  (1) Invalid argument: Incompatible shapes: [3024,4032,3] vs. [3245,4032,3]
     [[{{node parser/cond_26/then/_212/cond_1/then/_958/cond/else/_2029/while/body/_3255/cond/then/_4504/mul_5}}]]
     [[IteratorGetNext]]