Training records

[jaentrouble]

1. hr538e2_bugged

   • subtracted 1 instead of adding 1

2. hr538s20k

   • fixed bug of 1

3. hr538s20k_reg_lowlr

   • added regularization term

   • lr: lr_mul_inv_low

4. hr538s20k_reg_lowlr_lowgamma & hr538s20_120k_reg_lowlr_lowgamma

   • gamma_flow, gamma_mask = 0.001, 0.0005

   • lr : \~20k : lr_mul_inv_low, 20k\~ : lr_step7

   • abort at epoch44 (45th)

[jaentrouble]

5. hr538s100k_noreg

   • gamma_flow, gamma_mask = 0.0, 0.0

   • BN_momentum = 0.99999

   • lr : lr_step7

   • abort at epoch 4 (5th)

[jaentrouble]

6. hr538s100k_real

   • gamma_flow, gamma_mask = 0.0001, 0.00005

   • BN_momentum = 0.99999

   • lr : lr_step7

   • changed to real data

   • gamma/brightness/contrast/rgbshift excluded

   • aborted at epoch 57 (58th)

[jaentrouble]

7. hr538s58_100k_real

   • gamma_flow, gamma_mask = 0.000001, 0.0000005

   • BN_momentum = 0.99999

   • lr: low_lr (1e-5 constant)

   • aborted at epoch 1 (2nd)

[jaentrouble]

8. hr3216_100k_real

   • gamma_flow, gamma_mask = 0.000001, 0.0000005

   • BN_momentum = 0.99999

   • lr: lr_step7

   • image shape changed to (960,540)

   • batch size 5

   • abort at epoch 20 (21st)

[jaentrouble]

9. hr3216_21_100k_real

   • loss fixed

   • gamma_flow, gamma_mask = 0.01, 0.005

   • same as above

   • forgot to load; it is learned from the scratch

[jaentrouble]

10. hr3216_merged

    • merged flow

    • other things same as above

    • abort

[jaentrouble]

11. hr3216_half

    • frame doubler (For sanity check)

    • abort at epoch 23 (24th)

[jaentrouble]

12. hr3216_half_lowerlr

    • lr : lr_step8

    • abort at epoch 8 (9th)

[jaentrouble]

13. hr3216_half_noaug

    • lr: lr_step7

    • Only flipping and rotating

    • Checked flow -> flow does exist, but very little

[jaentrouble]

13. hr3216_half_noaug_200

    • lr: lr_step7

    • load from 12th epoch 99 (100th)

    • gamma: 0.01 & 0.005

[jaentrouble]

14. hr3216_half_noaug_300

    • lr: lr_step7

    • load from 13th epoch 99 (100th)

    • gamma: 0.03 & 0.015

[jaentrouble]

15. hr3216_half_noaug_400

    • lr: lr_step7

    • load from 14th epoch 99 (100th)

    • gamma: 0.1 & 0.015

[jaentrouble]

16. hr3216_half_noaug_500

    • load from 15th epoch 99 (100th)

    • anime-like data added

    • other things same as 15th

[jaentrouble]

17. hr3216_half_noaug_600

    • load from 16th epoch 99 (100th)

    • changed to epoch 20 * steps 5000

    • other things unchanged

[jaentrouble]

18. hr3216_half_noaug_700

    • load from 17th epoch 19 (20th)

    • added @tf.function to bilinear interpretation

    • other things unchanged

[jaentrouble]

19. hr3216_half_noaug_850

    • load from 18th epoch 19 (20th)

    • lr: lr_step9

    • gamma: 0.05 & 0.015

    • other things unchanged

[jaentrouble]

20. hr3216_half_noaug_950

    • load from 19th epoch 29 (30th)

    • batch size: 7

    • other things unchanged

[jaentrouble]

21. hr3216_half_noaug_1000

    • load from 20th epoch 19 (20th)

    • other things unchanged

[jaentrouble]

22. hr3216_half_bilinear_200

    • New training

    • changed deconvolution layer to bilinear upscaling

    • changed final convolution layer kernel size to 3x3 (from 1x1)

    • gamma: 0.05 & 0.015

    • batch size: 6

    • lr: lr_step7

[jaentrouble]

23. hr3216_half_bilinear_260

    • load from 22nd epoch 39 (40th)

    • lr: lr_mul_inv_low

[jaentrouble]

24. hr3216_half_bilinear_400

    • load from 23rd epoch 11 (12th)

    • lr: lr_step9

[jaentrouble]

25. hr3216_half_bilinear_500

    • load from 24th epoch 27 (28th)

    • lr: lr_step9

    • gamma: 0.1 & 0.03

[jaentrouble]

26. hr3216_half_bilinear_600

    • load from 25th epoch 20

    • lr: lr_step9

[jaentrouble]

27. hr3216_half_bilinear_700

    • load from 26th epoch 20

    • lr: lr_step9

[jaentrouble]

28. hr3216_half_bilinear_800

    • load from 27th epoch20

    • lr: lr_step9

[jaentrouble]

29. hr3216_half_bilinear_900

    • load from 28th epoch 20

    • lr: low_lr

[jaentrouble]

30. hr3216_half_bilinear_1000

    • load from 29th epoch 20

    • lr: lr_step9

    ~gamma: 100.0 & 50.0~ : Abort

    • Restart with gamma 0.01 & 0.005

[jaentrouble]

31. hr3216_half_ani_100

    • load from 30th epoch 20

    • Now training only with ani-like videos

    • No resize augmentation. Only flipping

    • lr: lr_step9

    • Other things untouched

[jaentrouble]

32. hr3216_half_ani_mv_100

    • load from 30th epoch 20 (31st was not so successful)

    • added shifting windows

    • parallel dataset added

    • lr: lr_step9

    • other things untouched

[jaentrouble]

33. hr3216_half_ani_mv_200

    • load from 32th epoch 20

    • lr: low_lr

    • BN_MOMENTUM = 0.999 (0.99999 seems too slow)

[jaentrouble]

34. hr3216_half_ani_mv_400

    • load from 33th epoch 20

    • lr: low_lr

    • epoch: 40 steps: 5000

[jaentrouble]

Change strategy

• Instead of shoving whole frame, divide into pieces

• Training is similar: Cut small portion of a frame.

  • However, delete augmentation which takes a whole frame

• Change input size to 320x320 and increase model size

• Try Efficient HRNet

[jaentrouble]

35. ehrb0_100

    • Used ehrb0_143_32

    • batch size: 24

    • lr: lr_step7

    • epoch: 100 steps 1000

    • gamma: 0.1 & 0.03 ~(NaN started after changing to 0.01&0.005 - need to check)~

      • gamma does not prevent NaN

~Changed last layer activation from tanh to relu (-1,1)~

Rolled back to tanh

[jaentrouble]

36. ehrb0_200

    • continue from 35th epoch 100

    • lr: low_lr

[jaentrouble]

37. ehrb0_500

    • continue from 36th epoch 100

    • lr: low_lr

    • epoch:300 steps 1000

Abort at epoch 225

[jaentrouble]

Change strategy

• Splitting into pieces loses too much information and is not so good.
• Especially with animation, bigger resolution does not mean more objects, i.e. movements.
• Therefore, resize input image to smaller size, and use that to make flowmap, and resize flowmap back to the original size. This can preserve high resolution image and still use less memory.

[jaentrouble]

38. ehrb0_resize_300
    • new start, using the strategy above
    • lr: lr_step7_3
    • batch size: 12
    • training frame size: (960,540)
    • flow_map_size: (512,288)
    • epoch 300, steps 1000

[jaentrouble]

Why weird 'holes'?

• It seems like the 'NaN' thing
• When NaN appears, the first thing to suspect is learning rate.
• More over, in animation, sudden scene change or sudden movement occurs often. Therefore, the original flow range [-1,1] may not be enough as it restricts to only half of the frame's width and height.

Things to try

• Lower learning rate
• Increase DELTA_MAX (training image window)
• multiply 2 to the final layer. i.e. the range of flow becomes [-2,2]

[jaentrouble]

39. ehrb0_double_300
    • New start, using strategy above
    • lr: lr_step8_2 (1/100 of lr_step7_3)
    • doubled flow; range (-2,2)
    • delta_max : 60
    • training frame size: (960,540)
    • flow_map_size: (512,288)
    • epoch 300, steps 1000
    • batch size: 12

Abort at epoch 79

[jaentrouble]

Lowering learning rate made it worse

• Therefore, the learning rate was rather 'too low' than 'too high'
• Slow learning showed how the learning process proceeds.
  1. First, all random flow becomes 'near zero'
  2. Then it learns more correctly
• Theorem: If some parts do not get 'near zero', these become the 'weird holes'
• If this is the case, since we already know that flows are usually 'near zero', then maybe initializing the final layer with zeros may help.
• Try increasing learning rate if zeroing out the final layer does not help.

[jaentrouble]

40. ehrb0_zero_300

• New start
• final convolution layer's kernel initializer = 'zeros'
• lr: lr_step7_3
• training frame size: (960,540)
• flow_map_size: (512,288)
• epoch 300, steps 1000
• batch size: 12

[jaentrouble]

41. ehrb0_edge_pre_100
    • New start
    • Pre training with edge maps (For later cyclic training)
    • lr: lr_step7_2
    • epoch 100 steps 1000
    • batch size: 16

[jaentrouble]

42. cyclic_100
    • Load from 41
    • Using full cyclic training model
    • lr: lr_step7_2
    • epoch 100 steps 1000
    • batch size : 8

[jaentrouble]

43. ehrb0_edge_pre2_100
    • New start
    • Found that there was a big mistake in functional HR model
    • lr: lr_step7_2
    • epoch 100 steps 1000
    • batch size: 16