Issue with Vespcn parameters questions?

[lhwwhl]

I'm really appreciate your sharing of this project . But I just started it, in vespcn.yaml, lr_decay: 1 would cause some problems. Traceback (most recent call last): File "run.py", line 47, in tf.app.run(main) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run _sys.exit(main(argv)) File "run.py", line 36, in main return Run.run(additional_functions) File "/Users/lhw/Worker/learn/opensrc/SR/VideoSuperResolution/VSR/Tools/Run.py", line 158, in run train_config, test_config, infer_config = init_loader_config(opt) File "/Users/lhw/Worker/learn/opensrc/SR/VideoSuperResolution/VSR/Tools/Run.py", line 107, in init_loader_config train_config.lr_schedule = lr_decay(lr=opt.lr, opt.lr_decay) TypeError: lr_decay() argument after ** must be a mapping, not int. Could you share your config? I'm looking forward to your reply, Thank you!

[LoSealL]

@lhwwhl Sorry for the late response. You are right, the parameters are forgotten to update. I fixed this issue. Please try again :).

[lhwwhl]

Using "python3 run.py --model vespcn --dataset mcl-v test none", I can train successfully. But in test, "python3 run.py --model vespcn --test mcl-v", cause some problem: Caused by op 'vespcn/MotionEstimation/add', defined at: File "run.py", line 47, in tf.app.run(main) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run _sys.exit(main(argv)) File "run.py", line 36, in main return Run.run(*additional_functions) File "/home/zhangxiangling/VideoSuperResolution/VSR/Tools/Run.py", line 163, in run with trainer(model, root, verbosity) as t: File "/home/zhangxiangling/VideoSuperResolution/VSR/Framework/Trainer.py", line 74, in enter self._startup() File "/home/zhangxiangling/VideoSuperResolution/VSR/Framework/Trainer.py", line 68, in _startup self._m.compile() File "/home/zhangxiangling/VideoSuperResolution/VSR/Framework/SuperResolution.py", line 83, in compile self.build_graph() File "/home/zhangxiangling/VideoSuperResolution/VSR/Models/Vespcn.py", line 98, in build_graph w, f = self._me(input_center, frames[i]) File "/home/zhangxiangling/VideoSuperResolution/VSR/Models/Vespcn.py", line 58, in _me w0 = warp(x1, u0, v0, True) File "/home/zhangxiangling/VideoSuperResolution/VSR/Framework/Motion.py", line 196, in warp u += G[..., 1] File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 850, in binary_op_wrapper return func(x, y, name=name) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 297, in add "Add", x=x, y=y, name=name) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func return func(args, **kwargs) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op op_def=op_def) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1717, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Incompatible shapes: [1,272,480] vs. [1,270,480] [[Node: vespcn/MotionEstimation/add = Add[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](vespcn/MotionEstimation/strided_slice, vespcn/MotionEstimation/strided_slice_17)]] [[Node: PSNR/All/_369 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_998_PSNR/All", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

I'm confused with it. Thank you!

[LoSealL]

@lhwwhl In training, it crops input images to patches as 96x96 (args: patch_size), but in testing, the input is the entire images. As in this case, MCL-V's image is 1920x1080, and the test input with x4 down-sampled is 480x270, however, in VESPCN flow estimation uses 2 strided conv layers, which mean the feature size is down scaled to w/4xh/4, as 96 can be divided by 16, so training is fine, but 270 can't be divided by 4 (and it pads to 272, hence causes size mismatch between outputs and labels).

To address this problem, you can:

• test with vid4 dataset
• crop or pad test images in mcl-v to a size divided by 16, you can doing so by adding --add_custom_callbacks=pad

** the function pad can be found in custom_api.py.

---

[BUG WARNING] ~~I found that add_custom_callbacks pads input features, but doesn't pad labels. So you still get exception while calculating metrics... For now, you have to use --infer to work around.~~

---

[BUG FIXED] Fixed this issue now. If there's any questions, feel free to let me know.

[LoSealL]

Note pad pads inputs to be divided by 64, you can change corresponding value in custom_api.py

[peppaseven]

Could you share how to prepare mcl-v trainning data and test data? I only download some yuv videos...

[LoSealL]

@peppaseven Hi. Please refer to ./Data/datasets.yaml, where I set MCL-V section to /mnt/data/datasets/MCL-V/train/*.yuv.

MCL-V only has 12 video clips for training. Just copy/move these .yuv files together and modify the path in datasets.yaml. VSR can read from YUV files directly. For testing and validation, I move 1 video from training folder into a testing folder. (See datasets.yaml:77 for details)

When you want to train with MCL-V, use python run.py --model=vespcn --dataset=mcl-v --threads=4 --memory_limit=1GB --epochs=200, --memory_limit=1GB will be needed because extract patches from MCL-V can reach to a huge memory allocation.

[peppaseven]

Thanks very much, I got this. so, only need randomly choose 12 videos clips as training set, others can be validation or testing set...right?

[LoSealL]

@peppaseven Yes. Mention that MCL-V only has 12 videos in total. So I use 11 of them to train, and 1 for test. Video dataset is surely a problem for VSR, especially compared to SISR. To extend MCL-V, I suggest to use GOPRO, a big ourdoor video dataset (organized as a series of PNG files) which contains 33 videos and thounds of frames in total.

[peppaseven]

I pick up the above as training set..

[LoSealL]

@peppaseven It's no problem. For example you just train without validation. And verify your model after training to test on other dataset (such as VID4).

Because in current version of VSR, we can't mix png files and raw files (such as .yuv) together for training and validation. A pratical solution is to move one of the file (i.e. Tennis_24fps.yuv) out of training folder and use it as a validation file.

[peppaseven]

thanks, actually, i just want to prepare training data for "HDRNET", want to find a good algorithm with whole training pipeline, when get training set, after tested that, choose this as training set for HDRNET. as we know, it it so difficult to prepare a training set with pictures retouched by adobe lightroom expert...until now, can't find a good algorithm for video enhancement. any advice for this?

[LoSealL]

@peppaseven I know little about this field. There do exist some works to HDR photo using CNN, as for video, I think the refining method may be the same as we do in VSR or video style transfer: motion-compensation and Conv3D.