Adding support for other networks and models to process IRL videos

[AbelVM]

When trying to run Caffe driver, I get this error message and exits

Exception in thread Master Service Request:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/dandere2x/dandere2x/src/dandere2x/dandere2x_service/__init__.py", line 72, in run
    self.__upscale_first_frame()
  File "/dandere2x/dandere2x/src/dandere2x/dandere2x_service/__init__.py", line 114, in __upscale_first_frame
    output_image=self.context.merged_dir + "merged_" + str(1) + ".jpg")
  File "/dandere2x/dandere2x/src/dandere2x/dandere2x_service/core/waifu2x/waifu2x_caffe.py", line 77, in upscale_file
    stdout=console_output)
  File "/usr/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/dandere2x/dandere2x/src/externals/waifu2x-caffe/waifu2x-caffe-cui': '/dandere2x/dandere2x/src/externals/waifu2x-caffe/waifu2x-caffe-cui'

[akai-katto]

My apologizes, I didn't add Caffe support. Is there any particular reason to use caffee, out of curiosity? I figured most people would be using Vulkan

[AbelVM]

I'm trying to upscale/denoise a real image video, not anime, and vulkan gave me very poor results.

I read somewhere that caffe has models trained with real faces/bodies/backgrounds that could fit better than vulkan

(on the other hand, I can't find in the docs how to choose the model to be used )

[akai-katto]

Adding support for caffee is back on my radar, thanks for letting me know there's interest.

I didn't think about adding custom configuration support, but this let me know there's interest for such a thing

[AbelVM]

In my very case, looks like the anime focused model is not enough for real image and I just wanted to try Caffe, as I've read it's been trained with real people and backgrounds.

Vulkan just gave an upscaled version of the same video but with the same quality issues, no enhancement or denoise at all :disappointed:

[akai-katto]

Updating this, I've been having some issues getting caffe building (from scratch) on 19.10.

I'm using video2x's docker image as reference, but not having a cached image is proving difficult getting a non-lts system build caffe. Will update as I work on this.

[akai-katto]

In the latest docker image, I added support for a custom config file to be loaded as a CLI.

Waifu2x-ncnn-vulkan has multiple models available, should one of those suite your needs over the anime model. I'll still keep churning away at a caffe addition / figure out what's going wrong with ubuntu 19.04.

edit: setting path seems not to be working with w2x-ncnn-vulkan

[AbelVM]

I'm ok with that! If I can choose the right model for my target, I actually don't care much whether waifu2x-ncnn-vulkan or caffe is used.

Once this feature is available, I'd love to see an example on how to call an specific model using docker and a descriptive list (or link to a list) of the available models.

[akai-katto]

Have a working solution, it turns out --model-path was causing issues, so just switching it to -m fixed it.

I have to update the Docker image itself since I changed python versions (3.8 is the new minimum), so the builds are out of date, but I have custom models on w2x-vulkan working locally on docker.

[akai-katto]

Here's my command,

docker run --rm -it --gpus all -v $PWD:/host akaikatto/dandere2x:3.1.1 -p singleprocess -c /host/test_config.yaml -ws ./workspace/ -i /host/yn_moving_480.mkv -o /host/yn_moving_upscaled.mkv

Where test_config.yaml can be found here: https://gist.github.com/akai-katto/581307157dbcf27747911410e1f653a4

Same principles as dandere2x's docker usage before, wherein you had to be cd'd into the dir containing the files you want.

---

unrelated, let me know if there's an issue, since idk how the Dockerfile on Dockerhub is working correctly as building it from scratch locally required many changes... but "it works on my docker' statement feels ok for me to make.

edit: It seems 3.1.1 lacks 3.2 features (obviously), still learning how Dockerhub's automated building works.

[AbelVM]

Ok, using a local config file instead of passing a param with the model. I will run it by the weekend and get back to you with the results. Maybe the hosted docker file is updated to 3.2.x in the meantime :slightly_smiling_face:

[akai-katto]

I learned how to update / tag Docker images, so 3.2 is now downloadable : )

[AbelVM]

Running latest docker image with your config yaml this way

docker run --rm -it \
    --gpus all \
    -v $PWD:/host akaikatto/dandere2x:3.2 \
    -p singleprocess \
    -n 0 \
    -q 95 \
    -b 10 \
    -s 2 \
    -c /host/test_config.yaml \
    -ws /host/workspace/ \
    -i /host/this.mp4 \
    -o /host/this.2x.photo.mp4

Or even some command as simple as

docker run --rm -it akaikatto/dandere2x:3.2 -h

I'm getting this error

Traceback (most recent call last):
  File "/dandere2x/dandere2x/src/main.py", line 4, in <module>
    from dandere2x import Dandere2x, set_dandere2x_logger
  File "/dandere2x/dandere2x/src/dandere2x/__init__.py", line 6, in <module>
    from dandere2x.dandere2x_service.service_types.dandere2x_service_interface import Dandere2xServiceInterface
  File "/dandere2x/dandere2x/src/dandere2x/dandere2x_service/__init__.py", line 11, in <module>
    from dandere2x.dandere2x_service.core.merge import Merge
  File "/dandere2x/dandere2x/src/dandere2x/dandere2x_service/core/merge.py", line 41, in <module>
    from dandere2x.dandere2xlib.wrappers.ffmpeg.pipe_thread import Pipe
  File "/dandere2x/dandere2x/src/dandere2x/dandere2xlib/wrappers/ffmpeg/pipe_thread.py", line 88
    if (hw_accel := self.context.service_request.output_options["ffmpeg"]["pipe_video"]["-hwaccel"]) is not None:
                 ^
SyntaxError: invalid syntax

Looks like there's some issue with 3.2 image :disappointed:

[akai-katto]

I forgot to change the docker image to build using 3.9 (I have a few images laying around), I'll work on this right now.

Sorry forgot to migrate my branch

[akai-katto]

I pushed 3.2 again with the fixed changes, I'm going to purge my local images then update this comment once I verified it works

[AbelVM]

Thanks!

Just launched the process with the brand new image, no errors so far. I will add info about the results as soon as it ends in... like 10 hours :rofl:

[AbelVM]

I talked way too soon...

...

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/dandere2x/dandere2x/src/dandere2x/__init__.py", line 57, in run
    self._root_service_thread.run()
  File "/dandere2x/dandere2x/src/dandere2x/dandere2x_service/service_types/singleprocess_service.py", line 60, in run
    self._pre_process()
  File "/dandere2x/dandere2x/src/dandere2x/dandere2x_service/service_types/singleprocess_service.py", line 51, in _pre_process
    re_encode_video(ffmpeg_dir=ffmpeg_path,
  File "/dandere2x/dandere2x/src/dandere2x/dandere2xlib/wrappers/ffmpeg/ffmpeg.py", line 28, in re_encode_video
    if (hw_accel := output_options["ffmpeg"]["pre_process_video"]["-hwaccel"]) is not None:
KeyError: '-hwaccel'
Total runtime duration: 165.5916006565094

[akai-katto]

I apologize for the sloppiness of this branch, since these were two separate branches I didn't properly migrate. This error occurs since the gist I linked was out of date with what the 3.2 config needed.

Get the config file from master (https://github.com/akai-katto/dandere2x/blob/master/src/config_files/output_options.yaml) and uncomment line 47 to use your model

Again, sorry, there were a lot of moving pieces with this I forgot to stitch together in a clean manner.

[AbelVM]

Don't worry, I'm happy to help!

[AbelVM]

The process finished successfully, but the result (even choosing the photo model) is quite disappointing :disappointed:

The video is upscaled, but not enhanced at all, it has same blurriness and artifacts of the original and the output file size is now 10x . I've tested with some short videos from youtube with the same perceived quality as my input video (just searched for TV ads from the 70s) and got the same results with [3.1.1 and default model] and [3.2 and photo model] :

• With real image video, looks like the photo model gives the very same resulting quality as the default one... or maybe the setting is ignored!! (no clue in the logs about the actually used model)
• Upscaled OK
• Same quality overall, but it looks worse in some specific scenes like close up faces
• Lots of "squares" in backgrounds or static scenes
• Output file size is now x10 (it was 2x with default model and 3.1.1)

[akai-katto]

The process finished successfully, but the result (even choosing the photo model) is quite disappointing 😞

The video is upscaled, but not enhanced at all, it has same blurriness and artifacts of the original and the output file size is now 10x . I've tested with some short videos from youtube with the same perceived quality as my input video (just searched for TV ads from the 70s) and got the same results with [3.1.1 and default model] and [3.2 and photo model] :

* With real image video, looks like the `photo` model gives the very same resulting quality as the default one... **or maybe the setting is ignored!!** (no clue in the logs about the actually used model)

* Upscaled OK

* Same quality overall, but it looks worse in some specific scenes like close up faces

* Lots of "squares" in backgrounds or static scenes

* Output file size is now x10 (it was 2x with default model and 3.1.1)

This week I'm hoping to get a more conservative loss metric (mean squared error) implemented, as some users found edge cases where mean-squared-error is failing entirely. It's q uite unfortunate, but I'm hoping giving users the option will reduce the blockiness. Dandere2x really wasn't modeled for irl video, but some users found ways to do it.

Just following up thoughts, setting the quality to 99 instead of the default 97 might of helped, but I'm guessing it might be inconsequential in terms of helping make the video less blocky. Perhaps adding logs to Docker images may help (Docker being ephemeral in nature) to make sure the model is working, but I can let you know when the metrics change occurs.

[AbelVM]

Using -ws /host/workspace leaves all the temp files and traces in host computer (you might want to delete them later), so there are some logs there... That I can't get use of. I've checked them, but found no info I could use to tweak the params or so.

I will test both NCNN Vulkan versions of SRMD and RealSR next weekend through video2X, as they supposedly get the best results for irl videos. But... missing the video compression logic used in dandere2x, it turns an 11h process in dandere2x into a 35h-ish one in video2x with SRMD, and maybe a 100h-ish one for RealSR. Is there any way to implement those models in dandere2x?

[akai-katto]

Using -ws /host/workspace leaves all the temp files and traces in host computer (you might want to delete them later), so there are some logs there... That I can't get use of. I've checked them, but found no info I could use to tweak the params or so.

I will test both NCNN Vulkan versions of SRMD and RealSR next weekend through video2X, as they supposedly get the best results for irl videos. But... missing the video compression logic used in dandere2x, it turns an 11h process in dandere2x into a 35h-ish one in video2x with SRMD, and maybe a 100h-ish one for RealSR. Is there any way to implement those models in dandere2x?

RealSR was implemented in dandere2x but I've been reluctant to maintain it - the amount of "bleed" required for it to work is extremely high (see this old post of mine https://www.reddit.com/r/Dandere2x/comments/b1q0zk/breakthrough_day_image_bleeding_solved/), although I'm open to re-adding it, as it's only a few lines of code.

I'll start this evening.

Edit:

I realized there's some older actual programming (I use the word actual in that lately its been mostly maintenance work, nothing wrong with it but it's been a while since I did dev work for dandere2x), dandere2x actually adds noise during the ffmpeg stage (a very small amount of noise that's removed by waifu2x) to balance out the luminosity layers. It's been a backburner project of mine to move this logic to the actual image processing itself so the images don't need to be noised, since in this case, the upscaled image will contain noise using realSR. I'll use this as an excuse to finish those migrations. It might take some time, and if you're still around, i'll reply to this thread, but it might take a week or so.

[AbelVM]

Cool, no rush on my side. I'm gonna edit the issue title to better fit these tasks.

[AbelVM]

RealSR in video2x gives me an average of 38 s/it :open_mouth: (vs. 0.6s/it with waifu2x-vulkan in dandere2x), so... I'll patiently wait for any implementation in dandere2x with its known optimizations :slightly_smiling_face: instead of burning down my laptop for like 715h :hourglass_flowing_sand:

[AbelVM]

Now the CPP rework is done, is there any plan to get back to this issue and re-add RealSR support?

[akai-katto]

Sorry for the slow response, I was out on a business trip that demanded my attention.

I'll try and get an update for you by the end of the week and implement something, thanks for reminding me about this (it has been a while).