Implementation Help

[alexandreteles]

Which steps would I take to port this implementation to other websites/players? Is there any documentation for your script?

Thank you very much!

[net2cn]

Technically, you could simply add an @match condition and the script would just work on the website you added.

The script works briefly as follows:

1. The script will try to find a <video> tag (if multiple tags are presented, the first one will be selected, you may need to manipulate the default behavior if the first video tag presented is not the video player tag you want).
2. The video tag will be hidden (by setting its display style to 'none'). Also, the parent <div> of the video tag will have its backgroundColor style set to black to prevent some weird glitches on some websites.
3. A new <canvas> tag which uses the <video> tag as input will be added. The filter itself is a branch of post-processing shaders applied to the canvas.

You can refer to this fork made by @Michael1297. It demonstrated adding @match conditions to apply the script to other sites.

[alexandreteles]

It works great, thank you. Some sites will result in a black screen because of CORS shenanigans that I'm not sure how to fix, I suppose this problem is related to this: https://webglfundamentals.org/webgl/lessons/webgl-cors-permission.html and without properly dealing with CORS those sites will fail with:

Uncaught DOMException: Failed to execute 'texImage2D' on 'WebGLRenderingContext': The video element contains cross-origin data, and may not be loaded.
    at updateTexture (chrome-extension://eeagobfjdenkkddmbclomhiblgggliao/Bilibili_Anime4K.user.js#2:101:8)
    at Scaler.render (chrome-extension://eeagobfjdenkkddmbclomhiblgggliao/Bilibili_Anime4K.user.js#2:930:9)
    at render (chrome-extension://eeagobfjdenkkddmbclomhiblgggliao/Bilibili_Anime4K.user.js#2:1267:26)

Ex.: https://animeyabu.com/ presents such behavior. I most cases It Just Works as expected. I'm not entirely sure in which step the script should deal with it. Any ideas?

[net2cn]

After poking around without success, unfortunately, I don't think I am capable of resolving this CORS-related issue. However, I do spot a major bug that may cause the script not to work on sites other than bilibili.com, and this problem has been addressed.

If you want to try solving this CORS issue, I've created a patch for you to investigate further on this issue.

---
 Bilibili_Anime4K.js | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Bilibili_Anime4K.js b/Bilibili_Anime4K.js
index 9d47393..5874a22 100644
--- a/Bilibili_Anime4K.js
+++ b/Bilibili_Anime4K.js
@@ -1191,6 +1191,12 @@ async function injectCanvas() {
     }
     div.style.backgroundColor = "black" // Patch for ACFun.

+    if ((new URL(globalMovOrig.src)).origin !== window.location.origin) {
+        console.log("CORS detected!");
+        globalMovOrig.crossOrigin = "anonymous"
+        globalMovOrig.load()
+    }
+
     if (!globalBoard){
         console.log("globalBoard not exists. Creating new one.")

--

This is the approach I've tried to solve the problem (but unfortunately without success). I noticed that this would cause a "302 Moved Temporarily" response for the video (usually you will get a"206 Partial Content").

If you find a working approach to address this issue, please submit a PR if you are willing to do so.

[alexandreteles]

      if ((new URL(globalMovOrig.src)).origin !== window.location.origin) {
              console.log("CORS detected!");
              globalMovOrig.crossOrigin = "anonymous"
              globalMovOrig.load()
      }

I've tried a similar approach, with some success. Some websites will only work if the HTTP headers include Access-Control-Allow-Origin, which isn't guaranteed for most websites.

Another thing:

new URL(globalMovOrig.src)).origin

For some players globalMovOrig.src can't be populated right away because of reasons (mostly because some websites update the video tag dynamically) and the script will fail with TypeError: Failed to construct 'URL': Invalid URL.

Taking this into consideration, and the fact that for the script to work with sites broken by CORS-related problems we need the crossOrigin attribute, this is probably a better way for us to run the CORS detection on injectCanvas:

      if (globalMovOrig.src) {
        console.log('Video Source Found!')
        videoOrigin = new URL(globalMovOrig.src).origin
        if (videoOrigin !== window.location.origin) {
          console.log("CORS detected!");
          globalMovOrig.crossOrigin = "anonymous"
          globalMovOrig.load()
        }
      } else {
        console.log('Video Source Not Found!')
      }

Sadly userscript can't change headers as they run after the page load, so to append the appropriate headers one can use an extension like CORS Unblock, which results in the script working as expected.

I think that the best approach to solve this problem would be to pack the script into a browser extension so it could inject the headers as needed.

[net2cn]

As for the globalMovOrig.src can't be populated issue you may try to add something like

    while(!globalMovOrig.src) {
        await new Promise(r => setTimeout(r, 500));
    }

before the script tries to forge the URL object to ensure the script gets the proper video source to do CORS detection. Note this approach may cause deadlock if the globalMovOrig used is deleted by other scripts on the website, hence more work should be done to perfect this approach. (afaik bilibili does this, and I had to use an ugly hack to resolve the problem)

Yes, I did think of making a browser extension out of the concept of the script (and maybe adding mpv shader support for users to enable other upscaling algorithms as they wanted, as the shader used in this userscript is actually ported from the early version of the original Anime4K mpv shader by NeuroWhAI. I have tried to port a newer version of the filter that you can have a look at the experimental version of the userscript in this repo). Unfortunately, as I'm not an experienced web developer, I don't have enough browser extension development skills or time to acquire the skills to implement the idea of making a browser extension. Maybe someday when I find myself having the proper skills required I will try to implement one myself.

[alexandreteles]

Honestly I don't think that for general use we should add the await because some sites like Animeyabu will delete and recreate the player in some cases (for example when the default one doesn't load, so they can load an alternative version) and cause a lock. Mitigating that lock would be a pain and possibly require solutions tailored to individual websites (which is obviously infeasible).

There's barely no documentation about porting those shaders, and not a lot of shaders already ported to WebGL. If we could load shaders on the script using an URL, for example, that would already make things a bit easier to experiment with ports as we wouldn't have to mess around the script itself too much.

I now have a general understanding of how the script works with the DOM tree and I can read the code properly, but the shader building/loading is still a bit cryptic for me. If you have any references to share on how to make those ports and any idea on how we could change the script for it to load the shader from an external file I'll will take my time reading it and trying to port some of the smaller Anime4K shaders.

Best case scenario we could work out a way to pipeline shaders in order to achieve better results following the Anime4K documentation.

[net2cn]

Basically, the shader fragments used in this script are mostly the same as the original Anime4K mpv shader, but with few tweaks. Take this shader I tried to port for the experimental version as an example. A mpv shader is just like a pipeline, you put a raw image in from the beginning, let it run through the pipeline, and you will get a processed image from the end. Each fragment in the mpv shader is executed in order.

Thus, I wrote a simple script to split the mpv shader pipeline into shader fragments (there are also some renaming and pre-process are being done when running the script. Results are in this folder, to try out the script you may want to change the save folder I hard-coded in).

However, the fragments we just splitter are not proper WebGL shader fragments yet. After looking around at the original Anime4K WebGL implementation, I noticed that the comments in each fragment (here, 0.glsl as an example)

//!DESC Anime4K-v3.1-Upscale(x2)+Deblur-CNN(M)-Conv-4x3x3x1
//!HOOK NATIVE
//!BIND HOOKED
//!WHEN OUTPUT.w NATIVE.w / 1.200 > OUTPUT.h NATIVE.h / 1.200 > *
//!SAVE LUMAN0
//!COMPONENTS 4

are actually parameters for the mpv to decide how to use the fragment, thus telling us what texture should be used as input (NATIVE) and what texture should be used to save to (LUMAN0). You can refer to this documentation from mpv for more information.

With this in mind, we can hand-craft our rendering process for the script, like copying and pasting the fragments generated earlier into the userscript (all the constants with a prefix of "frag", like "frag0"), fixing syntax errors (the GLSL used in mpv shaders are a bit different from the GLSL used in WebGL shaders), adding lots of texture for the fragments to use (that's why there is a brunch of variables with a suffix of "Texture", like "luman0Texture"), and calling each fragment in order (you can refer to function Scaler.prototype.render, I've added some comments to mark where the script calls each fragment). The scaler is actually the original one by NeuroWhAI, thus mostly the same as the stable userscript with an early version of the Anime4K filter.

The rendering process in this userscript can be summarized as follows:

1. Create a quad mesh that fills up the entire canvas, this is where we will draw our final output on for users to see
2. Create a lot of textures for the shader fragments to save and load from (like luman0Texture), and the texture with the raw pixel data from the video (inputText)
3. Check if we should run the shader (e.g., if the video is paused, there is no need to run the shader to update what's already drawn on the quad mesh as we are dealing with the same data as the last frame)
4. If we should run the shader, we run each fragment with the corresponding texture required by the comments in fragments (e.g., NATIVE for the inputTex)
5. Draw the final output to the quad mesh.

I don't know if I've provided all the information you need to understand how the shader rendering process works due to my messy English. If you have more questions feel free to leave more comments. I'll try my best to provide detailed information for you to work on.

[net2cn]

Closed due to inactivity. Reopen this issue if you need further assistance.