It is no longer working.

[Mikasa-san]

Please update it. It was working yesterday.

[avi12]

It seems that the playlist downloads got broken

I also noticed that no matter the browser I'm running it on (Chrome, Edge, Opera), they all mention the "Manifest version 2 will not be supported starting from January 2023" This sadly cannot be fixed. Instead, someone needs to rethink the extension's architecture to port it to Manifest V3, and I definitely can't do this alone, and weekends are definitely not enough (as I have a full-time job)

[guest271314]

This sadly cannot be fixed. Instead, someone needs to rethink the extension's architecture to port it to Manifest V3, and I definitely can't do this alone, and weekends are definitely not enough (as I have a full-time job)

What needs to be done?

[avi12]

As I said, a complete rethinking of the way the extension works I'm not even sure if some of the things will work in the first place For example, because a Service Worker is no longer a web page like the background page used to be, this line will fail: https://github.com/avi12/youtube-downloader/blob/8a42311323d2557bd7a3666c28b438f802e1df22/src/scripts/background.ts#L318 Which is basically a backbone, as the URL Web API is necessary to download the actual file A possible workaround is to send that URL to the content script and then trigger a download via file-saver

Next up:

1. The extension currently relies on FFmpeg to combine the video & audio (as YouTube provides them separately)
   1. I don't know whether FFmpeg can work in a Service Worker
   2. If it does, you'll have to take into account the 5-minute time limit of Service Workers This means: If the user wants to download a large video (lengthy and/or high quality), you need to make sure to first download the video & audio, probably wait until the Service Worker is killed (I don't know how to check such a thing), and then run an FFmpeg job to combine them, yet - depending on the user's hardware, it might take more than 5 minutes, in which case you'll need to show an error Of course, make sure to allow the user to cancel the download at any time
2. Regarding file downloads, you'll probably want to use Background Fetch
   • Make sure that you can calculate a download's progress, as well as cancel it at any time
   • After the download is complete, you can potentially store it in the storage as base64 and then fetch later to use for the FFmpeg job

Bonus: Try to support Firefox, as it currently faces the SharedArrayBuffer issue

Good luck! Since I have a full-time job, I can't guarantee to be available, but I can try to help in some stuff

[guest271314]

As I said, a complete rethinking of the way the extension works I'm not even sure if some of the things will work in the first place For example, because a Service Worker is no longer a web page like the background page used to be, this line will fail:

https://github.com/avi12/youtube-downloader/blob/8a42311323d2557bd7a3666c28b438f802e1df22/src/scripts/background.ts#L318

Which is basically a backbone, as the URL Web API is necessary to download the actual file

We can send dataFile to any other Web page using postMessage().

A possible workaround is to send that URL to the content script and then trigger a download via file-saver

File System Access API can be used on Chromium/Chrome.

Next up:

1. The extension currently relies on FFmpeg to combine the video & audio (as YouTube provides them separately)

   1. I don't know whether FFmpeg can work in a Service Worker

Yes.

Alternative approaches include using Native Messaging and FFmpeg installed on the users machine.

2. If it does, you'll have to take into account the 5-minute time limit of Service Workers This means: If the user wants to download a large video (lengthy and/or high quality), you need to make sure to first download the video & audio, probably wait until the Service Worker is killed (I don't know how to check such a thing), and then run an FFmpeg job to combine them, yet - depending on the user's hardware, it might take more than 5 minutes, in which case you'll need to show an error Of course, make sure to allow the user to cancel the download at any time

There are several ways to keep a ServiceWorker active indefinitely, see https://github.com/guest271314/persistent-serviceworker.

2. Regarding file downloads, you'll probably want to use Background Fetch

   • Make sure that you can calculate a download's progress, as well as cancel it at any time
   • After the download is complete, you can potentially store it in the storage as base64 and then fetch later to use for the FFmpeg job

Bonus: Try to support Firefox, as it currently faces the SharedArrayBuffer issue

We can remove Content-Security-Policy headers altogether, see https://github.com/guest271314/remove-csp-header. Where once removed it is possible to fetch chrome-extension: URL's or localhost from any Web page.

Good luck! Since I have a full-time job, I can't guarantee to be available, but I can try to help in some stuff

I do, too. I am trying to help where I am able. I have not used your specific code. I have used youtube-dl. The deprecation of MV2 and transition to MV3 can be worked around.

[avi12]

@guest271314 I forgot to mention that this needs to be worked around as well: https://github.com/avi12/youtube-downloader/blob/f6ff4f64f1f32b68c0d83d17f54f277d522a5642/src/scripts/yt-downloader-functions.ts#L41-L48 I went back to ytdlr, and apparently, the maintainer has updated the code to use YouTube's API, rather than relying on eval, which is great news

Alternative approaches include using Native Messaging and FFmpeg installed on the user's machine

Not good. I wish the extension to be standalone and not rely on the user installing anything

We can remove Content-Security-Policy headers altogether, see https://github.com/guest271314/remove-csp-header. Where once removed it is possible to fetch chrome-extension: URL's or localhost from any Web page.

That sounds very interesting. I'm curious to see how it can be applied in practice

I have used youtube-dl.

Me too. By debugging the code, I realized very important aspects that resulted in me seeking the decipher function I pasted a snippet of at the beginning of this comment

[guest271314]

@guest271314 I forgot to mention that this needs to be worked around as well:

https://github.com/avi12/youtube-downloader/blob/f6ff4f64f1f32b68c0d83d17f54f277d522a5642/src/scripts/yt-downloader-functions.ts#L41-L48

I don't know what that function does.

I went back to ytdlr, and apparently, the maintainer has updated the code to use YouTube's API, rather than relying on eval, which is great news

Alternative approaches include using Native Messaging and FFmpeg installed on the user's machine

Not good. I wish the extension to be standalone and not rely on the user installing anything

I'm not sure what you mean?

The user has to install FFmpeg.wasm. If FFmpeg is already installed on their machine no installation of anything else is necessary to utilize Native Messaging to communicate with the already installed application, e.g., I use already installed mkvmerge here https://github.com/guest271314/native-messaging-mkvmerge and already installed eSpeak NG here https://github.com/guest271314/native-messaging-espeak-ng.

We can remove Content-Security-Policy headers altogether, see https://github.com/guest271314/remove-csp-header. Where once removed it is possible to fetch chrome-extension: URL's or localhost from any Web page.

That sounds very interesting. I'm curious to see how it can be applied in practice

Simply put, you can fetch() resources from a local server or chrome-extension: protocol URL's from any Web page. If that functionality is necessary. For localhost just serve with the following headers:

Access-Control-Allow-Headers: Access-Control-Request-Private-Network Access-Control-Allow-Private-Network: true

I have used youtube-dl.

Me too. By debugging the code, I realized very important aspects that resulted in me seeking the decipher function I pasted a snippet of at the beginning of this comment

[avi12]

@guest271314 I'm starting to investigate what can be done for the download part and so far I'm getting negative results as regards Background Fetch, i.e. when I initialize a download for a video, it will keep failing

[guest271314]

The last time I checked Background Fetch is not implemented in MV3 ServiceWorker for extension https://github.com/mdn/content/issues/12370.

You can use a proxy to fetch() external resources from inside to outside of the ServiceWorker if the issue is CORS restrictions.

[guest271314]

What happens when you just use fetch()?

[avi12]

I believe this is the relevant issue

[avi12]

What happens when you just use fetch()?

I can use fetch as usual, but the issue is that a Service Worker in the MV3 extension has a 5-minute lifespan limit Imagine wanting to download a 1-hour podcast, only to be interrupted because Chrome killed the Service Worker The Background Fetch API was supposed to be the solution, but so far I cannot manage to download external files, so it's irrelevant

[guest271314]

I have kept the same MV3 ServiceWorker active for well over 24 hours using the approaches described here https://github.com/guest271314/persistent-serviceworker.

[avi12]

Did you use this one? https://github.dev/guest271314/persistent-serviceworker/blob/c97f3092b099c33d43a86d1e2add176238e1cc3b/chromium_extension_web_accessible_resources_iframe_message_event

[guest271314]

All of them.

[guest271314]

If any of the approaches do not work as expected kindly file an issue.

[avi12]

Right, but what would be the best fit for this extension's use case?

[guest271314]

Any one. Message passing between an <iframe> and ServiceWorker works https://github.com/guest271314/persistent-serviceworker/tree/main/chromium_extension_web_accessible_resources_iframe_message_event. That is what I use to stream and record hours of live audio. Technically any never ending fetch request works.

[guest271314]

I suggest testing all of the approaches. Then make a decision based on evidence.

[guest271314]

Imagine wanting to download a 1-hour podcast

It shouldn't take long to download a 1 hour podcast. I fetch Node.js binary download that includes node executable, npm, etc. in a few seconds.

[avi12]

Basically, I need to fetch the podcast, both the video and the audio, send both over to FFmpeg to be merged into a single MP4 (or any other video format, for that matter), and then initialize the download on the user's end so he can both of view the download progress and cancel it at any time

[guest271314]

All possible. The ServiceWorker becoming inactive can be worked around using any of the approached linked above. The download can use fetch() piped to a File System Access API FileSystemWritableFileStream with AbortController set.

[avi12]

Sounds great, I will investigate the approaches mentioned in your repo in a few hours

[avi12]

I tried testing chromium_extension_web_accessible_resources_iframe_message_event, but clicking on the extension icon threw an error:

Uncaught (in promise) Error: Cannot access a chrome:// URL

at this line

[guest271314]

Does the manifest.json include the following?

"host_permissions": ["<all_urls>"],

[avi12]

Yes, I cloned the repo and tried sideloading the extension as-is

On Sat, Oct 22, 2022 at 5:45 PM guest271314 @.***> wrote:

Does the manifest.json include the following?

"host_permissions": [""],

— Reply to this email directly, view it on GitHub https://github.com/avi12/youtube-downloader/issues/2#issuecomment-1287814315, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRACFCGFVQNKBSWKPGV6HTWEP4YBANCNFSM5RSPB23A . You are receiving this because you modified the open/close state.Message ID: @.***>

[guest271314]

Do you really need to run the code on a chrome: URL?

[avi12]

I just cloned and sideloaded the repo to see that it works and I don't get errors

[guest271314]

What happens when you run the code on a URL that is not chrome: protocol?

[guest271314]

It will require further code to access a chrome: URL or chrome-extension: from another extension; i.e., modify or remove CSP headers and explicitly set the chrome: URL you want to access. I think you are testing on a chrome: protocol URL, which is probably not what users will do. You can create an index.html page in the extension for users to use if that is an option you want to provide. By default chrome extensions cannot access chrome URL's or other extensions. When you navigate to chrome://flags you will note the flag

Extensions on chrome:// URLs

Enables running extensions on chrome:// URLs, where extensions explicitly request this permission. – Mac, Windows, Linux, ChromeOS, Android, Fuchsia, Lacros

#extensions-on-chrome-urls

[guest271314]

See https://groups.google.com/a/chromium.org/g/chromium-extensions/c/9uC1EiIxMlQ/m/F-ElRDkRAgAJ

2) the focused tab was on a page that does not allow script injection. Notably these include Chrome pages (chrome://) and extension pages (chrome-extension://)

[avi12]

Ok, that's useful info, but I'm not sure what's my solution

[guest271314]

Solution for what? As long as you don't try to inject the script on pages such as chrome-extensions or chrome://settings the ServiceWorker establish communication with the Tab and remain actie indefinitely,

[avi12]

Can you make a quick repo example of having a constant fetch to a certain site? I didn't fully understand how to approach it programmatically

[guest271314]

I don't think you need a constant fetch. However, that is what this does using EventSource https://guest271314.github.io/persistent-serviceworker/ and this does with fetch() and respondWith() https://github.com/guest271314/persistent-serviceworker/tree/main/readablestream-fetch-respondwith.

How do you fetch audio and video from YouTube?

[avi12]

How do you fetch audio and video from YouTube?

const url = "https://www.youtube.com/watch?v=u4nkpIyPkSE";
const text = await (await fetch(url)).text();
const match = text.match(/ytInitialPlayerResponse\s*=\s*({.+?})\s*;\s*(?:var\s+meta|<\/script|\n)/)[1];
const json = JSON.parse(match);
const { adapticeFormats } = json.streamingData;
const videoObj = adaptiveFormats.find(({ mimeType }) => mimeType.startsWith("video"));
const audioObj = adaptiveFormats.find(({ mimeType }) => mimeType.startsWith("audio"));
const urlVideo = videoObj.url;
const urlAudio = audioObj.url;

const [dataVIdeo, dataAudio] = await Promise.all([fetch(urlVideo), fetch(urlAudio)]);

The last line has to be a little bit more complex, has to include features such as progress tracking and the ability to cancel the fetching, but that's the gist of it To combine them, I need to use a program like FFmpeg, then start the download using chrome.downloads

[avi12]

I don't think you need a constant fetch

Imagine a user trying to download an hour long 4K video on a slow Internet connection By default, the fetch will be disrupted by the limitation of the Service Worker's 5-minute lifetime

[guest271314]

I don't think it will take 5 minutes to download an hour long 4K video.

Either of the extension options in https://github.com/guest271314/persistent-serviceworker will keep the MV3 ServiceWorker active indefinitely.

[avi12]

Either of the extension options in https://github.com/guest271314/persistent-serviceworker will keep the MV3 ServiceWorker active indefinitely.

Right, but when I tried running chromium_extension_web_accessible_resources_iframe_message_event as-is, it failed to run, so I'm not sure how the code is supposed to be structured to make it possible

I don't think it will take 5 minutes to download an hour long 4K video.

A 20Mb/s download rate will take a long time, possibly a few hours

[guest271314]

Right, but when I tried running chromium_extension_web_accessible_resources_iframe_message_event as-is, it failed to run, so I'm not sure how the code is supposed to be structured to make it possible

That is because you evidently ran the code on a URL beginning with chrome:. That won't work. Try on a URL that does not begin with chrome: protocol.

[avi12]

But how would it look programmatically? My goal is to fetch the video/audio data on the Service Worker side, not on the content script

[guest271314]

It looks the same. On a non-chrome: URL click the icon, an <iframe> is appended to the doument which exchanges messages with the ServiceWorker to keep it active indefinitely, video and audio are requested in the ServiceWorker.

[guest271314]

I just fetched the audio and video. Do you have an example URL for "an hour long 4K video"?

[avi12]

Quick YouTube search gave me this

[guest271314]

What comment did you delete?

[avi12]

The comment that fixed my code I edited my comment to include your fix

[guest271314]

Just fetched audio and video in ServiceWorker; took a few seconds.

// ...
const [dataVideo, dataAudio] = await Promise.all([fetch(urlVideo), fetch(urlAudio)]);
    console.log(dataVideo.headers.get('content-length'), dataAudio.headers.get('content-length'));
}
51990310498 586277884

[guest271314]

I can record my screen while using the https://github.com/guest271314/persistent-serviceworker/tree/main/chromium_extension_web_accessible_resources_iframe_message_event version, if that will be helpful - though, again, I don't think your issue here is keeping the ServiceWorker active over 5 minutes. You are not recording the video in real-time.

[avi12]

Just fetched audio and video in ServiceWorker; took a few seconds.

Awesome, I assume your download speed is at least 100Mb/s But consider users with a poor internet connection who wish to download large videos - their download will need to take at least an hour

[guest271314]

Have you tested the persistent ServiceWorker code on an non-chrome: URL?

[avi12]

No I assume you mean to run on a content script, right?