Cobalt API Status 403 Cloudflare

[DFreezerR]

bug description

Hello! I created a Chrome extension for myself using cobalt API to download videos for myself. At first everything worked fine but after about a week I'm getting 403 responses that includes Cloudflare protection which tells me that I'm blocked. Did I broke some rules or maybe I overlooked something? Maybe I hit limited number of API usage? (if this is a thing)

reproduction steps

For steps I can only share the code with request I have:

let url = msg.url
    let body = JSON.stringify(
        {
            url: encodeURI(url),
            vCodec: requestOptions.vCodec,
            vQuality: requestOptions.vQuality,
            filenamePattern: requestOptions.filenamePattern,
            twitterGif: requestOptions.twitterGif
        })
    let req = new Request(reqURL, {
        body: body,
        method: "POST",
        headers: {
            "Accept": "application/json",
            "Content-Type": "application/json",
            "User-Agent": "ChromeExtension/0.1.1"
        },
    })

And specific parameters I pass here (picked random video): Body

{
    "url":"https://youtu.be/ADhMF-T1DKI",
    "vCodec":"h264",
    "vQuality":"720",
    "filenamePattern":"nerdy",
    "twitterGif":true
}

At first I didn't have User-Agent but it worked unlit Cloudflare blocked me. After searching github issues I tried adding User-Agent but I'm still blocked.

links

response.txt - response html page I got after requests I removed divs with my IP address just in case

platform information

• Windows 11 23H2
• Vivaldi 6.8.3381.50 (Stable channel) (64-bit)

additional context

This project never meant to be commercial and only for personal use to make task of downloading videos easier. I should also specify that this response is the same regardless of video's website source (twitter/x, youtube)

[KwiatekMiki]

"User-Agent": "ChromeExtension/0.1.1"

Instead of the generic "ChromeExtension", use the extension name... or just host your own instance.

[KwiatekMiki]

Did I broke some rules

Probably not.

Maybe I hit limited number of API usage? (if this is a thing)

There is a rate limit (I think it's 30 20 _{(the default, not the example of what you can change it to)} request/minute) but there is no limit on how much you can use the API (that I'm aware of)

I should also specify that this response is the same regardless of video's website source (twitter/x, youtube)

That doesn't matter.

[lostdusty]

its 20 req/min

[DFreezerR]

"User-Agent": "ChromeExtension/0.1.1"

Instead of the generic "ChromeExtension", use the extension name... or just host your own instance.

Sadly renaming User Agent doesn't work (Changed it to "CobaltToolsVideoDLExtension/0.1.1" like in manifest but without spaces). But for hosting myself.. I want to leave this for the last option, just want to know if this block situation is something I can deal with.

Did I broke some rules

Probably not.

Just asked it because I had theory that maybe I was blocked manually by the owners (for breaking rules or whatever reason)

And for rate limiting, thanks, will keep it in mind, but I'm sure I didn't spam it that much since the only requests I did was manual pressing download button a have in my extension (no loops and other stuff). And if I DID go beyond 20 req/min I would not have been blocked for several days, right? Or...

[KwiatekMiki]

Sadly renaming User Agent doesn't work (Changed it to "CobaltToolsVideoDLExtension/0.1.1" like in manifest but without spaces).

weird....... it works for me

But for hosting myself.. I want to leave this for the last option

Is that because you don't know how or..?

And if I DID go beyond 20 req/min I would not have been blocked for several days, right?

You just get an error saying that you're doing a lot of requests and that's it.

I spammed cobalt's API multiple times (sorry) and I can still use it.

[DFreezerR]

Is that because you don't know how or..?

No I can, but I don't want additional services running in my background, I think if I can make it work with extension than thats better, if not than my last option would be self host. To be honest I was just curious why I got cloudflare blocked🤷‍♂️ since I was enjoying doing my little "work"

I spammed cobalt's API multiple times (sorry) and I can still use it.

Than, rate limit is not at fault in this case. At first I even got surprised since after request I didn't get JSON response but a HTML page back.

Tried throwing manual request from vscode extension and it worked. Well...

Manual request

 

Also found out you can't simply change user agent in Request constructor. You have to use chrome.declarativeNetRequest API since I don't know why I started coding extension with manifest V3...

DevTools


But when I actually changed User Agent it's still the same cloudflare blocking message

DevTools


I even paste a payload, maybe it helps idk

Payload


I really can not find any differences in sending request with vscode extension and actual request with JS, except maybe some additional headers prevents it from working?

[KwiatekMiki]

Tried throwing manual request from vscode extension

Did you set the same user-agent there?

[DFreezerR]

Tried throwing manual request from vscode extension

Did you set the same user-agent there?

Yes, if you open image in spoiler you can see it

[DFreezerR]

Ok, after some testing, found out that browser sending additional headers with every request made with extension. The specific header that prevents request passing CloudFlare was "Origin" header. Browser is setting this header automatically to the path of that extension, and somehow it's get blocked by cloudflare (my guess it's because the Origin does not match the default http/https scheme)

DevTools


To fix it I had to change "User-Agent" and "Origin" headers using chrome.declarativeNetRequest API (since I'm using manifest V3). In service-worker.js specify rules with needed headers and conditions when these rules will apply:

service-worker.js

```js const rules = { //removes old rules removeRuleIds: [1], //adding our new rules addRules: [ { id: 1, priority: 1, action: { type: 'modifyHeaders', requestHeaders: [ { header: 'Content-Type', operation: 'set', value: 'application/json' }, { header: 'Accept', operation: 'set', value: 'application/json' }, { header: 'User-Agent', operation: 'set', value: `CobaltToolsVideoDLExtension/0.1.1` }, //set Origin header to "null" or "*" { header: "Origin", operation: "set", value: "null" } ], }, //setting conditions that these headers will be applied on cobalt.tools website //and only using xmlhttprequest type, that would work using "fetch" functions condition: { resourceTypes: ['xmlhttprequest'], urlFilter: '*://*.cobalt.tools/*' }, }, ], } ```

Then to make things more convenient, apply these rules using chrome.runtime.onInstalled event:

service-worker.js

```js chrome.runtime.onInstalled.addListener(function (details) { chrome.declarativeNetRequest.updateDynamicRules(rules); }); ```

Then this cloudflare problem will be gone, at least for me. And lastly sending simple fetch request will work:

service-worker.js

```js let body = JSON.stringify( { url: encodeURI(url), vCodec: vCodec, vQuality: vQuality, filenamePattern: filenamePattern }) let req = new Request("https://api.cobalt.tools/api/json", { body: body, method: "POST" }) let response = await fetch(req) ```

Getting status 200:

DevTools


And for manifest V2 I believe this will work: Stackoverflow

I still doesn't understand how everything worked until some days ago, maybe someone more knowledgeable can tell. But this what fixed my problem. Thanks for everyone time.

[KwiatekMiki]

If you fixed this then maybe you should close the issue.

[DFreezerR]

Thank you for help, closed