Use proxy

[sclsj]

Is it possible to use http/https proxy set in env var, or as a command-line option? I still can't get ytcog and ytcog-dl to work in my home, and I suspect it having something to do with the way I set up system-wide proxy/VPN. Maybe making ytcog do this itself will solve the problem.

[gatecrasher777]

There is no means to use a proxy in ytcog-dl at present. But I will add a --proxy option in the next update.

You can use a proxy in ytcog by providing an agent object in the minigetOptions.

var ProxyAgent = require('proxy-agent');
let proxyUri = process.env.http_proxy || 'http://168.63.43.102:3128';
let minigetOptions = { new ProxyAgent(proxyUri) };
let session = new ytcog.Session(minigetOptions);

[sclsj]

Thank you very much! I will try to modify my code. (Do I just paste it somewhere (in the beginning?) or do I need to do something with the session?) Here is the output from the ytcog-dl command that just stalls for ~ 1 hour on my laptop:

ytcog-dl --download rB83DpBJQsE

Video info saved to ./3Blue1Brown_2019-09-17 152256_Divergence and curl  The language of Maxwell's equations, fluid flow, and more_rB83DpBJQsE_info.json
Raw video json saved to ./3Blue1Brown_2019-09-17 152256_Divergence and curl  The language of Maxwell's equations, fluid flow, and more_rB83DpBJQsE_info_raw.json
internal/child_process/serialization.js:118
    const string = JSONStringify(message) + '\n';
                   ^

TypeError: Converting circular structure to JSON
    --> starting at object with constructor 'Object'
    --- property 'issuerCertificate' closes the circle
    at stringify (<anonymous>)
    at writeChannelMessage (internal/child_process/serialization.js:118:20)
    at process.target._send (internal/child_process.js:784:17)
    at process.target.send (internal/child_process.js:682:19)
    at send (/usr/local/lib/node_modules/ytcog-dl/node_modules/ytcog/lib/dl.js:19:28)
    at Download.send (/usr/local/lib/node_modules/ytcog-dl/node_modules/ytcog/lib/dl.js:288:3)
    at Download.dataError (/usr/local/lib/node_modules/ytcog-dl/node_modules/ytcog/lib/dl.js:122:8)
    at PassThrough.emit (events.js:315:20)
    at ClientRequest.onError (/usr/local/lib/node_modules/ytcog-dl/node_modules/miniget/dist/index.js:152:24)
    at ClientRequest.emit (events.js:315:20)

[sclsj]

Here is my modified code, and it did not work (it did work for the sample video in the sample code, but not for the video I want to download in my code). Maybe I should just wait until the next release of ytcog-dl?

const ytcog = require('ytcog');
var ProxyAgent = require('proxy-agent');
let proxyUri = process.env.http_proxy || 'http://127.0.0.1:1087';
let minigetOptions = { agent: new ProxyAgent(proxyUri) };
let session = new ytcog.Session(minigetOptions);

let videoOptions = {
id: 'rB83DpBJQsE',
filename: '${channelId} - ${datetime} - ${id} - ${videoQuality}',
container: 'mkv',
videoQuality: 'highest',
audioQuality: 'highest',
mediaBitrate: 'highest',
metadata: 'author,title,genre=sport,year=2021'
}
ytcog.dl(videoOptions,minigetOptions);

_http_client.js:134
    throw new ERR_INVALID_ARG_TYPE('options.agent',
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "options.agent" property must be one of Agent-like Object, undefined, or false. Received an instance of Object
    at new ClientRequest (_http_client.js:134:11)
    at Object.request (https.js:310:10)
    at doDownload (/private/var/root/ytd/node_modules/miniget/dist/index.js:174:33)
    at processTicksAndRejections (internal/process/task_queues.js:79:11) {
  code: 'ERR_INVALID_ARG_TYPE'
}

[gatecrasher777]

For ytcog.dl, you need to provide cookie and userAgent strings (even if empty) if you provide minigetOptions. So....

ytcog.dl(videoOptions, '', '', minigetOptions);

also you don't need

let session = new ytcog.Session(minigetOptions);

if you use the ytcog.dl function.

[gatecrasher777]

Can't replicate ytcog-dl issue, but I will look into the error message you produced. My output:

C:\Dev\ytcog-dl>ytcog-dl --download rB83DpBJQsE

Video info saved to ./3Blue1Brown_2019-09-17 152256_Divergence and curl  The language of Maxwell's equations, fluid flow, and more_rB83DpBJQsE_info.json
Raw video json saved to ./3Blue1Brown_2019-09-17 152256_Divergence and curl  The language of Maxwell's equations, fluid flow, and more_rB83DpBJQsE_info_raw.json
Downloading... 100.0% 260.4MB @ 8.209MB/s
Downloaded Media saved to:
./3Blue1Brown_2019-09-17 152256_Divergence and curl  The language of Maxwell's equations, fluid flow, and more_rB83DpBJQsE_1080p60_vp9_opus.webm

[sclsj]

For ytcog.dl, you need to provide cookie and userAgent strings (even if empty) if you provide minigetOptions. So....

ytcog.dl(videoOptions, '', '', minigetOptions);

also you don't need

let session = new ytcog.Session(minigetOptions);

if you use the ytcog.dl function.

I actually did that after I read the documentation and your replies in ytcog issue. Still the same error.

[sclsj]

Code:

const ytcog = require('ytcog');
var ProxyAgent = require('proxy-agent');
let proxyUri = process.env.http_proxy || 'http://127.0.0.1:1087';
let minigetOptions = { agent: new ProxyAgent(proxyUri) };

let videoOptions = {
id: 'rB83DpBJQsE',
filename: '${channelId} - ${datetime} - ${id} - ${videoQuality}',
container: 'mkv',
videoQuality: 'highest',
audioQuality: 'highest',
mediaBitrate: 'highest',
metadata: 'author,title,genre=sport,year=2021'
}
ytcog.dl(videoOptions,'','',minigetOptions);

Output:

{
  "minigetOptions": {
    "agent": {
      "proxy": {
        "protocol": "http:",
        "slashes": true,
        "auth": null,
        "host": "127.0.0.1:1087",
        "port": "1087",
        "hostname": "127.0.0.1",
        "hash": null,
        "search": null,
        "query": null,
        "pathname": "/",
        "path": "/",
        "href": "http://127.0.0.1:1087/"
      },
      "proxyUri": "http://127.0.0.1:1087"
    }
  },
  "id": "rB83DpBJQsE",
  "channelId": "UCYO_jab_esuFRV4b17AJtAw",
  "duration": 942,
  "views": 2546148,
  "rating": 4.971737,
  "updated": "now",
  "published": "2y1m ago",
  "title": "Divergence and curl:  The language of Maxwell's equations, fluid flow, and more",
  "category": "Education",
  "description": "Visualizing two core operations in calculus.  (Small error correction below) \nHelp fund future projects: https://www.patreon.com/3blue1brown\nAn equally valuable form of support is to simply share some of the videos.\nSpecial thanks to these supporters: http://3b1b.co/divcurl-thanks\n\nMy work on this topic at Khan Academy: https://www.khanacademy.org/math/multivariable-calculus/multivariable-derivatives\n\nError: At 4:55, the narration should say \"counterclockwise rotation gives positive curl, clockwise rotation gives negative curl\".  The diagram is correct, though.\n\nFor more fun fluid-flow illustrations, which heavily influenced how I animated this video, I think you'll really enjoy this site: \nhttps://anvaka.github.io/fieldplay/\n\nMusic by Vincent Rubinetti: \nhttps://vincerubinetti.bandcamp.com/album/the-music-of-3blue1brown\n\n------------------\n\n3blue1brown is a channel about animating math, in all senses of the word animate.  And you know the drill with YouTube, if you want to stay posted on new videos, subscribe, and click the bell to receive notifications (if you're into that).\n\nIf you are new to this channel and want to see more, a good place to start is this playlist: http://3b1b.co/recommended\n\nVarious social media stuffs:\nWebsite: https://www.3blue1brown.com\nTwitter: https://twitter.com/3blue1brown\nPatreon: https://patreon.com/3blue1brown\nFacebook: https://www.facebook.com/3blue1brown\nReddit: https://www.reddit.com/r/3blue1brown",
  "author": "3Blue1Brown",
  "keywords": [
    "brown",
    "complex analysis",
    "multivariable calculus",
    "vector calculus",
    "three blue one brown",
    "three brown one blue",
    "3 blue 1 brown",
    "3 brown 1 blue",
    "Mathematics",
    "3b1b",
    "one",
    "divergence",
    "three",
    "blue",
    "3brown1blue",
    "curl"
  ]
}
_http_client.js:134
    throw new ERR_INVALID_ARG_TYPE('options.agent',
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "options.agent" property must be one of Agent-like Object, undefined, or false. Received an instance of Object
    at new ClientRequest (_http_client.js:134:11)
    at Object.request (https.js:310:10)
    at doDownload (/private/var/root/ytd/node_modules/miniget/dist/index.js:174:33)
    at processTicksAndRejections (internal/process/task_queues.js:79:11) {
  code: 'ERR_INVALID_ARG_TYPE'
}

[sclsj]

Also, I remember having to install proxy-agent after specifying proxy:

internal/modules/cjs/loader.js:955
  throw err;
  ^

Error: Cannot find module 'proxy-agent'
Require stack:
- /private/var/root/ytd/video.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:952:15)
    at Function.Module._load (internal/modules/cjs/loader.js:835:27)
    at Module.require (internal/modules/cjs/loader.js:1012:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Object.<anonymous> (/private/var/root/ytd/video.js:1:18)
    at Module._compile (internal/modules/cjs/loader.js:1123:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1143:10)
    at Module.load (internal/modules/cjs/loader.js:972:32)
    at Function.Module._load (internal/modules/cjs/loader.js:872:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [ '/private/var/root/ytd/video.js' ]
}

npm install proxy-agent
npm WARN saveError ENOENT: no such file or directory, open '/private/var/root/ytd/package.json'
npm WARN enoent ENOENT: no such file or directory, open '/private/var/root/ytd/package.json'
npm WARN ytd No description
npm WARN ytd No repository field.
npm WARN ytd No README data
npm WARN ytd No license field.

+ proxy-agent@5.0.0
added 50 packages from 76 contributors and audited 100 packages in 19.36s
found 0 vulnerabilities

Maybe that has something to do with the error?

[gatecrasher777]

To get a good nodejs environment, start with an empty folder. Then npm init . Then npm install ytcog. Then npm install proxy-agent. Then copy your script into it.

But despite the warnings it looks okay. The error is being raised in miniget which is a different project.
I guess I'm going to have to play around a bit with proxies and see if I can replicate.

[gatecrasher777]

Try this:

let agent = new ProxyAgent(proxyUri)
let minigetOptions = { agent };

instead of

let minigetOptions = { agent: new ProxyAgent(proxyUri) };

You can also look at this https://github.com/fent/node-ytdl-core/blob/master/example/proxy.js which also uses miniget. So minigetOptions is similar to requestOptions in that project.

[sclsj]

I tried the example you gave me. Here's the code:

const ytcog = require('ytcog');
const HttpsProxyAgent = require('https-proxy-agent');
const proxy = 'http://127.0.0.1:1087';
const agent = HttpsProxyAgent(proxy);
let minigetOptions = { agent };

let videoOptions = {
id: 'rB83DpBJQsE',
filename: '${channelId} - ${datetime} - ${id} - ${videoQuality}',
container: 'mkv',
videoQuality: 'highest',
audioQuality: 'highest',
mediaBitrate: 'highest',
metadata: 'author,title,genre=sport,year=2021'
}
ytcog.dl(videoOptions,'','',minigetOptions);

And it did not work.

_http_client.js:134
    throw new ERR_INVALID_ARG_TYPE('options.agent',
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "options.agent" property must be one of Agent-like Object, undefined, or false. Received an instance of Object
    at new ClientRequest (_http_client.js:134:11)
    at Object.request (https.js:310:10)
    at doDownload (/private/var/root/ytd/node_modules/miniget/dist/index.js:174:33)
    at processTicksAndRejections (internal/process/task_queues.js:79:11) {
  code: 'ERR_INVALID_ARG_TYPE'
}

[sclsj]

And the correction you gave me:

const ytcog = require('ytcog');
var ProxyAgent = require('proxy-agent');
let proxyUri = process.env.http_proxy || 'http://127.0.0.1:1087';
let agent = new ProxyAgent(proxyUri)
let minigetOptions = { agent };

let videoOptions = {
id: 'rB83DpBJQsE',
filename: '${channelId} - ${datetime} - ${id} - ${videoQuality}',
container: 'mkv',
videoQuality: 'highest',
audioQuality: 'highest',
mediaBitrate: 'highest',
metadata: 'author,title,genre=sport,year=2021'
}
ytcog.dl(videoOptions,'','',minigetOptions);

Did not work.

_http_client.js:134
    throw new ERR_INVALID_ARG_TYPE('options.agent',
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "options.agent" property must be one of Agent-like Object, undefined, or false. Received an instance of Object
    at new ClientRequest (_http_client.js:134:11)
    at Object.request (https.js:310:10)
    at doDownload (/private/var/root/ytd/node_modules/miniget/dist/index.js:174:33)
    at processTicksAndRejections (internal/process/task_queues.js:79:11) {
  code: 'ERR_INVALID_ARG_TYPE'
}

[sclsj]

Thank you for your help!

[gatecrasher777]

I've replicated your error. The proxy works to collect the video info. The problem is that the proxy is not initialized in the stand-alone downloader - just passing the object over IPC doesn't work. Will release a fix in the next few days for ytcog and --proxy option for ytcog-dl. The new version will have --subtitle options as well.

[gatecrasher777]

You code would now be:

const ytcog = require('ytcog');
let proxy = 'http://127.0.0.1:1087';

let videoOptions = {
  id: 'rB83DpBJQsE',
  filename: '${channelId} - ${datetime} - ${id} - ${videoQuality}', 
  container: 'mkv',
  videoQuality: 'highest',
  audioQuality: 'highest',
  mediaBitrate: 'highest',
  metadata: 'author,title,genre=sport,year=2021'
}

ytcog.dl(videoOptions,'','',proxy);

and with ytcog-dl on the command line:

media>ytcog-dl -d rB83DpBJQsE -x "http://127.0.0.1:1087"

[sclsj]

Thank you! This worked.

[sclsj]

However, adding --metadata failed. Is my command incorrect?

ytcog-dl --metadata description -d P6C3szts5C4 -x 'http://127.0.0.1:1087'

TypeError: Cannot read property 'length' of undefined
    at getMetadata (/usr/local/lib/node_modules/ytcog-dl/index.js:416:53)
    at Command.run (/usr/local/lib/node_modules/ytcog-dl/index.js:473:36)
    at Object.<anonymous> (/usr/local/lib/node_modules/ytcog-dl/index.js:604:5)
    at Module._compile (internal/modules/cjs/loader.js:1123:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1143:10)
    at Module.load (internal/modules/cjs/loader.js:972:32)
    at Function.Module._load (internal/modules/cjs/loader.js:872:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47

ytcog-dl -d P6C3szts5C4 -x 'http://127.0.0.1:1087' --metadata description
TypeError: Cannot read property 'length' of undefined
    at getMetadata (/usr/local/lib/node_modules/ytcog-dl/index.js:416:53)
    at Command.run (/usr/local/lib/node_modules/ytcog-dl/index.js:473:36)
    at Object.<anonymous> (/usr/local/lib/node_modules/ytcog-dl/index.js:604:5)
    at Module._compile (internal/modules/cjs/loader.js:1123:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1143:10)
    at Module.load (internal/modules/cjs/loader.js:972:32)
    at Function.Module._load (internal/modules/cjs/loader.js:872:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47

ytcog-dl --metadata --subtitles -d P6C3szts5C4 -x 'http://127.0.0.1:1087'
TypeError: Cannot read property 'length' of undefined
    at getMetadata (/usr/local/lib/node_modules/ytcog-dl/index.js:416:53)
    at Command.run (/usr/local/lib/node_modules/ytcog-dl/index.js:473:36)
    at Object.<anonymous> (/usr/local/lib/node_modules/ytcog-dl/index.js:604:5)
    at Module._compile (internal/modules/cjs/loader.js:1123:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1143:10)
    at Module.load (internal/modules/cjs/loader.js:972:32)
    at Function.Module._load (internal/modules/cjs/loader.js:872:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47

[sclsj]

ytcog-dl --subtitles -d P6C3szts5C4 -x 'http://127.0.0.1:1087'
Error: Unrecognised input P6C3szts5C4
    at getId (/usr/local/lib/node_modules/ytcog-dl/index.js:456:12)
    at Command.run (/usr/local/lib/node_modules/ytcog-dl/index.js:501:15)
    at Object.<anonymous> (/usr/local/lib/node_modules/ytcog-dl/index.js:604:5)
    at Module._compile (internal/modules/cjs/loader.js:1123:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1143:10)
    at Module.load (internal/modules/cjs/loader.js:972:32)
    at Function.Module._load (internal/modules/cjs/loader.js:872:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47

[gatecrasher777]

There are a few problem with your ytcog-dl commands:

The action -d must be the first argument. Id next. Then options. --subtitles requires a string of the language for the subtitles. One or more of language code . ie --subtitles "en, zh-Hans, ja" On the command line you need to use " instead of '. so -x "http://127.0.0.1:1087". My bad for posting that. I'll fix that in the original post for future readers.

However, I did pick up a bug with parsing the --metadata option. I'll push a fix through shortly. Look for v1.1.1.