Unable to add channel by URL

[NightMean]

Hey,

I wanted to add a new channel https://www.youtube.com/c/HomesystemSk to whitelist but when I used the URL from browser I got an error message below. Also tried to use https://www.youtube.com/channel/HomesystemSk but ended up with the same error message. Could you take a look into it?

Sep 18 20:23:26 LinPlex systemd[1]: Started YouTube email notification system from Github - bbyen.
Sep 18 20:23:27 LinPlex node[9864]: [error] [main]: Exhausted all methods of getting the ID for channel 'https://www.youtube.com/c/HomesystemSk'. If this is a mistake, please open a GitHub issue and show them this message: https://github.com/MarcelRobitaille/bbyen/issues/new {}
Sep 18 20:23:27 LinPlex node[9864]: Error: Exhausted all methods of getting the ID for channel 'https://www.youtube.com/c/HomesystemSk'. If this is a mistake, please open a GitHub issue and show them this message: https://github.com/MarcelRobitaille/bbyen/issues/new
Sep 18 20:23:27 LinPlex node[9864]:     at normalizeChannel (/opt/YoutubeNotify-bbyen/src/config.js:46:9)
Sep 18 20:23:27 LinPlex node[9864]:     at Array.map (<anonymous>)
Sep 18 20:23:27 LinPlex node[9864]:     at normalizeConfig (/opt/YoutubeNotify-bbyen/src/config.js:57:46)
Sep 18 20:23:27 LinPlex node[9864]:     at loadConfig (/opt/YoutubeNotify-bbyen/src/config.js:65:33)
Sep 18 20:23:27 LinPlex node[9864]:     at async main (/opt/YoutubeNotify-bbyen/src/index.js:21:18)

[MarcelRobitaille]

That's strange. I just tested that string and it is being matched properly. Could you please check if there are any invisible characters or anything you might have accidentally copied into the configuration file? If that does not work, could you please try the latest version and set your log level in your configuration file to verbose? I added some additional logging.

[NightMean]

Hey, sorry for late reply. I think there's something wrong with the latest commit. I received an error right upon startup. Log level is set to verbose. I've setup bbyen as a fresh install only with my auth token, config and db copied into the directory.

node src/index.js
node:internal/validators:120
    throw new ERR_INVALID_ARG_TYPE(name, 'string', value);
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined
    at new NodeError (node:internal/errors:372:5)
    at validateString (node:internal/validators:120:11)
    at Module.require (node:internal/modules/cjs/loader:998:3)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/opt/tempfolder/bbyen/src/lib/logger.js:4:16)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19) {
  code: 'ERR_INVALID_ARG_TYPE'
}

Edit: Yes, it is broken with the latest logging commit 8e37064aa83572b19fd6a6a648f1a5b747a4d535 Invisible characters are not there.

[MarcelRobitaille]

Thanks for checking that commit. This new error is puzzling. Could you please share your entire configuration file?

[NightMean]

Sure, I've changed the extension to .txt as GitHub doesn't like json files.

Here: Config.json.txt

[MarcelRobitaille]

I think it was just a dependency loop and an unhelpful error message. Would you try the latest commit please?

[NightMean]

Sure, Unfortunately it still throws the same error. Using latest commit 236e849d93518ef0a8232815bf4d016751bf8445

2022-09-28T18:36:48.300Z [verbose] [config]: Normalizing channel string 'UC2J-0g_nxlwcD9JBK1eTleQ' {}
2022-09-28T18:36:48.300Z [verbose] [config]: String matches ID {}
2022-09-28T18:36:48.300Z [verbose] [config]: Normalizing channel string 'https://www.youtube.com/channel/HomesystemSk' {}
2022-09-28T18:36:48.301Z [error] [main]: Exhausted all methods of getting the ID for channel 'https://www.youtube.com/channel/HomesystemSk'. If this is a mistake, please open a GitHub issue and show them this message: https://github.com/MarcelRobitaille/bbyen/issues/new {}
Error: Exhausted all methods of getting the ID for channel 'https://www.youtube.com/channel/HomesystemSk'. If this is a mistake, please open a GitHub issue and show them this message: https://github.com/MarcelRobitaille/bbyen/issues/new
    at normalizeChannel (/opt/tempfolder/bbyen/src/config.js:76:9)
    at Array.map (<anonymous>)
    at normalizeConfig (/opt/tempfolder/bbyen/src/config.js:87:46)
    at loadConfig (/opt/tempfolder/bbyen/src/config.js:96:33)
    at async main (/opt/tempfolder/bbyen/src/index.js:21:18)

[MarcelRobitaille]

I downloaded you config and I was able to reproduce your issue for https://www.youtube.com/channel/HomesystemSk. The latest commit should support that.

However, I was not able to reproduce the issue for https://www.youtube.com/c/HomesystemSk. Could you please try the latest commit and let me know if you are still getting that error?

I am using node 16.2.0, f89dbf67, and I used your config.json exactly.

[NightMean]

Unfortunately yeah I still do receive the same error.

I'm using node v16.15.1, could that be the issue?

2022-09-30T07:06:59.439Z [verbose] [config]: Normalizing channel string 'UC2J-0g_nxlwcD9JBK1eTleQ' {}
2022-09-30T07:06:59.439Z [verbose] [config]: String matches ID {}
2022-09-30T07:06:59.439Z [verbose] [config]: Normalizing channel string 'https://www.youtube.com/channel/HomesystemSk' {}
2022-09-30T07:06:59.440Z [verbose] [config]: String matches URL with channel name {}
2022-09-30T07:06:59.960Z [error] [main]: Could not find channel ID using the API for 'https://www.youtube.com/channel/HomesystemSk'. Please open a GitHub issue and show them this message: https://github.com/MarcelRobitaille/bbyen/issues/new {}
Error: Could not find channel ID using the API for 'https://www.youtube.com/channel/HomesystemSk'. Please open a GitHub issue and show them this message: https://github.com/MarcelRobitaille/bbyen/issues/new
    at normalizeChannel (/opt/tempfolder/bbyen/src/config.js:72:11)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 316)
    at async normalizeConfig (/opt/tempfolder/bbyen/src/config.js:88:6)
    at async loadConfig (/opt/tempfolder/bbyen/src/config.js:97:27)
    at async main (/opt/tempfolder/bbyen/src/index.js:21:18)

Same error for /c/HomesystemSK

2022-09-30T07:10:34.069Z [verbose] [config]: Normalizing channel string 'UC2J-0g_nxlwcD9JBK1eTleQ' {}
2022-09-30T07:10:34.069Z [verbose] [config]: String matches ID {}
2022-09-30T07:10:34.069Z [verbose] [config]: Normalizing channel string 'https://www.youtube.com/c/HomesystemSk' {}
2022-09-30T07:10:34.069Z [verbose] [config]: String matches URL with channel name {}
2022-09-30T07:10:34.502Z [error] [main]: Could not find channel ID using the API for 'https://www.youtube.com/c/HomesystemSk'. Please open a GitHub issue and show them this message: https://github.com/MarcelRobitaille/bbyen/issues/new {}
Error: Could not find channel ID using the API for 'https://www.youtube.com/c/HomesystemSk'. Please open a GitHub issue and show them this message: https://github.com/MarcelRobitaille/bbyen/issues/new
    at normalizeChannel (/opt/tempfolder/bbyen/src/config.js:72:11)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 316)
    at async normalizeConfig (/opt/tempfolder/bbyen/src/config.js:88:6)
    at async loadConfig (/opt/tempfolder/bbyen/src/config.js:97:27)
    at async main (/opt/tempfolder/bbyen/src/index.js:21:18)

[MarcelRobitaille]

That is interesting. This is actually a different error message. Notice String matches URL with channel name that is now present. That means that BBYEN is not correctly detecting that you have a channel URL with a custom name.

The new error message Could not find channel ID using the API, which is different than before, is saying that it was not possible to find the channel ID for that URL with the YouTube API.

Searching for this channel URL seems to give no results in the YouTube data API.

Would you mind manually getting the ID for this channel using this method for now and I will work on adding a more reliable method that works for all channel URLs in the future?

[NightMean]

Aah, that would make sense. How did it work for you though? I have no issue adding the URL from the RSS method as I did previously, just wanted to test it using this method :)

[MarcelRobitaille]

I only fixed it to make the first error message you reported go away. I don't have the google auth stuff set up in my development environment right now, so I didn't get past that part.

I will try to add something for that other method.

[NightMean]

Ahh okay, I misunderstood. Thanks!

[MarcelRobitaille]

The latest commit c27fa4c09aa6bca37c998a51468703a3669575ab adds an alternative method to get the channel ID from the URL by parsing the ytInitalData object like this method. Both methods are still implemented, and the first method to return a valid result will be used. I have tested for the URL https://www.youtube.com/c/HomesystemSk and it is working. @NightMean Would you mind testing that commit?

[NightMean]

Works perfectly. The URL got converted to the channel ID without any issue. 2022-10-04T15:03:47.779Z [verbose] [videos]: Checking channel HomeSystem.sk (UCVOWBEMYDcDOYo_2L2zHrqA) {} Thanks a lot!

Quick question, I've seen the commit e615487398b5f48de111f02cb9b7c674e52ecba1. Does that makes it so I'll receive email whenever Google decides to refuse my API credentials e.g as mentioned in https://github.com/MarcelRobitaille/bbyen/issues/7?

[MarcelRobitaille]

Unfortunately no. It only sends emails about errors encountered after the authentication is set up, the config is validated, etc. The reason is that the email setup depends on the config, which depends on Google authentication (to get the channel IDs for channel URLs).

However, the purpose is to notify you if an email is encountered after the service has been running for a long time. If the authentication fails, you should know immediately after starting the service.

If the new method I added today to get the channel ID is reliable, maybe I can remove the YouTube API method, which would remove the config validation's dependency on Google authentication. That way, I could set up the config and email first and send email notifications on Google authentication errors.

[NightMean]

If the new method I added today to get the channel ID is reliable, maybe I can remove the YouTube API method, which would remove the config validation's dependency on Google authentication. That way, I could set up the config and email first and send email notifications on Google authentication errors.

That would be perfect! Currently I have to re-authorize every week or so as I receive the invalid grant mentioned in #7. Currently I have no way of knowing if there wasn't any new video uploaded from the whitelisted channels or that the authorization failed again. The new method seems to work fine and reliable. I believe issue can be closed. Thank you

[MarcelRobitaille]

@NightMean Ah, you should still get emails if you get an invalid grant. That is related to authentication, but is not caused by the initial authentication setup itself. Rather, it's caused by making an API call with credentials that Google has invalidated for whatever reason. Please let me know if you get an email the next time that invalid grand error arises.

You should not be getting an invalid grant and having to re-authenticate every week. You can open an issue about it, but I think the issue lies with google and there is likely little I can do.

If you don't mind, I will close this for now.