Search page doesn't consistently find the video

parkeraddison commented 3 years ago

Problem

Currently, when we click 'Watch on Nebula' it brings us to a Nebula search page and queries the video title. Unfortunately, search queries submitted to Nebula don't consistently find the target video. This is due to a number of reasons:

Differences between video titles on YouTube and Nebula
Videos that are only on YouTube (e.g. #shorts)
Inconsistent behavior when querying the creator name

Examples

Creators may choose a different YouTube title, or change them frequently after posting, resulting in a mismatch Nebula YouTube
Some videos (I've noticed a trend with #shorts) are only for YouTube Nebula YouTube
In some cases, including the creator name in a search term causes zero results! Other times it works fine. Without With (broken) With (working)

Solutions

This is an open issue that accepts any potential solutions, please post if you have an idea!

Possible ideas:

Search for the creator name plus a keyword from the video title
- Could utilize a general term importance dataset to just pull out the most important words
- Is still affected by the creator name issue (e.g. querying "Real Engineering Hydrogen" shows zero results!)
Just link to the creator's Nebula page and let the user find the video
- Great for new videos
- Annoying for old videos
Query by creator and date-range (+- a few days from the YouTube publish date)
- Not supported by Nebula... (hopefully it will be!)
- Probably somewhat robust

What do you think?

Welteam commented 3 years ago

How I would improve on that (don't solve all issues): -Check if channel is part of nebula -Make the extension send the search query (I hope this is possible) -Parse the result to see if it found results -If yes, show pop up -If not, do nothing (maybe something to indicate nothing was found)

To that I would add a "Nebula" Button next to the channel name that would link to the nebula page. Maybe something like the subscribe button but blue.

Edit: the url of nebula videos is also just "channel name"+"video name" (scrapping all non alphanumerical characters and replacing spaces with hyphens). Maybe it can try this before the search query

Edit 2: some creators like medlife crisis already put the link to the video itself in the youtube description so this should be the first thing to check

parkeraddison commented 3 years ago

I really like your ideas! Thank you

After some exploring, here's what I found:

Checking for a direct link in the description would be a really smart idea -- this should be pursued

Adding a 'Nebula' button next to the subscribe button would be really fun! I think it's as simple as:

let sbr = document.querySelector('ytd-subscribe-button-renderer');
let nebButton = sbr.querySelector('paper-button').cloneNode();

nebButton.id = 'neb-button';

sbr.insertAdjacentElement('afterbegin', nebButton);
nebString.innerText = "Nebula";

plus a simple background color css rule and some js for setting the button action!

I think you just stumbled upon the reason for the creator-name weirdness:

the url of nebula videos is also just "channel name" + "video name"

That is true for some urls, e.g. https://watchnebula.com/videos/real-engineering-the-missing-link-in-renewables but not for others, e.g. https://watchnebula.com/videos/the-truth-about-hydrogen

(notice that those are the same videos shown in the example above! This must be some quirk of the backend search API)
As for conducting a query... loading a query page on Nebula submits a backend API request which populates the page once it returns. Therefore, the initial result when loading any query page is "No results".

That being said, I was able to use the network viewer when navigating to a /search page and discovered that if we wait for the "videos" API request to return, then we can tell if there truly are or aren't any results (pages will be zero if there are none)

Seems like a pretty involved approach...
Going to an invalid address at Nebula results in an empty page. I guess this is an indication that we probably wouldn't have found the video through the currently existing search (channel name + video title) anyway, so perhaps we can check this page first and if it's blank then we resort to a message saying "this might be on Nebula, you're welcome to go to the channel by clicking the button".

Overall, it looks like some nice and relatively easy additions would be:

[ ] Check the description for a link
[ ] Add a Nebula channel button next to the subscribe button

Since it seems that we're unlikely to find a video via search if the url doesn't match channel-name-video-title, then I suppose it would make sense to pursue that approach too.

Queries are still a bit of an issue, and I'm hesitant to spend any time on that because I thought I heard the backend was changing soon?

parkeraddison / watch-on-nebula