Podcast and Videos search

[tomastiminskas]

We have 2 endpoints for podcast and video search - GET search_podcasts - GET search_youtube

Right now they are only searching by title.

1. As a first improvements we should check the API we are using on Podcast Index and Youtube and find a way to improve them (maybe adding some fields on the search)

2. As a long term solution we should start storing the results coming from external searches in our system and build an elastic search index so then we can first search in our index and then get the content of the found results in the external source.

[tomastiminskas]

Search APIs research

YouTube API search

YouTube search API allows episodes level search. We are currently using type = playlist, channel, but there's one more option for type called video which will allow the episodes level search.

What I found playing around with the API is that if you set the 3 types together, the api will return mostly videos, so you won't get playlist or channel results at all (at least in the first 50 results which are the ones we are showing in Sphinx). So we should do some UI switch in the app to search PLAYLIST/CHANNELS or VIDEOS

Podcast Index API search

There seems to be no way to search by episodes title.

As it's visible in the image, the only episode level search available in the api is searching episodes by author. So in this case the only option we would have is start storing data in tribes server db and then use that cached data when doing an episodes level search (maybe create an elastic search index).

We could keep using podcast index search api when doing podcast level search and store data on our side every time a call to the api is done. Then when doing episodes level search use our data.

On any case we will need a switch in the app as well (PODCAST or EPISODES)

[Evanfeenstra]

Testing the byperson search endpoint, it seems like it can search any term (not just names). https://podcastindex-org.github.io/docs-api/#get-/search/byperson

[tomastiminskas]

@Evanfeenstra as I mentioned to you the byPerson endpoint finally searches just by person. Some words seem to return results like "bitcoin", but most of them won't return any. If you search by names like "Adam" it will return results, so I asked Milan to adapt the designs to that. Just writing this here to have all the information in one place.

In the other hand Alejandro has been building a first implementation of YouTube videos search on Android and he found something interesting. When we do a videos level search on YouTube, we get the feed url on each item so then when selecting a search results we can fetch that entire feed and present the player for it with the video (selected on the search results) as the video to be played immediately. The thing is that most of the times the video on the search results is not contained in the items of that feed when fetching the entire feed using the RSS feed url, and this happens because that RSS feed is limited to 15 results

Here is an article that explain that and how to solve it. It seems we need to start using YouTube Data API v3 to fetch the feed items. If you can take a look and see if it's posible to replace the call to the RSS feed by a call to this api I think we should do it, since otherwise we will be always showing limited information about YouTube channel/playlist to the users (just 15 videos when it could contain many more)

https://stackoverflow.com/questions/42258132/fetching-videos-list-from-a-youtube-playlist-through-rss-doesnt-return-more-tha

[Evanfeenstra]

hmm ya luckily we are already using api v3. But we need to switch from Search.List to PlaylistItems.List https://developers.google.com/youtube/v3/docs/playlistItems/list

[tomastiminskas]

@Evanfeenstra I think you are confused. I'm not talking about the search endpoint, but the /feed?url endpoint. The one we are using to fetch the feed from the RSS feed. So when you select a search result from the new searches (like a YouTube video) we hit the endpoint we already had there to fetch the entire feed and show the player for that feed (preselecting that video). But as the RSS feed on YouTube is limited to 15 results, that video can be not included in the feed items.

https://github.com/stakwork/sphinx-tribes/blob/1ef010ca0d7358f4191c92f385b5a7a42f24d2c1/feeds/generator.go#L20

I think that's the method where the server basically does a GET to the url sent in the /feed?url endpoint param. This is what we will need to replace to API v3 data api.

Let me know if I'm wrong and we can jump on a call

[tomastiminskas]

@Evanfeenstra let's actually hold on on this since Paul thinks the search by person is confusing and it doesn't worth it to implement it just on YouTube. He will try to talk to someone on Podcast Index to check if they are planning to release an episode level search soon