Advanced search

[Chocobozzz]

For now, we search on the video name, account username and account host fields (substring search).

It could be interesting to search on:

• Tags (exact search with disambiguation)
• Category
• Licence
• Language

[imtube]

Perfect ! Search is very important!

[nvivant]

Accent insensitive search would be interesting as well, so that videos with "février" are displayed when you request "fevrier" (and the other way around).

[Serkan-devel]

Has anyone considered using a combination of yacy and elasticsearch as a decentralized search engine?

[rigelk]

@Serkan-devel yacy and elasticsearch are two different search mechanisms with partially overlapping functionalities. I'm not sure what you suggest we do with them. Especially since they're added external dependencies.

[Serkan-devel]

yacy is a decentralized search engine and elastic can scope the search results better

[lanodan]

I think you can avoid requiring such heavy dependencies by using the already included fuzzy match of PostgreSQL. (It’s used by pleroma btw and it’s more than enough for a local search) Related documentation: https://www.postgresql.org/docs/9.2/static/pgtrgm.html

[Chocobozzz]

Thanks @lanodan, I think we will use this :)

[ghost]

How I see the Advanced search work:

• declare all variables such as tags, video title, video description, comments, license, language, time published, keywords (everything that's on a page)
• group them into defined and undefined. Defined are those with limits, like for example we have exactly 4 types of licenses or exactly 45 languages as a total from which people can choose. And undefined are all the others.
• declare rules, such as: --should include --should exclude --should range in between
• declare rule types, such as: --exact match --not exact match --begins with --ends with

An example of how it'll work in a text-based format, by typing this command directly into the search bar: 1..4 cats "green" -"yellow" -dog daterange:2455332-2455334 language:fr license:cc tag:cat tag:paw -tag:dog -intitle:*bad indescription:good

explanaition:

• 1..4 - number range
• cats - is a broad keyword
• "green" - keyword in quotes means it's an exact keyword that has to be present somewhere on that page
• -"yellow" - keyword in quotes and with a - sign meaning that this exact keyword should not be present on the page
• -dog - means this broad keyword should not be present
• daterange:2455332-2455334 - means we are looking for a page created in between these dates, in Julian format.
• language:fr - language
• license:cc - I'm not sure if a search operator like this exists, and so I came up with it, if there is a standard one, we better use that. so here it means it should be licenced cc
• tag:cat & tag:paw - it must have these tags
• -tag:dog - it must not have this tag
• -intitle:*bad - in the page/video title should not be any words that end with these letters "bad"
• indescription:good - again, I'm not sure if a search operator like this exists, and so I came up with it, if there is a standard one, we better use that. In this example it means that the page/video description should have the broad keyword "good".

This is only the text-based way of searching, a GUI one is better to have as well, for less technical people.

As a reference we can see how Search Engine Optimization (SEO) is doing it, we don't necessarily have to re-coin the wheel :) Another good resource here. Here's how google is doing it, but other Search Engines do it too, for example in duckduckgo you can refine search results with the same commands, like: cat -dogs insite:ututu.com "go" so it looks like a "standard" and I believe we should not deviate from it.

[lanodan]

I think you should drop the in on intitle and indescription so it’s an actual word and it’s more consistent.

[Chocobozzz]

@Zig-03 Thanks, but a search string is a little bit too complicated for now. I tried to make something we can extend easily based on query parameters: https://github.com/Chocobozzz/PeerTube/pull/829. And nothing prevents in the future to create your search parser, as I agree this is a very powerful and flexible way to query data.

[Serkan-devel]

Will advanced search also fetch data from other peertube instances?

[Chocobozzz]

@Serkan-devel No, only local videos and videos from instances followed by your admin (for now). See https://github.com/Chocobozzz/PeerTube/issues/824 for search on the entire fediverse

[georgejdanforth]

I reckon this could be made to work well with Postgres' full text search capabilities. Perhaps there could be a lightweight table that contains tsvectors of tokenized video titles, tags, (transcriptions ultimately?), and references to video locations that may span multiple hosts. The tricky part would be replicating this table between hosts, but if that could be achieved then there would be host-independent search.

https://www.postgresql.org/docs/10/static/textsearch.html

[Chocobozzz]

Implemented in https://github.com/Chocobozzz/PeerTube/commit/57c36b277e68b764dd34cb2e449f6e2ca3d1e9b6 & https://github.com/Chocobozzz/PeerTube/commit/0b18f4aa80df8868bf34605423c7a298dffbb2aa & https://github.com/Chocobozzz/PeerTube/commit/d525fc399a14a8b16eaad6d4c0bc0a9c4093c3c9

[lanodan]

Is there a reason why there is no global search? I mean at least with peered videos.

[Serkan-devel]

@lanodan maybe the instances need to be updated for that. This has been freshly commited

[lanodan]

That would be weird, I mean the local instance already got the metadata of the video that is given with ActivityPub, right?

EDIT: Nevermind, I must have misunderstood the #824 issue…

[Serkan-devel]

@lanodan I don't have an instance, so I'm not shure

[Chocobozzz]

Is there a reason why there is no global search? I mean at least with peered videos.

I don't understand, the search occurs on all videos of instances followed by your admin (so peered videos too).