Search for chat history coming any time soon?

[FlxVctr]

Just tested the new team feature with some colleagues, and an important question arose:

What is the full chat history good for if there's no search?

Is search going to be implemented any time soon?

[NyanHelsing]

Is there such thing as an encrypted search index? Dropping the index and regenerating it when a user logs out/in sounds resource intensive if there's a lot of history.

[animepauly]

I second this question. If we can't search through chat history, the chat history isn't very helpful as stated above. Is there any timeline to this? Is this even going to be implemented? This makes it difficult to keep using keybase over Slack if we can't search for things that were discussed.

[mutagenfork]

Just adding my voice to those requesting the ability to search chat history.

[arjanski]

For me/us, a chat history search is a must-have as well. I cannot imagine using keybase on a daily basis not being able to quickly jump to a certain section/conversation that happened weeks ago.

[herwix]

Second this request. We are using keybase for organizing our team and love it so far but search should be highly prioritized to keep users onboard :) It's one of those must have features!

[RRgermany]

Yes! Keybase needs a chat history search.

[jcausey-astate]

+1 from me. We have been using Keybase chat for small groups (~2 people at at time), and I would love to roll it out to our larger group... But being able to search the chat history is a requirement before I can do that. Works great otherwise!

[ajankit]

+1 here there is absolutely no way we can use this service without having an ability to search through chat history. I've convinced my co-workers to give it a try and I think we'll have to abandon it

UPDATE: Just wanted to chime in again and say my team has abandoned Keybase due to its lack of search functionality. If you work for Keybase and see this, please add the option to search chats!! I want to use your service!

[NyanHelsing]

Hey does anyone have thoughts about how this would be implemented? It might be an easier feature to put on a road map if it was bikeshedded a little bit.

[maxtaco]

Already exists on the CLI

cc @joshblum and @mmaxim

[joshblum]

CLI usage:

$ keybase chat search --help
NAME:
   keybase chat search - Search via regex within a conversation

USAGE:
   keybase chat search [command options] <conversation> <query>

OPTIONS:
   --topic-type "chat"  Specify topic type of the conversation. Has to be chat or dev
   --channel            Specify the conversation channel.
   --public         Only select public conversations. Exclusive to --private
   --private            Only select private conversations. Exclusive to --public. If both --public and --private are present, --private takes priority.
   -r, --regex          Make the given query a regex
   --max-hits "10"      Specify the maximum number of search hits to get. Default is 10
   --max-messages "10000"   Specify the maximum number of messages to search. Default is 10000

[WTurunen]

Do you have a timeline estimate for the search implementation on the desktop client?

[digininja]

+1 just been looking for it assuming it was an obvious feature that must be there but it isn't.

[skrish13]

@joshblum even the CLI usage is restricted to a particular conversation? can it be global? is there an option? (like whatsapp)

[gavin-s-smith]

+1 for this feature.

A comment to add to what has already been posted in case it is of any use: In addition to keybase chat search which in my mind is limited in utility (since it only gives 1 line of context either side of the found line of chat history and there is no parameter to change this) there is also the json api keybase chat api h which could be used to search, return and explore a search result.

[joshblum]

@gavin-s-smith parametrizing the search context is a good point and will help with CLI usability.

@skrish13 right now the search is restricted to the conversation you specify, I'll update the issue if this changes.

cc @malgorithms

[dfucci]

+1

[VolodymyrBiryuk]

+1 Yes please

[pborawski78]

So due to lack of search and not much word from the devs on when and if it will be in the GUI, plus the recent notification bugs my team is seeing with the latest updates, we are deciding to leave Keybase. Hope this team figure this out because it could be a good product but is more cumbersome than helpful and lack of search is really painful.

[maxtaco]

Oh, sorry not to reply sooner. In progress.

[joshblum]

@pborawski78 fwiw full inbox chat search in the CLI just got into master with https://github.com/keybase/client/pull/14279, a GUI integration would be next, don't have any specifics on dates unfortunately.

if your team is experiencing issues a keybase log send on the CLI or feedback submit in the app will help us squash bugs!

[digininja]

Can you give a couple of lines on how to use it from the command line?

On Thu, 18 Oct 2018 at 18:17, Joshua Blum notifications@github.com wrote:

@pborawski78 https://github.com/pborawski78 fwiw full inbox chat search in the CLI just got into master with #14279 https://github.com/keybase/client/pull/14279, a GUI integration would be next, don't have any specifics on dates unfortunately.

if your team is experiencing issues a keybase log send on the CLI or feedback submit in the app will help us squash bugs!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/keybase/client/issues/8523#issuecomment-431091307, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHJWbTHsW3tAbTrjKvqxln_uHkukpOyks5umLfEgaJpZM4Pb1lx .

[joshblum]

@digininja try keybase chat search --help, the current client releases have the old style conversation search but a build from master will have the new full inbox search. we will be releasing builds across desktop platforms next week

[digininja]

Great, thanks.

How come this has taken so long to get in? Is there something about the system that makes searching hard or was there just other stuff before it in the queue?

On Thu, 18 Oct 2018, 18:41 Joshua Blum, notifications@github.com wrote:

@digininja https://github.com/digininja try keybase chat search --help, the current client releases have the old style conversation search but a build from master will have the new full inbox search. we will be releasing builds across desktop platforms next week

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/keybase/client/issues/8523#issuecomment-431098981, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHJWZLBRv_HlPwE9w8N6iOhy3MW4HDWks5umL1ZgaJpZM4Pb1lx .

[maxtaco]

For starters, your chat is end-to-end encrypted, so the server can't just index it.

[digininja]

So the search is happening client side on the cached messages. That will be slower than having a nice big index cache on the server but still doesn't feel like it should be too hard.

This isn't meant to be putting down your work, I am just trying to understand what's going on behind the scenes.

On Thu, 18 Oct 2018, 19:53 Maxwell Krohn, notifications@github.com wrote:

For starters, your chat is end-to-end encrypted, so the server can't just index it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/keybase/client/issues/8523#issuecomment-431121034, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHJWTARljdNqY6pgBYngCTAqCG022Urks5umM3sgaJpZM4Pb1lx .

[maxtaco]

It had to perform, and we have to make front-end designs and integrations on mobile and desktop. We also spent some time investigating various server-side schemes to search over encrypted data, but in the end we rejected those since they had unacceptable security weaknesses or bad performance. Also, we're doing a million other things, so it's a question of prioritization. As you suggest, @joshblum actually banged out the code in a few weeks once there was time in the schedule and we tested the naive client-side approach to perform well enough. But that's minus a UI integration, which is still in the works.

[digininja]

Fair enough, makes sense.

On Thu, 18 Oct 2018, 20:12 Maxwell Krohn, notifications@github.com wrote:

It had to perform, and we have to make front-end designs and integrations on mobile and desktop. We also spent some time investigating various server-side schemes to search over encrypted data, but in the end we rejected those since they had unacceptable security weaknesses or bad performance. Also, we're doing a million other things, so it's a question of prioritization. As you suggest, @joshblum https://github.com/joshblum actually banged out the code in a few weeks once there was time in the schedule and we tested the naive client-side approach to perform well enough. But that's minus a UI integration, which is still in the works.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/keybase/client/issues/8523#issuecomment-431126097, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHJWU9OCIs-KuVAa6_COITKTj9YJsh-ks5umNKbgaJpZM4Pb1lx .

[ephb]

The client not supporting search is the number one thing that prevents me from recommending Keybase to other people. We currently use it in our company but manually looking for that one message you know you've sent a few days ago might really be a dealbreaker in the long run. I don't care if it would we slow or anything but it not existing is really weird for a communication tool.

[mmaxim]

You can check out both keybase chat search and keybase chat search-regexp from the command line. I know its not really what you want, but hopefully it can help while we get this done in the UI.

[SkypLabs]

FYI, the Tutanota team (@tutao) has developed and open-sourced a full-text search feature for encrypted data for their email client since the data they have to process are end-to-end encrypted as well.

[derekmahar]

@gavin-s-smith parametrizing the search context is a good point and will help with CLI usability.

@skrish13 right now the search is restricted to the conversation you specify, I'll update the issue if this changes.

How does one specify a conversation?

[derekmahar]

Where may I find examples of keybase chat search and keybase chat search-regexp?

[Nock-zz]

Example

By trial and error: keybase chat search-regexp -r -A 1 --channel '#mychannel' myteam '(?:\n).*WAL'

Here there is a channel of '#mychannel' and a team of myteam (but 'team' is referred to as a 'conversation' in the help) and then finally a regexp. You can use non-capturing groups but not lookahead/lookbehind.

To use keybase chat search-regexp you MUST specify a -r option The -A 1 option is to show the next record after the match. And for two records after the match -A 2 The -A 0 option seems to show all records after the match and not zero records The -B 1 is the same as -A 1 except this shows records before the match The -B 0 option seems to show all records after the match and not zero records

Some official examples would be helpful.

keybase chat search 'WAL' where 'WAL' is the query Here WAL is a part of the text. Note that the you can also use 'wal' as the search appears to be case insensitive However non-alphabetic characters do not always work. keybase chat search '\n' fails to show any matches If you extend this to show some letters for example keybase chat search '\nThe' Then you can get matches

[mathieujobin]

Where is the index stored? Client or server?

[maxtaco]

Client. Each device builds an independent index.

[FlxVctr]

As full text search is now available in the GUI, I think we can close this issue. Thanks for implementing it! Now the full chat history is another reason to switch from other clients 👍

[maxtaco]

Indeed. Kudos to @mmaxim and @joshblum for making this feature a reality.

[immartian]

seems --regex doesn't work in CLI:

~$ keybase chat search --regex "https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&//=]*)"^C
~$Error parsing command line arguments: flag provided but not defined: -regex

[immartian]

Actually, I also want to search if a user has posted anything ever in a channel(but I have no prior knowledge what he/she has posted), but no way to get the result.

[joshblum]

@immartian try keybase chat search-regexp instead. If you have --sent-by <username> --regex ".*" it should accomplish what you want. --help will show all of the options.

[immartian]

Thanks, @joshblum , didn't notice there's a search-regexp.

Nevertheless, the command can't return any result, e.g(using keybasefriends as test bed).

$ keybase chat search-regexp keybasefriends "^sorry" 
Search complete. No results found.

adding --sent-by has the same result.

[joshblum]

you need the --regex flag: $ keybase chat search-regexp --regex keybasefriends "^sorry"

[immartian]

It works! Much appreciated. The command itself confused me that --regex was as default.