is it possible to 100000 intents?

[ganbaaelmer]

What problem are you trying to solve?

is it possible to 100000 intents? Rasa can handle 10000 response?

What's your suggested solution?

No response

Examples (if relevant)

No response

Is anything blocking this from being implemented? (if relevant)

No response

Definition of Done

No response

[sara-tagger]

Exalate commented:

sara-tagger commented:

Thanks for submitting this feature request

🚀 @joejuzl will get back to you about it soon! ✨

[koaning]

Exalate commented:

koaning commented:

The word "possible" is always possible, but I'm more concerned if it's pragmatic.

Just so I understand, what is the use-case? Do you have 100_000 intents with 20 examples each? What domain?

[ganbaaelmer]

Exalate commented:

ganbaaelmer commented:

@koaning Law domain. Theres is so many cases in law domain. so need to divide specifically. i mean its around 10,000 answers and infinite questions (100,000 - 10,000,000). So its possible? if its possible how about computation time. i mean calculation of cosine similarity (response time)?

[koaning]

Exalate commented:

koaning commented:

Yeah, part of me is wondering if what you're trying to do here is perhaps more of a "search" problem than a "intent detection" problem. Is this like a FAQ kind of situation?

[ganbaaelmer]

Exalate commented:

ganbaaelmer commented:

@koaning yes almost same as FAQ

[koaning]

Exalate commented:

koaning commented:

Part of me is thinking this is perhaps best implemented with a custom action and using a tool like ElasticSearch in the backend.

@dakshvar22 might know more about the limits of the response selector though.

[ganbaaelmer]

Exalate commented:

ganbaaelmer commented:

@koaning Elastic search is for keyword search right? i need context search not keyword search. Please help me? What should i do?

[koaning]

Exalate commented:

koaning commented:

There's also tools like jina that can add "more context" but it'll still be a search engine. Note that ElasticSearch also supports text similarity search with vector fields.

Before investing in a lot of tech, I would try to test your use case with end-users as early as possible though. It might be that out of the 100_000 questions only 100 of them get asked 99% of the time.

[ganbaaelmer]

Exalate commented:

ganbaaelmer commented:

@koaning if i use elastic search in custom action. so how to train conversation after using elastic search? ( in my rasa x) i think your DIET model is training my model after tagging intents correctly by rasa x? Problem is elastic search is fixed vectors not updated by rasa x. i mean no more training is possible after using elastic search vector. So elasticsearch and rasa have different indexes right also search mechanism is different right? Please explain about it. I need to train after train. This process is endlessly because of wrong predictions needed to fix correctly. (rasa interactive learning) please help me. I need million scale vector similarity search by Rasa.

[ganbaaelmer]

Exalate commented:

ganbaaelmer commented:

@koaning @dakshvar22 please help me

[dakshvar22]

Exalate commented:

dakshvar22 commented:

Can you show some examples of a few intents? I would like to see the granularity at which you are defining the intents.

[sky-2002]

Exalate commented:

ganbaaelmer commented:

@koaning Elastic search is for keyword search right? i need context search not keyword search. Please help me? What should i do?

May be you could try out weaviate vector search engine and database. It vectorizes text based on BERT models so as to capture context.

[m-vdb]

Closing as this is more a question / discussion for the forum. If you're still interested in finding the answer, feel free to repost it here: https://forum.rasa.com/