search

RasaHQ / financial-demo

A demo for a financial services bot
Apache License 2.0
311 stars 398 forks source link

Update responses for twilio voice channel #116

Closed hsm207 closed 2 years ago

hsm207 commented 3 years ago

Trying out the twilio voice connector.

Summary of conversation related changes:

Channel specific responses

github-actions[bot] commented 3 years ago

Commit: 75b19f2624f7974272a8f3b832f03e568c6f5fe0 Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8339 (0.00) 0.9083 (0.00) no data 0.9989 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 554 0.8266 N/A
weighted avg 554 0.8317 N/A
inform 111 0.8075 transfer_money(6), pay_cc(4)
check_balance 54 0.9043 search_transactions(2)
transfer_money 51 0.9057 pay_cc(2), check_recipients(1)
pay_cc 39 0.8500 inform(4), check_balance(1)
affirm 34 0.7324 thankyou(3), inform(3)
search_transactions 32 0.9538 check_balance(1)
goodbye 29 0.7451 inform(5), affirm(3)
thankyou 27 0.7925 affirm(3), help(1)
human_handoff 24 0.8750 transfer_money(1), deny(1)
out_of_scope 23 0.6500 check_earnings(3), inform(2)
check_earnings 22 0.6667 check_balance(4), inform(2)
check_recipients 22 0.9778 N/A
greet 22 0.8636 thankyou(1), help(1)
ask_transfer_charge 22 0.8837 check_balance(2), check_earnings(1)
deny 22 0.8085 thankyou(1), human_handoff(1)
help 20 0.8095 out_of_scope(2), deny(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 344 0.9083 0.9408 0.8779
macro avg 344 0.8741 0.9106 0.8480
weighted avg 344 0.8959 0.9222 0.8779
account_type 112 0.9867 0.9823 0.9911
credit_card 78 0.9605 0.9865 0.9359
amount-of-money 73 0.9865 0.9733 1.0000
PERSON 56 0.4783 0.6111 0.3929
vendor_name 25 0.9583 1.0000 0.9200
github-actions[bot] commented 3 years ago

Commit: 44543e63330e73b88df880435cf98f770f9855e8 Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8517 (0.00) 0.8979 (0.00) no data 0.9989 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 553 0.8511 N/A
weighted avg 553 0.8498 N/A
inform 111 0.8057 pay_cc(8), transfer_money(6)
check_balance 53 0.9189 search_transactions(1), inform(1)
transfer_money 51 0.9444 N/A
pay_cc 39 0.8537 inform(4)
affirm 34 0.7500 thankyou(3), inform(2)
search_transactions 32 0.9231 check_earnings(1), check_balance(1)
goodbye 29 0.7059 inform(3), thankyou(2)
thankyou 27 0.8136 help(1), goodbye(1)
human_handoff 24 0.9565 inform(1), help(1)
out_of_scope 23 0.7317 inform(3), help(2)
deny 22 0.8095 affirm(4), thankyou(1)
greet 22 0.8837 thankyou(1), help(1)
check_recipients 22 0.9778 N/A
ask_transfer_charge 22 0.9524 check_balance(2)
check_earnings 22 0.7907 check_balance(2), out_of_scope(1)
help 20 0.8000 out_of_scope(1), affirm(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 344 0.8979 0.9286 0.8692
macro avg 344 0.8616 0.8949 0.8390
weighted avg 344 0.8839 0.9071 0.8692
account_type 112 0.9821 0.9821 0.9821
credit_card 78 0.9481 0.9605 0.9359
amount-of-money 73 0.9799 0.9605 1.0000
PERSON 56 0.4396 0.5714 0.3571
vendor_name 25 0.9583 1.0000 0.9200
github-actions[bot] commented 3 years ago

Commit: f99ef6fc9bd932cd3980e60d3482458ce9540b7b Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8237 (0.00) 0.8996 (0.00) no data 0.9720 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 573 0.8154 N/A
weighted avg 573 0.8199 N/A
inform 111 0.8038 transfer_money(8), pay_cc(4)
check_balance 53 0.8850 affirm(1), search_transactions(1)
transfer_money 51 0.8762 pay_cc(3), ask_transfer_charge(2)
pay_cc 39 0.8706 inform(2)
affirm 34 0.7143 inform(4), thankyou(3)
search_transactions 32 0.8923 check_balance(2), check_earnings(1)
goodbye 29 0.7170 inform(4), affirm(2)
thankyou 27 0.8421 check_human(2), affirm(1)
human_handoff 24 0.8800 pay_cc(1), inform(1)
greet 23 0.8182 affirm(2), help(1)
out_of_scope 23 0.6111 human_handoff(3), ask_transfer_charge(2)
check_earnings 22 0.7143 check_balance(4), affirm(1)
check_recipients 22 0.9167 N/A
ask_transfer_charge 22 0.7727 check_balance(2), help(2)
deny 22 0.8571 inform(1), check_recipients(1)
help 20 0.7907 search_transactions(1), ask_transfer_charge(1)
check_human 19 0.9000 human_handoff(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 344 0.8996 0.9288 0.8721
macro avg 344 0.8635 0.8937 0.8408
weighted avg 344 0.8873 0.9084 0.8721
account_type 112 0.9867 0.9823 0.9911
credit_card 78 0.9605 0.9865 0.9359
amount-of-money 73 0.9865 0.9733 1.0000
PERSON 56 0.4255 0.5263 0.3571
vendor_name 25 0.9583 1.0000 0.9200
github-actions[bot] commented 3 years ago

Commit: 39cc3de319229c47dfd94fb1c49fc221c0a487cc Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8202 (0.00) 0.8936 (0.00) no data 0.9720 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 573 0.8075 N/A
weighted avg 573 0.8152 N/A
inform 111 0.8039 transfer_money(9), pay_cc(7)
check_balance 53 0.9189 transfer_money(1), search_transactions(1)
transfer_money 51 0.8649 pay_cc(2), ask_transfer_charge(1)
pay_cc 39 0.8372 inform(2), transfer_money(1)
affirm 34 0.6957 thankyou(3), deny(2)
search_transactions 32 0.9687 check_balance(1)
goodbye 29 0.6792 affirm(4), inform(4)
thankyou 27 0.7869 greet(1), check_human(1)
human_handoff 24 0.9167 deny(1), thankyou(1)
greet 23 0.7619 goodbye(2), affirm(2)
out_of_scope 23 0.5143 check_earnings(3), inform(2)
deny 22 0.7556 greet(1), thankyou(1)
check_recipients 22 0.9778 N/A
check_earnings 22 0.7143 check_balance(4), pay_cc(1)
ask_transfer_charge 22 0.9130 check_balance(1)
help 20 0.8000 affirm(1), ask_transfer_charge(1)
check_human 19 0.8182 human_handoff(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 344 0.8936 0.9226 0.8663
macro avg 344 0.8595 0.8901 0.8363
weighted avg 344 0.8815 0.9027 0.8663
account_type 112 0.9735 0.9649 0.9821
credit_card 78 0.9605 0.9865 0.9359
amount-of-money 73 0.9796 0.9730 0.9863
PERSON 56 0.4255 0.5263 0.3571
vendor_name 25 0.9583 1.0000 0.9200
github-actions[bot] commented 3 years ago

Commit: ef5298157580f3eef7ef63e9055739d9e305d768 Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8377 (0.00) 0.8976 (0.00) no data 0.9989 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 573 0.8344 N/A
weighted avg 573 0.8357 N/A
inform 111 0.8134 pay_cc(10), transfer_money(9)
check_balance 53 0.9298 N/A
transfer_money 51 0.8519 pay_cc(4), check_recipients(1)
pay_cc 39 0.8043 inform(2)
affirm 34 0.7692 inform(5), thankyou(2)
search_transactions 32 0.9841 check_balance(1)
goodbye 29 0.7273 affirm(3), inform(3)
thankyou 27 0.7857 check_human(2), inform(1)
human_handoff 24 0.8627 pay_cc(1), goodbye(1)
out_of_scope 23 0.7500 transfer_money(2), help(2)
greet 23 0.7805 thankyou(2), affirm(2)
deny 22 0.8636 pay_cc(1), thankyou(1)
ask_transfer_charge 22 0.8636 check_balance(2), human_handoff(1)
check_recipients 22 0.9778 N/A
check_earnings 22 0.7500 check_balance(3), affirm(1)
help 20 0.8293 ask_transfer_charge(2), out_of_scope(1)
check_human 19 0.8421 human_handoff(2), greet(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 344 0.8976 0.9312 0.8663
macro avg 344 0.8554 0.8965 0.8248
weighted avg 344 0.8847 0.9107 0.8663
account_type 112 0.9867 0.9823 0.9911
credit_card 78 0.9605 0.9865 0.9359
amount-of-money 73 0.9865 0.9733 1.0000
PERSON 56 0.4301 0.5405 0.3571
vendor_name 25 0.9130 1.0000 0.8400
github-actions[bot] commented 3 years ago

Commit: b1921dce09b574675632b1b5117c5cb13b8dea87 Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8377 (0.00) 0.8962 (0.00) no data 0.9989 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 573 0.8347 N/A
weighted avg 573 0.8356 N/A
inform 111 0.8224 transfer_money(7), check_earnings(4)
check_balance 53 0.8829 inform(2), search_transactions(1)
transfer_money 51 0.8544 pay_cc(5), inform(1)
pay_cc 39 0.8636 inform(1)
affirm 34 0.7536 inform(4), thankyou(3)
search_transactions 32 0.9375 check_balance(1), check_earnings(1)
goodbye 29 0.7500 inform(4), affirm(2)
thankyou 27 0.7797 check_human(2), out_of_scope(1)
human_handoff 24 0.9362 pay_cc(1), inform(1)
out_of_scope 23 0.6500 affirm(2), check_earnings(2)
greet 23 0.8182 thankyou(2), affirm(2)
check_earnings 22 0.6818 check_balance(4), affirm(1)
check_recipients 22 0.9565 N/A
deny 22 0.9091 affirm(1), thankyou(1)
ask_transfer_charge 22 0.8837 check_balance(2), check_human(1)
help 20 0.8095 search_transactions(1), ask_transfer_charge(1)
check_human 19 0.9000 goodbye(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 344 0.8962 0.9283 0.8663
macro avg 344 0.8556 0.8898 0.8337
weighted avg 344 0.8797 0.9028 0.8663
account_type 112 0.9867 0.9823 0.9911
credit_card 78 0.9419 0.9481 0.9359
amount-of-money 73 0.9865 0.9733 1.0000
PERSON 56 0.4045 0.5455 0.3214
vendor_name 25 0.9583 1.0000 0.9200
github-actions[bot] commented 3 years ago

Commit: 4049502303b9ccc785bf6d692fd1f707f1f6d6e5 Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8255 (0.00) 0.8996 (0.00) no data 0.9859 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 573 0.8229 N/A
weighted avg 573 0.8222 N/A
inform 111 0.7817 transfer_money(9), pay_cc(8)
check_balance 53 0.9189 search_transactions(1), out_of_scope(1)
transfer_money 51 0.8598 pay_cc(4), inform(1)
pay_cc 39 0.8315 inform(2)
affirm 34 0.7536 thankyou(3), inform(2)
search_transactions 32 0.9394 check_balance(1)
goodbye 29 0.7273 greet(3), inform(2)
thankyou 27 0.7586 check_human(2), greet(1)
human_handoff 24 0.8800 inform(1), thankyou(1)
out_of_scope 23 0.6842 human_handoff(2), help(2)
greet 23 0.7111 goodbye(3), help(2)
check_earnings 22 0.7143 check_balance(4), affirm(1)
check_recipients 22 0.9778 N/A
ask_transfer_charge 22 0.9091 pay_cc(1), check_balance(1)
deny 22 0.8696 affirm(1), thankyou(1)
help 20 0.8000 greet(1), ask_transfer_charge(1)
check_human 19 0.8718 affirm(1), goodbye(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 344 0.8996 0.9288 0.8721
macro avg 344 0.8632 0.8939 0.8408
weighted avg 344 0.8866 0.9077 0.8721
account_type 112 0.9867 0.9823 0.9911
credit_card 78 0.9542 0.9733 0.9359
amount-of-money 73 0.9865 0.9733 1.0000
PERSON 56 0.4301 0.5405 0.3571
vendor_name 25 0.9583 1.0000 0.9200
github-actions[bot] commented 3 years ago

Commit: b81e6e65d6dd0ea09f69c94e096ac1e6ac401c89 Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8394 (0.00) 0.8953 (0.00) no data 0.9989 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 573 0.8354 N/A
weighted avg 573 0.8353 N/A
inform 111 0.7960 transfer_money(6), pay_cc(6)
check_balance 53 0.8947 search_transactions(1), check_earnings(1)
transfer_money 51 0.8868 ask_transfer_charge(2), pay_cc(2)
pay_cc 39 0.8837 inform(1)
affirm 34 0.7714 inform(4), thankyou(2)
search_transactions 32 0.9254 check_balance(1)
goodbye 29 0.7719 affirm(3), inform(2)
thankyou 27 0.8148 check_human(2), affirm(1)
human_handoff 24 0.8800 pay_cc(1), goodbye(1)
greet 23 0.8372 goodbye(2), help(1)
out_of_scope 23 0.6486 check_earnings(3), transfer_money(2)
check_earnings 22 0.6190 check_balance(3), search_transactions(2)
ask_transfer_charge 22 0.8696 check_balance(2)
deny 22 0.8837 affirm(1), goodbye(1)
check_recipients 22 0.9778 N/A
help 20 0.8837 out_of_scope(1)
check_human 19 0.8571 human_handoff(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 344 0.8953 0.9365 0.8576
macro avg 344 0.8551 0.8973 0.8255
weighted avg 344 0.8794 0.9113 0.8576
account_type 112 0.9867 0.9823 0.9911
credit_card 78 0.9605 0.9865 0.9359
amount-of-money 73 0.9655 0.9722 0.9589
PERSON 56 0.4045 0.5455 0.3214
vendor_name 25 0.9583 1.0000 0.9200
github-actions[bot] commented 3 years ago

Commit: c180a8ea5efb607e9fc00e121be9ea7fedc1391a Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8080 (0.00) 0.9023 (0.00) no data 0.9989 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 573 0.8094 N/A
weighted avg 573 0.8039 N/A
inform 111 0.7337 transfer_money(9), pay_cc(8)
check_balance 53 0.9107 search_transactions(1), ask_transfer_charge(1)
transfer_money 51 0.8598 pay_cc(4), check_recipients(1)
pay_cc 39 0.8090 inform(2), check_balance(1)
affirm 34 0.7385 inform(4), thankyou(2)
search_transactions 32 0.9538 check_balance(1)
goodbye 29 0.6667 inform(4), affirm(2)
thankyou 27 0.7857 check_human(2), inform(1)
human_handoff 24 0.8696 transfer_money(1), check_human(1)
out_of_scope 23 0.6222 ask_transfer_charge(2), inform(2)
greet 23 0.7391 goodbye(2), affirm(2)
deny 22 0.8571 pay_cc(1), inform(1)
ask_transfer_charge 22 0.8400 check_balance(1)
check_recipients 22 0.9167 N/A
check_earnings 22 0.7273 check_balance(4), inform(1)
help 20 0.8293 affirm(1), ask_transfer_charge(1)
check_human 19 0.9000 human_handoff(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 344 0.9023 0.9346 0.8721
macro avg 344 0.8657 0.8981 0.8408
weighted avg 344 0.8902 0.9143 0.8721
account_type 112 0.9911 0.9911 0.9911
credit_card 78 0.9669 1.0000 0.9359
amount-of-money 73 0.9865 0.9733 1.0000
PERSON 56 0.4255 0.5263 0.3571
vendor_name 25 0.9583 1.0000 0.9200
github-actions[bot] commented 3 years ago

Commit: 0b38ba5e778f78e7666c8c7f2c85752e25c276a7 Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8342 (0.00) 0.8992 (0.00) no data 0.9989 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 573 0.8315 N/A
weighted avg 573 0.8315 N/A
inform 111 0.8134 transfer_money(9), pay_cc(6)
check_balance 53 0.9107 search_transactions(1), ask_transfer_charge(1)
transfer_money 51 0.8519 pay_cc(3), inform(1)
pay_cc 39 0.8372 inform(2), transfer_money(1)
affirm 34 0.7536 inform(3), thankyou(3)
search_transactions 32 0.8857 check_balance(1)
goodbye 29 0.7143 inform(4), affirm(3)
thankyou 27 0.7547 greet(2), check_human(2)
human_handoff 24 0.8980 pay_cc(1), inform(1)
greet 23 0.8636 goodbye(2), check_human(1)
out_of_scope 23 0.7027 search_transactions(2), help(2)
check_recipients 22 0.9778 N/A
check_earnings 22 0.7500 check_balance(4), pay_cc(1)
ask_transfer_charge 22 0.9091 check_balance(2)
deny 22 0.8636 affirm(1), goodbye(1)
help 20 0.8293 affirm(1), search_transactions(1)
check_human 19 0.8205 human_handoff(2), goodbye(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 344 0.8992 0.9315 0.8692
macro avg 344 0.8635 0.8954 0.8390
weighted avg 344 0.8873 0.9112 0.8692
account_type 112 0.9865 0.9910 0.9821
credit_card 78 0.9605 0.9865 0.9359
amount-of-money 73 0.9865 0.9733 1.0000
PERSON 56 0.4255 0.5263 0.3571
vendor_name 25 0.9583 1.0000 0.9200
ArjaanBuijk commented 3 years ago

@hsm207 , The setup now worked flawlessly, and I was able to talk to it from my cell phone. That was really neat!

The conversation did not go that smooth, but I think it is ready to be merged, so we then can deploy it and collect the conversations in Rasa Enterprise, and apply some CDD to fix the conversations to work for voice.

Do we have a rasa twilio phone number? Right now I am using my own trial account.

mvielkind commented 3 years ago

The conversation did not go that smooth

@ArjaanBuijk - can you share what about the conversation wasn't smooth? If there are pieces that don't go smoothly we can suggest best practices for how to configure the channel.

github-actions[bot] commented 3 years ago

Commit: 1b2c860ab20a80dfa4b1a2f7b4b9bf2d92b2dc39 Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8115 (0.00) 0.8939 (0.00) no data 0.9989 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 573 0.8047 N/A
weighted avg 573 0.8098 N/A
inform 111 0.8039 transfer_money(7), pay_cc(5)
check_balance 53 0.8909 search_transactions(2), inform(2)
transfer_money 51 0.8598 pay_cc(3), check_recipients(1)
pay_cc 39 0.8675 inform(2), check_balance(1)
affirm 34 0.6761 thankyou(3), goodbye(3)
search_transactions 32 0.9231 check_balance(1), check_earnings(1)
goodbye 29 0.5965 affirm(4), inform(2)
thankyou 27 0.7667 transfer_money(1), check_human(1)
human_handoff 24 0.8627 inform(1), thankyou(1)
out_of_scope 23 0.7027 ask_transfer_charge(3), check_earnings(2)
greet 23 0.7234 goodbye(3), affirm(2)
ask_transfer_charge 22 0.8182 help(2), check_balance(1)
check_recipients 22 0.9545 transfer_money(1)
deny 22 0.7826 thankyou(2), affirm(1)
check_earnings 22 0.7317 check_balance(3), greet(1)
help 20 0.8780 ask_transfer_charge(1), out_of_scope(1)
check_human 19 0.8421 human_handoff(2), goodbye(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 344 0.8939 0.9335 0.8576
macro avg 344 0.8553 0.8948 0.8283
weighted avg 344 0.8780 0.9084 0.8576
account_type 112 0.9730 0.9818 0.9643
credit_card 78 0.9542 0.9733 0.9359
amount-of-money 73 0.9865 0.9733 1.0000
PERSON 56 0.4045 0.5455 0.3214
vendor_name 25 0.9583 1.0000 0.9200
github-actions[bot] commented 3 years ago

Commit: b949a79558827b7b1ede2315ddd3d06b4e26c42d Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8202 (0.00) 0.8979 (0.00) no data 0.9731 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 573 0.8161 N/A
weighted avg 573 0.8172 N/A
inform 111 0.7900 transfer_money(8), pay_cc(7)
check_balance 53 0.8870 search_transactions(1), check_earnings(1)
transfer_money 51 0.8364 pay_cc(3), inform(1)
pay_cc 39 0.8409 transfer_money(1), inform(1)
affirm 34 0.7536 thankyou(3), inform(2)
search_transactions 32 0.9524 ask_transfer_charge(1), check_balance(1)
goodbye 29 0.7778 affirm(3), thankyou(1)
thankyou 27 0.8070 transfer_money(1), out_of_scope(1)
human_handoff 24 0.8400 inform(2), transfer_money(1)
greet 23 0.7727 thankyou(2), affirm(2)
out_of_scope 23 0.6154 ask_transfer_charge(2), check_earnings(2)
check_earnings 22 0.6512 check_balance(5), inform(2)
ask_transfer_charge 22 0.8000 check_balance(3), check_human(1)
check_recipients 22 0.9130 transfer_money(1)
deny 22 0.8696 affirm(1), thankyou(1)
help 20 0.8718 ask_transfer_charge(1), out_of_scope(1)
check_human 19 0.8947 human_handoff(2)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 344 0.8979 0.9286 0.8692
macro avg 344 0.8597 0.8903 0.8373
weighted avg 344 0.8845 0.9062 0.8692
account_type 112 0.9911 0.9911 0.9911
credit_card 78 0.9542 0.9733 0.9359
amount-of-money 73 0.9865 0.9733 1.0000
PERSON 56 0.4086 0.5135 0.3393
vendor_name 25 0.9583 1.0000 0.9200
github-actions[bot] commented 3 years ago

Commit: 1d34c433453b3f9ec96bc9e8836006417801da0f Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8499 (0.00) 0.9017 (0.00) no data 0.9989 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 573 0.8458 N/A
weighted avg 573 0.8475 N/A
inform 111 0.8134 pay_cc(8), transfer_money(6)
check_balance 53 0.9286 search_transactions(1)
transfer_money 51 0.9038 inform(2), check_recipients(2)
pay_cc 39 0.8605 inform(2)
affirm 34 0.7761 inform(4), thankyou(3)
search_transactions 32 0.9231 check_balance(1), check_earnings(1)
goodbye 29 0.8000 affirm(3), inform(2)
thankyou 27 0.8070 greet(1), out_of_scope(1)
human_handoff 24 0.9167 pay_cc(1), inform(1)
greet 23 0.8750 help(1), thankyou(1)
out_of_scope 23 0.7179 ask_transfer_charge(2), check_earnings(2)
deny 22 0.8636 check_earnings(1), inform(1)
check_recipients 22 0.8980 N/A
ask_transfer_charge 22 0.8889 search_transactions(1), check_balance(1)
check_earnings 22 0.6818 check_balance(4), pay_cc(1)
help 20 0.8293 check_recipients(2), greet(1)
check_human 19 0.8947 human_handoff(1), thankyou(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 344 0.9017 0.9401 0.8663
macro avg 344 0.8659 0.9070 0.8373
weighted avg 344 0.8874 0.9186 0.8663
account_type 112 0.9732 0.9732 0.9732
credit_card 78 0.9669 1.0000 0.9359
amount-of-money 73 0.9865 0.9733 1.0000
PERSON 56 0.4444 0.5882 0.3571
vendor_name 25 0.9583 1.0000 0.9200
github-actions[bot] commented 3 years ago

Commit: 176a04a88e549d463a68096921e9d002d3473005 Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8325 (0.00) 0.9009 (0.00) no data 0.9989 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 573 0.8245 N/A
weighted avg 573 0.8299 N/A
inform 111 0.8098 transfer_money(7), check_earnings(5)
check_balance 53 0.9189 search_transactions(2)
transfer_money 51 0.8972 pay_cc(2), check_recipients(1)
pay_cc 39 0.8675 inform(3)
affirm 34 0.7941 inform(3), thankyou(2)
search_transactions 32 0.9091 check_balance(1), check_earnings(1)
goodbye 29 0.6538 affirm(3), inform(3)
thankyou 27 0.7586 check_human(2), affirm(1)
human_handoff 24 0.8750 transfer_money(1), pay_cc(1)
greet 23 0.7826 goodbye(3), thankyou(1)
out_of_scope 23 0.6500 check_earnings(2), help(2)
deny 22 0.9302 check_earnings(1), thankyou(1)
check_recipients 22 0.8936 check_balance(1)
check_earnings 22 0.6383 check_balance(4), out_of_scope(1)
ask_transfer_charge 22 0.9091 pay_cc(1), check_balance(1)
help 20 0.8571 ask_transfer_charge(1), thankyou(1)
check_human 19 0.8718 goodbye(1), human_handoff(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 344 0.9009 0.9317 0.8721
macro avg 344 0.8641 0.8969 0.8408
weighted avg 344 0.8874 0.9102 0.8721
account_type 112 0.9867 0.9823 0.9911
credit_card 78 0.9542 0.9733 0.9359
amount-of-money 73 0.9865 0.9733 1.0000
PERSON 56 0.4348 0.5556 0.3571
vendor_name 25 0.9583 1.0000 0.9200
github-actions[bot] commented 3 years ago

Commit: 9585eff747b54d9b00597033a1035adf895ab6c8 Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8133 (0.00) 0.8985 (0.00) no data 0.9859 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 573 0.8055 N/A
weighted avg 573 0.8097 N/A
inform 111 0.7864 pay_cc(8), transfer_money(7)
check_balance 53 0.8718 search_transactions(1), ask_transfer_charge(1)
transfer_money 51 0.8889 pay_cc(2), inform(1)
pay_cc 39 0.8043 inform(2)
affirm 34 0.7273 inform(3), thankyou(3)
search_transactions 32 0.9524 transfer_money(1), pay_cc(1)
goodbye 29 0.7018 inform(4), affirm(2)
thankyou 27 0.8276 check_human(1), help(1)
human_handoff 24 0.8936 ask_transfer_charge(1), pay_cc(1)
greet 23 0.7907 goodbye(2), affirm(2)
out_of_scope 23 0.5789 pay_cc(3), help(2)
ask_transfer_charge 22 0.8182 check_balance(2), check_human(1)
check_earnings 22 0.6341 check_balance(6), affirm(1)
check_recipients 22 0.9545 transfer_money(1)
deny 22 0.8000 greet(2), inform(1)
help 20 0.7907 out_of_scope(2), ask_transfer_charge(1)
check_human 19 0.8718 goodbye(1), human_handoff(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 344 0.8985 0.9233 0.8750
macro avg 344 0.8643 0.8909 0.8444
weighted avg 344 0.8872 0.9045 0.8750
account_type 112 0.9867 0.9823 0.9911
credit_card 78 0.9481 0.9605 0.9359
amount-of-money 73 0.9865 0.9733 1.0000
PERSON 56 0.4421 0.5385 0.3750
vendor_name 25 0.9583 1.0000 0.9200
ArjaanBuijk commented 3 years ago

@mvielkind

The conversation did not go that smooth

can you share what about the conversation wasn't smooth? If there are pieces that don't go smoothly we can suggest best practices for how to configure the channel.

hsm207 commented 3 years ago

@ArjaanBuijk if the user reconnected before the session expired, shouldn't jumping back to the previous conversation be the intended behavior?

ArjaanBuijk commented 3 years ago

@ArjaanBuijk if the user reconnected before the session expired, shouldn't jumping back to the previous conversation be the intended behavior?

I am not sure about it. I found it unexpected, because I consider hanging up, or being disconnected, kind of like a restart.

github-actions[bot] commented 3 years ago

Commit: 6338da72c66f890a7cacf28ee1b8c7a1f7270a19 Data: default

Configuration Intent Classification Micro F1 Entity Recognition Micro F1 Response Selection Micro F1 Story Recognition Micro F1
config.yml 0.8226 (0.00) 0.8959 (0.00) no data 0.9859 (0.00)

Intent Cross-Validation Results

class support f1-score confused_with
macro avg 575 0.8167 N/A
weighted avg 575 0.8197 N/A
inform 113 0.7907 transfer_money(6), pay_cc(5)
check_balance 53 0.9091 search_transactions(2), pay_cc(1)
transfer_money 51 0.8762 pay_cc(4), inform(1)
pay_cc 39 0.8471 inform(2), check_balance(1)
affirm 34 0.7937 inform(5), thankyou(2)
search_transactions 32 0.8986 check_earnings(1)
goodbye 29 0.6909 inform(5), deny(2)
thankyou 27 0.7797 goodbye(2), affirm(1)
human_handoff 24 0.8462 inform(1), thankyou(1)
out_of_scope 23 0.6316 human_handoff(3), help(3)
greet 23 0.7556 inform(2), thankyou(2)
check_earnings 22 0.6667 check_balance(4), inform(1)
check_recipients 22 0.9545 transfer_money(1)
deny 22 0.8095 goodbye(2), transfer_money(1)
ask_transfer_charge 22 0.8780 check_balance(2), check_earnings(2)
help 20 0.8837 out_of_scope(1)
check_human 19 0.8718 human_handoff(1), thankyou(1)

Entity Cross-Validation Results

entity support f1-score precision recall
micro avg 346 0.8959 0.9369 0.8584
macro avg 346 0.7152 0.7566 0.6874
weighted avg 346 0.8793 0.9122 0.8584
account_type 112 0.9776 0.9820 0.9732
credit_card 78 0.9419 0.9481 0.9359
amount-of-money 73 0.9865 0.9733 1
PERSON 56 0.4719 0.6364 0.3750
vendor_name 25 0.9130 1 0.8400
search_type 2 N/A N/A N/A