search

kercre123 / wire-pod

Free, fully-featured server software for the Anki Vector robot.
MIT License
465 stars 159 forks source link

How to add new language (polish)? #125

Closed witold-gren closed 1 year ago

witold-gren commented 1 year ago

Hey, this project is really good. Thanks for you amazing work.

I have one question. It is possible to add new language to this project? My native language is polish. I also checked that VOSK support polish language (https://alphacephei.com/vosk/models/vosk-model-small-pl-0.22.zip) I know that this model is not super ready, but is worth to check how it works 😀 also I can prepare and translate intent data.

What I must do to add new language and how to set it on vector? Of course I will prepare this new configuration locally and test it before I will create PR (to be sure that it make sens)

kercre123 commented 1 year ago

Languages can be added, but the code for it is kinda all over the place. If you are willing to translate the phrases in chipper/intent-data/en-US.json to Polish (output to chipper/intent-data/pl-PL.json), I can add the model to the code.

unbuilt commented 1 year ago

Replace the models in en-US folder, and change en-US.json intent data, it works, simple but not a good way.

This will make the Vector recognize languages other than English, but it only speaks English.

I wonder how to support other language for TTS?

One of the approaches is to change the TextToSpeechProvider. Another way is to use SDK API to play a wav stream, the wav stream can be generated in chipper side, and supports any languages.

witold-gren commented 1 year ago

@kercre123 I created PR with polish intent https://github.com/kercre123/wire-pod/pull/126

witold-gren commented 1 year ago

I also probably found a small bug, two times we see intent_amazon_signin:

  1. https://github.com/kercre123/wire-pod/blob/main/chipper/intent-data/en-US.json#L55
  2. https://github.com/kercre123/wire-pod/blob/main/chipper/intent-data/en-US.json#L59
witold-gren commented 1 year ago

BTW. This bug exist in all intent files 😀 But it should be intent_amazon_signout. Am I right? https://randym32.github.io/Anki.Vector.Documentation/protocols/intents.html

witold-gren commented 1 year ago

hm.. I also found intent which are not implemented. Is it correct or can we just add them?

intent_status_feeling
intent_system_noaudio
intent_play_specific_extend
intent_play_anygame
intent_knowledge_no_response
intent_knowledge_response_extend
intent_imperative_scold
intent_imperative_lookoverthere
intent_imperative_eyecolor_specific_extend
intent_global_delete_extend
intent_blackjack_playagain
witold-gren commented 1 year ago

Replace the models in en-US folder, and change en-US.json intent data, it works, simple but not a good way.

This will make the Vector recognize languages other than English, but it only speaks English.

I wonder how to support other language for TTS?

One of the approaches is to change the TextToSpeechProvider. Another way is to use SDK API to play a wav stream, the wav stream can be generated in chipper side, and supports any languages.

I see one options to quickly TTS with "polish pronunciation": https://github.com/ggegoge/PyTDM -> "transform" polish text to specific english text which will sound like polish 😀

The second option is use correct TextToSpeechProvider.. but for now I do not have such abilities to combine something like that. I checked project https://github.com/rhasspy/piper and the polish voice sound quite good.

witold-gren commented 1 year ago

I also have one observation when I replace the models in en-US folder to polish model, and change en-US.json intent data to polish data, all command which need extra params didn't work correctly. This is the example log:

kwi 17 15:56:12 escapepod start.sh[473]: Bot 1 Stream type: OPUS
kwi 17 15:56:12 escapepod start.sh[473]: (Bot 1, Vosk) Processing...
kwi 17 15:56:17 escapepod start.sh[473]: (Bot 1) End of speech detected.
kwi 17 15:56:17 escapepod start.sh[473]: Bot 1 Transcribed text: jestem witold
kwi 17 15:56:17 escapepod start.sh[473]: Not a custom intent
kwi 17 15:56:17 escapepod start.sh[473]: Partial match for intent intent_names_username_extend (jestem)
kwi 17 15:56:17 escapepod start.sh[473]: Checking params for candidate intent intent_names_username_extend
kwi 17 15:56:17 escapepod start.sh[473]: No name parsed from speech
kwi 17 15:56:17 escapepod start.sh[473]: Bot 1 Intent Sent: intent_names_username_extend
kwi 17 15:56:17 escapepod start.sh[473]: Bot 1 Parameters Sent: map[username:]
kwi 17 15:56:17 escapepod start.sh[473]: Bot 1 request served.
kwi 17 15:56:26 escapepod start.sh[473]: 2023/04/17 15:56:26 http: panic serving 192.168.1.49:55178: runtime error: invalid memory address or nil pointer dereference
kwi 17 15:56:26 escapepod start.sh[473]: goroutine 81 [running]:
kwi 17 15:56:26 escapepod start.sh[473]: net/http.(*conn).serve.func1()
kwi 17 15:56:26 escapepod start.sh[473]:         /usr/local/go/src/net/http/server.go:1850 +0xb8
kwi 17 15:56:26 escapepod start.sh[473]: panic({0x11a6a40, 0x1a714a0})
kwi 17 15:56:26 escapepod start.sh[473]:         /usr/local/go/src/runtime/panic.go:890 +0x260
kwi 17 15:56:26 escapepod start.sh[473]: github.com/kercre123/chipper/pkg/wirepod/sdkapp.SdkapiHandler({0x14415d0?, 0x40003f2620}, 0x4000128c00)
kwi 17 15:56:26 escapepod start.sh[473]:         /home/witold/wire-pod/chipper/pkg/wirepod/sdkapp/server.go:110 +0x2808
kwi 17 15:56:26 escapepod start.sh[473]: net/http.HandlerFunc.ServeHTTP(0x4000683ad8?, {0x14415d0?, 0x40003f2620?}, 0x0?)
kwi 17 15:56:26 escapepod start.sh[473]:         /usr/local/go/src/net/http/server.go:2109 +0x38
kwi 17 15:56:26 escapepod start.sh[473]: net/http.(*ServeMux).ServeHTTP(0x400090c81f?, {0x14415d0, 0x40003f2620}, 0x4000128c00)
kwi 17 15:56:26 escapepod start.sh[473]:         /usr/local/go/src/net/http/server.go:2487 +0x140
kwi 17 15:56:26 escapepod start.sh[473]: net/http.serverHandler.ServeHTTP({0x400040f290?}, {0x14415d0, 0x40003f2620}, 0x4000128c00)
kwi 17 15:56:26 escapepod start.sh[473]:         /usr/local/go/src/net/http/server.go:2947 +0x2cc
kwi 17 15:56:26 escapepod start.sh[473]: net/http.(*conn).serve(0x4000424500, {0x1442190, 0x4000386180})
kwi 17 15:56:26 escapepod start.sh[473]:         /usr/local/go/src/net/http/server.go:1991 +0x544
kwi 17 15:56:26 escapepod start.sh[473]: created by net/http.(*Server).Serve
kwi 17 15:56:26 escapepod start.sh[473]:         /usr/local/go/src/net/http/server.go:3102 +0x43c
kwi 17 15:57:03 escapepod start.sh[473]: 2023/04/17 15:57:03 http: panic serving 192.168.1.49:55193: runtime error: invalid memory address or nil pointer dereference
kwi 17 15:57:03 escapepod start.sh[473]: goroutine 444 [running]:
kwi 17 15:57:03 escapepod start.sh[473]: net/http.(*conn).serve.func1()
kwi 17 15:57:03 escapepod start.sh[473]:         /usr/local/go/src/net/http/server.go:1850 +0xb8
kwi 17 15:57:03 escapepod start.sh[473]: panic({0x11a6a40, 0x1a714a0})
kwi 17 15:57:03 escapepod start.sh[473]:         /usr/local/go/src/runtime/panic.go:890 +0x260
kwi 17 15:57:03 escapepod start.sh[473]: github.com/kercre123/chipper/pkg/wirepod/sdkapp.SdkapiHandler({0x14415d0?, 0x40003f22a0}, 0x400003e300)
kwi 17 15:57:03 escapepod start.sh[473]:         /home/witold/wire-pod/chipper/pkg/wirepod/sdkapp/server.go:110 +0x2808
kwi 17 15:57:03 escapepod start.sh[473]: net/http.HandlerFunc.ServeHTTP(0x4000321ad8?, {0x14415d0?, 0x40003f22a0?}, 0x0?)
kwi 17 15:57:03 escapepod start.sh[473]:         /usr/local/go/src/net/http/server.go:2109 +0x38
kwi 17 15:57:03 escapepod start.sh[473]: net/http.(*ServeMux).ServeHTTP(0x40000b4b1f?, {0x14415d0, 0x40003f22a0}, 0x400003e300)
kwi 17 15:57:03 escapepod start.sh[473]:         /usr/local/go/src/net/http/server.go:2487 +0x140
kwi 17 15:57:03 escapepod start.sh[473]: net/http.serverHandler.ServeHTTP({0x40004ccf60?}, {0x14415d0, 0x40003f22a0}, 0x400003e300)
kwi 17 15:57:03 escapepod start.sh[473]:         /usr/local/go/src/net/http/server.go:2947 +0x2cc
kwi 17 15:57:03 escapepod start.sh[473]: net/http.(*conn).serve(0x40003ae780, {0x1442190, 0x4000386180})
kwi 17 15:57:03 escapepod start.sh[473]:         /usr/local/go/src/net/http/server.go:1991 +0x544
kwi 17 15:57:03 escapepod start.sh[473]: created by net/http.(*Server).Serve
kwi 17 15:57:03 escapepod start.sh[473]:         /usr/local/go/src/net/http/server.go:3102 +0x43c
kwi 17 15:57:56 escapepod start.sh[473]: Bot 1 Stream type: OPUS
kwi 17 15:57:56 escapepod start.sh[473]: (Bot 1, Vosk) Processing...

and other one:

kwi 17 16:04:54 escapepod start.sh[473]: Not a custom intent
kwi 17 16:04:54 escapepod start.sh[473]: Partial match for intent intent_message_recordmessage_extend (nagraj)
kwi 17 16:04:54 escapepod start.sh[473]: Checking params for candidate intent intent_message_recordmessage_extend
kwi 17 16:04:54 escapepod start.sh[473]: Bot 1 Intent Sent: intent_message_recordmessage_extend
kwi 17 16:04:54 escapepod start.sh[473]: Bot 1 Parameters Sent: map[given_name:]
kwi 17 16:04:54 escapepod start.sh[473]: Bot 1 request served.
kwi 17 16:13:46 escapepod start.sh[473]: Bot 1 Stream type: OPUS
kwi 17 16:13:46 escapepod start.sh[473]: (Bot 1, Vosk) Processing...
kwi 17 16:13:50 escapepod start.sh[473]: (Bot 1) End of speech detected.

do you know where is a problem?

witold-gren commented 1 year ago

Hey @kercre123 I fixed all problems and now Vector understand polish language correctly 😀 You can check my PR. Also problem above was fixed in commit https://github.com/kercre123/wire-pod/pull/126/commits/291059cbbb3f29acafcfc247e8dc285c544ed8f7 Apologies for stupid questions but it's the first time I'm coding a project written in golang 😀

I have two extra question:

  1. On my robot if I set any non English location then robot stop talking. If I choose German, Japan, French my robot not talk anything. What could be the reason?
  2. In polish we can use sometimes two or three split words when we create question, example: STR_WEATHER_IN: {" w ", " we ", " na "}, it will be nice in the future to add ability to set more then one word 😀

Thanks for the great work you put into the project.