search

alphacep / vosk-server

WebSocket, gRPC and WebRTC speech recognition server based on Vosk and Kaldi libraries
Apache License 2.0
930 stars 249 forks source link

Jitsi meet, duplicate strings #89

Open 88fingerslukee opened 3 years ago

88fingerslukee commented 3 years ago

Hi there,

I've hooked up vosk-server with jitsi-meet to transcribe my meetings. The issue I'm having is that I'm getting duplicate lines. See the below transcription:

Transcript of conference held at Jan 25, 2021 in room roomname@conference.meet.domain.com
Initial people present at 9:08:30 PM:

Transcript, started at 9:08:30 PM:
________________________________________________________________________________
<9:08:30 PM> User: joined the conference
<9:08:34 PM> User: testing 
<9:08:34 PM> User: testing testing 
<9:08:38 PM> User: hello 
<9:08:39 PM> User: hello this is 
<9:08:39 PM> User: hello this is user
<9:08:45 PM> User: why are you 
<9:08:45 PM> User: why are you doubling my 
<9:08:46 PM> User: why are you doubling my speech 
<9:08:51 PM> User left the conference
________________________________________________________________________________

End of transcript at Jan 25, 2021 9:08:53 PM

Is this a problem with vosk or a problem with the jitsi-meet client not sending some sort of command to end the stream and then return the text?

nicolas-san commented 3 years ago

Hi @88fingerslukee if you have documented your implementation, I will try it on my jitsi server, and come back to you, I'm really interested !

88fingerslukee commented 3 years ago

You bet. I'll put my settings here on Monday when I have a chance.

Thanks!

nicolas-san commented 3 years ago

have you simply used the jigasi setup with vosk ? https://github.com/jitsi/jigasi#vosk-configuration @88fingerslukee

88fingerslukee commented 3 years ago

have you simply used the jigasi setup with vosk ? https://github.com/jitsi/jigasi#vosk-configuration @88fingerslukee

Sort of. The docker container would not run the asr_server.py application correctly due to some pip package errors. I ended up running the server application natively (downloaded the binaries and then the language models).

The server app ran fine after that but it obviously wasn't providing the right end of message command or something.

88fingerslukee commented 3 years ago

I just tried again and this is the output, it's an error now. Not even getting transcriptions:

# ./asr_server.py /opt/vosk-model/vosk-model-en-us-daanzu-20200905/
LOG (VoskAPI:ReadDataFiles():vosk/model.cc:194) Decoding params beam=13 max-active=7000 lattice-beam=6
LOG (VoskAPI:ReadDataFiles():vosk/model.cc:197) Silence phones 1:2:3:4:5:6:7:8:9:10:11:12:13:14:15
LOG (VoskAPI:RemoveOrphanNodes():nnet-nnet.cc:948) Removed 1 orphan nodes.
LOG (VoskAPI:RemoveOrphanComponents():nnet-nnet.cc:847) Removing 2 orphan components.
LOG (VoskAPI:Collapse():nnet-utils.cc:1488) Added 1 components, removed 2
LOG (VoskAPI:CompileLooped():nnet-compile-looped.cc:345) Spent 0.039654 seconds in looped compilation.
LOG (VoskAPI:ReadDataFiles():vosk/model.cc:221) Loading i-vector extractor from /opt/vosk-model/vosk-model-en-us-daanzu-20200905//ivector/final.ie
LOG (VoskAPI:ComputeDerivedVars():ivector-extractor.cc:183) Computing derived variables for iVector extractor
LOG (VoskAPI:ComputeDerivedVars():ivector-extractor.cc:204) Done.
LOG (VoskAPI:ReadDataFiles():vosk/model.cc:246) Loading HCLG from /opt/vosk-model/vosk-model-en-us-daanzu-20200905//graph/HCLG.fst
LOG (VoskAPI:ReadDataFiles():vosk/model.cc:265) Loading words from /opt/vosk-model/vosk-model-en-us-daanzu-20200905//graph/words.txt
LOG (VoskAPI:ReadDataFiles():vosk/model.cc:273) Loading winfo /opt/vosk-model/vosk-model-en-us-daanzu-20200905//graph/phones/word_boundary.int
LOG (VoskAPI:ReadDataFiles():vosk/model.cc:281) Loading CARPA model from /opt/vosk-model/vosk-model-en-us-daanzu-20200905//rescore/G.carpa
Error in connection handler
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 827, in transfer_data
    message = await self.read_message()
  File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 895, in read_message
    frame = await self.read_data_frame(max_size=self.max_size)
  File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 971, in read_data_frame
    frame = await self.read_frame(max_size)
  File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 1051, in read_frame
    extensions=self.extensions,
  File "/usr/local/lib/python3.6/dist-packages/websockets/framing.py", line 105, in read
    data = await reader(2)
  File "/usr/lib/python3.6/asyncio/streams.py", line 672, in readexactly
    raise IncompleteReadError(incomplete, n)
asyncio.streams.IncompleteReadError: 0 bytes read on a total of 2 expected bytes

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/websockets/server.py", line 191, in handler
    await self.ws_handler(self, path)
  File "./asr_server.py", line 55, in recognize
    message = await websocket.recv()
  File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 509, in recv
    await self.ensure_open()
  File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 803, in ensure_open
    raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason
nshmyrev commented 3 years ago

I just tried again and this is the output, it's an error now. Not even getting transcriptions:

This error is just about broken connection. The details must be on the client side (jitsi), not on the server side. Check jitsi logs for details please.

nicolas-san commented 3 years ago

at your knowledge, is it possible to have the real time transcription on caption in the room like a subtitle ?

88fingerslukee commented 3 years ago

I just tried again and this is the output, it's an error now. Not even getting transcriptions:

This error is just about broken connection. The details must be on the client side (jitsi), not on the server side. Check jitsi logs for details please.

This is what I have on the Jitsi side:

2021-02-22 12:41:12.135 INFO: [845] org.jitsi.jigasi.xmpp.CallControl.handleDialIq().195 [ctx=1614026472135211964760] Got dial request null -> jitsi_meet_transcribe room: room@conference.meet.domain.com
2021-02-22 12:41:12.136 INFO: [845] org.jitsi.jigasi.JvbConference.start().490 [ctx=1614026472135211964760] Starting JVB conference room: room@conference.meet.domain.com
2021-02-22 12:41:12.138 INFO: [845] org.jitsi.jigasi.JvbConference.setXmppProvider().633 [ctx=1614026472135211964760] Using ProtocolProviderServiceJabberImpl(Jabber:transcriber@meet.domain.com/transcriber)
2021-02-22 12:41:12.142 INFO: [995] org.igniterealtime.jbosh.BOSHClient.init() Starting with 1 request processors
2021-02-22 12:41:12.215 INFO: [995] impl.protocol.jabber.OperationSetBasicTelephonyJabberImpl.registrationStateChanged().127 Jingle : ON
2021-02-22 12:41:12.216 INFO: [995] org.jitsi.jigasi.JvbConference.registrationStateChanged().684 [ctx=1614026472135211964760] Registering XMPP.
2021-02-22 12:41:12.254 INFO: [995] impl.protocol.jabber.ProtocolProviderServiceJabberImpl.authenticated().2423 Authenticated: false
2021-02-22 12:41:12.257 INFO: [995] org.jitsi.jigasi.JvbConference.joinConferenceRoom().775 [ctx=1614026472135211964760] Joining JVB conference room: room@conference.meet.domain.com
2021-02-22 12:41:12.282 INFO: [999] impl.protocol.jabber.ChatRoomJabberImpl.joined().1323 room@conference.meet.domain.com/focus has joined the room@conference.meet.domain.com chat room.
2021-02-22 12:41:12.283 INFO: [999] impl.protocol.jabber.ChatRoomJabberImpl.joined().1323 room@conference.meet.domain.com/462fff7b has joined the room@conference.meet.domain.com chat room.
2021-02-22 12:41:12.284 INFO: [999] impl.protocol.jabber.ChatRoomJabberImpl.joined().1323 room@conference.meet.domain.com/transcriber has joined the room@conference.meet.domain.com chat room.
2021-02-22 12:41:12.285 SEVERE: [995] org.jitsi.jigasi.JvbConference.registrationStateChanged().673 [ctx=1614026472135211964760] Registered bosh sid: b3c11c24-6363-42d6-a5ae-94fb285ffcd1
2021-02-22 12:41:12.398 INFO: [1003] impl.protocol.jabber.IceUdpTransportManager.createIceAgent().347 End gathering harvester within 2 ms
2021-02-22 12:41:12.443 INFO: [1003] impl.protocol.jabber.CallPeerMediaHandlerJabberImpl.harvestCandidates().1198 End candidate harvest within 45 ms
2021-02-22 12:41:12.444 INFO: [1003] org.jitsi.jigasi.JvbConference.incomingCallReceived().1380 [ctx=1614026472135211964760] Got invite from focus
2021-02-22 12:41:12.444 INFO: [1003] org.jitsi.jigasi.stats.StatsHandler.getStatsServiceWrapper().276 [ctx=1614026472135211964760] Jitsi-stats library initializing for account: Jabber:jigasi@auth.meet.domain.com
2021-02-22 12:41:12.444 WARNING: [1003] org.jitsi.stats.media.StatsServiceFactory.log() KeyID/keyPath missing, will try using appSecret
2021-02-22 12:41:12.445 SEVERE: [1003] org.jitsi.jigasi.stats.StatsHandler.error().398 [ctx=1614026472135211964760] Jitsi-stats library failed to initialize with reason: Missing parameres and error message: appSecret missing
2021-02-22 12:41:12.445 WARNING: [1003] org.jitsi.stats.media.StatsServiceFactory.log() appSecret missing. Skipping callstats init
2021-02-22 12:41:12.508 INFO: [103] org.jitsi.jigasi.TranscriptionGatewaySession.log() Transcriber: Media Device Audio
2021-02-22 12:41:12.511 INFO: [103] service.protocol.media.MediaHandler.registerDynamicPTsWithStream().1003 Dynamic PT map: 126=rtpmap:-1 telephone-event/8000; 111=rtpmap:-1 opus/48000/2 fmtp:useinbandfec=1;minptime=10; 103=rtpmap:-1 unknown/90000;
2021-02-22 12:41:12.512 INFO: [103] service.protocol.media.MediaHandler.registerDynamicPTsWithStream().1020 PT overrides [103->104 ]
2021-02-22 12:41:12.514 INFO: [103] service.protocol.media.CallPeerMediaHandler.start().1961 Starting
2021-02-22 12:41:12.540 INFO: [103] org.jitsi.jigasi.JvbConference.callStateChanged().1491 [ctx=1614026472135211964760] JVB conference call IN_PROGRESS.
2021-02-22 12:41:12.540 WARNING: [103] org.jitsi.jigasi.stats.StatsHandler.startConferencePeriodicRunnable().329 [ctx=1614026472135211964760] Stats handler missing for call:Call: id=16140264723921603740496 peers=1
2021-02-22 12:41:19.215 SEVERE: [1015] org.jitsi.jigasi.transcription.VoskTranscriptionService.log() Error while streaming audio data to transcription service

Something about an appsecret?

nshmyrev commented 3 years ago

at your knowledge, is it possible to have the real time transcription on caption in the room like a subtitle ?

Yes, sure

nshmyrev commented 3 years ago

Something about an appsecret?

No, it is certainly not about appsecret. Jigasi requires 8khz transcription and will not work with daanzu model you are trying. You need to try 8khz model (aspire) or even docker directly.

Docker en image must work, if doesn't you'd better describe the error in details.

88fingerslukee commented 3 years ago

Something about an appsecret?

No, it is certainly not about appsecret. Jigasi requires 8khz transcription and will not work with daanzu model you are trying. You need to try 8khz model (aspire) or even docker directly.

Docker en image must work, if doesn't you'd better describe the error in details.

I retried the docker installation and it worked, once. It still has duplicate strings like my original post. However, Then when I close the meeting and open a new one and try transcription, I get the following errors:

2021-02-22 13:35:30.336 SEVERE: [378] org.jitsi.jigasi.transcription.VoskTranscriptionService.log() Error while streaming audio data to transcription service
java.lang.NullPointerException
        at org.jitsi.jigasi.transcription.Participant.getIdentityExtensionOrNull(Participant.java:373)
        at org.jitsi.jigasi.transcription.Participant.getAvatarUrl(Participant.java:264)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.addParticipantDescription(LocalJsonTranscriptHandler.java:406)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.addEventDescriptions(LocalJsonTranscriptHandler.java:343)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.createTranscriptionJSONObject(LocalJsonTranscriptHandler.java:255)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.publish(LocalJsonTranscriptHandler.java:230)
        at org.jitsi.jigasi.transcription.TranscriptHandler.publishTranscriptionResult(TranscriptHandler.java:146)
        at org.jitsi.jigasi.TranscriptionGatewaySession.sendTranscriptionResultToRoom(TranscriptionGatewaySession.java:657)
        at org.jitsi.jigasi.TranscriptionGatewaySession.notify(TranscriptionGatewaySession.java:394)
        at org.jitsi.jigasi.transcription.Transcriber.notify(Transcriber.java:813)
        at org.jitsi.jigasi.transcription.Participant.notify(Participant.java:583)
        at org.jitsi.jigasi.transcription.VoskTranscriptionService$VoskWebsocketStreamingSession.onMessage(VoskTranscriptionService.java:221)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:71)
        at org.eclipse.jetty.websocket.common.events.annotated.OptionalSessionCallableMethod.call(OptionalSessionCallableMethod.java:72)
        at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onTextMessage(JettyAnnotatedEventDriver.java:241)
        at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
        at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onTextFrame(JettyAnnotatedEventDriver.java:233)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:150)
        at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:319)
        at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:201)
        at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:221)
        at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:246)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:577)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:434)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)
2021-02-22 13:35:30.338 SEVERE: [378] org.jitsi.jigasi.transcription.VoskTranscriptionService.log() Error while streaming audio data to transcription service
java.io.EOFException: Disconnected
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:323)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:193)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:229)
        at org.eclipse.jetty.websocket.common.WebSocketSession.close(WebSocketSession.java:113)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:171)
        at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:319)
        at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:201)
        at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:221)
        at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:246)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:577)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:434)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)
2021-02-22 13:35:30.857 SEVERE: [429] org.jitsi.jigasi.transcription.VoskTranscriptionService.log() Error sending single req
java.lang.NullPointerException
        at org.jitsi.jigasi.transcription.Participant.getIdentityExtensionOrNull(Participant.java:373)
        at org.jitsi.jigasi.transcription.Participant.getAvatarUrl(Participant.java:264)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.addParticipantDescription(LocalJsonTranscriptHandler.java:406)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.addEventDescriptions(LocalJsonTranscriptHandler.java:343)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.createTranscriptionJSONObject(LocalJsonTranscriptHandler.java:255)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.publish(LocalJsonTranscriptHandler.java:230)
        at org.jitsi.jigasi.transcription.TranscriptHandler.publishTranscriptionResult(TranscriptHandler.java:146)
        at org.jitsi.jigasi.TranscriptionGatewaySession.sendTranscriptionResultToRoom(TranscriptionGatewaySession.java:657)
        at org.jitsi.jigasi.TranscriptionGatewaySession.notify(TranscriptionGatewaySession.java:394)
        at org.jitsi.jigasi.transcription.Transcriber.notify(Transcriber.java:813)
        at org.jitsi.jigasi.transcription.Participant.notify(Participant.java:583)
        at org.jitsi.jigasi.transcription.VoskTranscriptionService.sendSingleRequest(VoskTranscriptionService.java:115)
        at org.jitsi.jigasi.transcription.Participant.lambda$sendRequest$1(Participant.java:696)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

So something is breaking after an initial successful transcription and then there are the duplicate string issues.

Thoughts?

nshmyrev commented 3 years ago

It still has duplicate strings like my original post.

For duplicate lines try to change !partial to simply partial here:

https://github.com/jitsi/jigasi/blob/4f05fbd57864ac0836b815dc32bce72dc9677210/src/main/java/org/jitsi/jigasi/transcription/VoskTranscriptionService.java#L224

hopefully it will work.

As for exception, needs investigation. You need to check both client and the server. Feels like server exited for some reason.

88fingerslukee commented 3 years ago

For duplicate lines try to change !partial to simply partial here:

https://github.com/jitsi/jigasi/blob/4f05fbd57864ac0836b815dc32bce72dc9677210/src/main/java/org/jitsi/jigasi/transcription/VoskTranscriptionService.java#L224

Forgive my ignorance, but since I'm not building jitsi from source, am I able to make these changes after jitsi is installed?

nshmyrev commented 3 years ago

Forgive my ignorance, but since I'm not building jitsi from source, am I able to make these changes after jitsi is installed?

No

marcmonfort commented 3 years ago

Something about an appsecret?

No, it is certainly not about appsecret. Jigasi requires 8khz transcription and will not work with daanzu model you are trying. You need to try 8khz model (aspire) or even docker directly. Docker en image must work, if doesn't you'd better describe the error in details.

I retried the docker installation and it worked, once. It still has duplicate strings like my original post. However, Then when I close the meeting and open a new one and try transcription, I get the following errors:

2021-02-22 13:35:30.336 SEVERE: [378] org.jitsi.jigasi.transcription.VoskTranscriptionService.log() Error while streaming audio data to transcription service
java.lang.NullPointerException
        at org.jitsi.jigasi.transcription.Participant.getIdentityExtensionOrNull(Participant.java:373)
        at org.jitsi.jigasi.transcription.Participant.getAvatarUrl(Participant.java:264)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.addParticipantDescription(LocalJsonTranscriptHandler.java:406)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.addEventDescriptions(LocalJsonTranscriptHandler.java:343)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.createTranscriptionJSONObject(LocalJsonTranscriptHandler.java:255)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.publish(LocalJsonTranscriptHandler.java:230)
        at org.jitsi.jigasi.transcription.TranscriptHandler.publishTranscriptionResult(TranscriptHandler.java:146)
        at org.jitsi.jigasi.TranscriptionGatewaySession.sendTranscriptionResultToRoom(TranscriptionGatewaySession.java:657)
        at org.jitsi.jigasi.TranscriptionGatewaySession.notify(TranscriptionGatewaySession.java:394)
        at org.jitsi.jigasi.transcription.Transcriber.notify(Transcriber.java:813)
        at org.jitsi.jigasi.transcription.Participant.notify(Participant.java:583)
        at org.jitsi.jigasi.transcription.VoskTranscriptionService$VoskWebsocketStreamingSession.onMessage(VoskTranscriptionService.java:221)
        at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:71)
        at org.eclipse.jetty.websocket.common.events.annotated.OptionalSessionCallableMethod.call(OptionalSessionCallableMethod.java:72)
        at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onTextMessage(JettyAnnotatedEventDriver.java:241)
        at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
        at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onTextFrame(JettyAnnotatedEventDriver.java:233)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:150)
        at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:319)
        at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:201)
        at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:221)
        at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:246)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:577)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:434)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)
2021-02-22 13:35:30.338 SEVERE: [378] org.jitsi.jigasi.transcription.VoskTranscriptionService.log() Error while streaming audio data to transcription service
java.io.EOFException: Disconnected
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.disconnect(AbstractWebSocketConnection.java:323)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:193)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:229)
        at org.eclipse.jetty.websocket.common.WebSocketSession.close(WebSocketSession.java:113)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:171)
        at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:319)
        at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:201)
        at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:221)
        at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:246)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:577)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:434)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:748)
2021-02-22 13:35:30.857 SEVERE: [429] org.jitsi.jigasi.transcription.VoskTranscriptionService.log() Error sending single req
java.lang.NullPointerException
        at org.jitsi.jigasi.transcription.Participant.getIdentityExtensionOrNull(Participant.java:373)
        at org.jitsi.jigasi.transcription.Participant.getAvatarUrl(Participant.java:264)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.addParticipantDescription(LocalJsonTranscriptHandler.java:406)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.addEventDescriptions(LocalJsonTranscriptHandler.java:343)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.createTranscriptionJSONObject(LocalJsonTranscriptHandler.java:255)
        at org.jitsi.jigasi.transcription.LocalJsonTranscriptHandler.publish(LocalJsonTranscriptHandler.java:230)
        at org.jitsi.jigasi.transcription.TranscriptHandler.publishTranscriptionResult(TranscriptHandler.java:146)
        at org.jitsi.jigasi.TranscriptionGatewaySession.sendTranscriptionResultToRoom(TranscriptionGatewaySession.java:657)
        at org.jitsi.jigasi.TranscriptionGatewaySession.notify(TranscriptionGatewaySession.java:394)
        at org.jitsi.jigasi.transcription.Transcriber.notify(Transcriber.java:813)
        at org.jitsi.jigasi.transcription.Participant.notify(Participant.java:583)
        at org.jitsi.jigasi.transcription.VoskTranscriptionService.sendSingleRequest(VoskTranscriptionService.java:115)
        at org.jitsi.jigasi.transcription.Participant.lambda$sendRequest$1(Participant.java:696)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

So something is breaking after an initial successful transcription and then there are the duplicate string issues.

Thoughts?

I got the same error. Apparently if you start the transcription (CC button) with the microphone deactivated, you stop getting this error and works correctly.

88fingerslukee commented 3 years ago

I got the same error. Apparently if you start the transcription (CC button) with the microphone deactivated, you stop getting this error and works correctly.

do you still get duplicate strings?

marcmonfort commented 3 years ago

I got the same error. Apparently if you start the transcription (CC button) with the microphone deactivated, you stop getting this error and works correctly.

do you still get duplicate strings?

Yes. I think you still need to make this change.

It still has duplicate strings like my original post.

For duplicate lines try to change !partial to simply partial here:

https://github.com/jitsi/jigasi/blob/4f05fbd57864ac0836b815dc32bce72dc9677210/src/main/java/org/jitsi/jigasi/transcription/VoskTranscriptionService.java#L224

hopefully it will work.

As for exception, needs investigation. You need to check both client and the server. Feels like server exited for some reason.

nshmyrev commented 3 years ago

Given https://github.com/jitsi/jigasi/pull/344 is merged, this should be fixed. Thank you @88fingerslukee

For new problems like exceptions please open new issues.

Closing this.

88fingerslukee commented 3 years ago

@nshmyrev so I downloaded and installed the nightly build that was created with the new PR and it still gives me the partial strings. For some reason they aren't being discarded.

I wish I were a java developer but I'm not. I'm a little embarassed that I pushed wrong code so I'd like to fix this for everybody ASAP. I've set up a dev instance of jitsi and vosk so I'm happy to test changes to your code to see if we can get it to work.

Please let me know your thoughts.

nshmyrev commented 3 years ago

I'm a little embarassed that I pushed wrong code

I don't think it is wrong, I think just nightly build didn't update properly. Try to build from source.

nshmyrev commented 3 years ago

Can someone advise what status is for this to be fixed? We work a lot with Deaf and Hard of Hearing and really need to get this working. We will gladly test and provide feedback if it would help.

What is the problem you have exactly?

VoskTranscriptionService is definitely included in this nightly build package:

https://download.jitsi.org/unstable/jigasi_1.1-185-g09b2999-1_amd64.deb

jbg commented 3 years ago

We're still getting duplicate strings with Vosk & latest Jigasi from Git master. Let me know if I can provide any useful debugging info.

nshmyrev commented 3 years ago

It is a bug in jigasi vosk module implementation it seems. We need to handle partial results more intelligently.

alexk2037 commented 3 years ago

@nshmyrev If you have any ideas for how to improve the handling of partial results I am more than happy to test them out.

neurosys-zero commented 2 years ago

+1

wokawoka commented 2 years ago

following it would be a great solution

abdullahrkw commented 2 years ago

Any idea if this will get fixed?

rpgresearch commented 2 years ago

+1