Integrate vosk-server in Jigasi

nshmyrev commented 4 years ago

Like watson here:

https://github.com/jitsi/jigasi/pull/87/files

nshmyrev commented 4 years ago

Going on https://github.com/jitsi/jigasi/pull/294

nshmyrev commented 4 years ago

Patch has been integrated, now we need to write more docs and integrate with https://github.com/jitsi/docker-jitsi-meet/

damienlaine commented 3 years ago

Greetings, Many thanks for this contribution. Any update on the integration with https://github.com/jitsi/docker-jitsi-meet/ ?

I'm currently trying some hack with a Dockerfile that uses a newer .deb, but i'm not sure i'll get this running, I use as a reference https://github.com/jitsi/docker-jitsi-meet/blob/master/jigasi/ that proposes a Dockerfile and some startup script, but i feel the "rootfs" might require modifications.

Thanks for your feedback, do you want to get updated on how things go on with my try ?

nshmyrev commented 3 years ago

Thanks for your feedback, do you want to get updated on how things go on with my try ?

Sure, please let us know.

damienlaine commented 3 years ago

Ok, here's what i did so far (these are the main steps, it's kinda dirty...) but i'm close to make it running

I got inspired by https://github.com/jitsi/docker-jitsi-meet/tree/master/jigasi structure and decided to build a new Dockerfile that would use the same structure. I downloaded the source form the Jigasi's github and mainly updated sip-communicator.properties with the newer version (from the source) I edited "etc" and removed all the Google bloat and tweaked the startup scripts... i.e (the current, and obsolete) Jigasi-Docker can't start if transcription env is set without references for GCloud API keys I built the .deb package Here's the Dockerfile i came with:

`ARG JITSI_REPO=jitsi FROM ${JITSI_REPO}/base-java

COPY jigasi_1.1-169-g4f05fbd-1_amd64.deb /jigasi_1.1-169-g4f05fbd-1_amd64.deb

RUN \ apt-dpkg-wrap apt-get update && \ apt-dpkg-wrap apt-get install -y jq libxss1 ruby-hocon && \ dpkg -i jigasi_1.1-169-g4f05fbd-1_amd64.deb && \ apt-cleanup

COPY rootfs/ /

VOLUME ["/config", "/tmp/transcripts"]l`

I then built my custom docker Jigasi image that seems to work as intended.

I afterward created a huge stack file (docker-compose) file, which is in my case very specific as i run with traefik-v2 in swarm mode (Traefik labels are bound to services) and i started the whole stack, including a Vosk transcription instance.

The Jigasi service seems to work as intended,.. unfortunately, so far, i'm unable to start a live transcript with Vosk. Watching the logs, i see my custom Jigasi registering itself in prosody, i also see logs appearing when i click the famous "CC" button in the interface but at least two things are broken :

I struggle to configure de MUC room name, which is always NULL, therefore the logs complains about trying to ask jigasi to join a non-existant room... it might have something to do with : net.java.sip.communicator.impl.protocol.sip.acc1.JITSI_MEET_ROOM_HEADER_NAME=X-Room-Name net.java.sip.communicator.impl.protocol.sip.acc1.JITSI_MEET_DOMAIN_BASE_HEADER_NAME=X-Domain-Base I tried with and without the bosh (http-bind) directives... same problems.

NOTE : Theses properties are not present anymore in the newer sip-communicator.properties file... it might also relate to the newer bosh directives that has been implemented... i dunno so far

Jigasi seems to fail to connect to SIP server with rejected credentials (i however used the same infos that were generated in the old sip-communicator.properties being written on disk when launching the old jigasi docker), something might be missing or wrong arround : net.java.sip.communicator.impl.protocol.sip.acc1.ACCOUNT_UID=SIP\:test@sip2sip.info net.java.sip.communicator.impl.protocol.sip.acc1.PASSWORD=xxxxxxxxxxxx net.java.sip.communicator.impl.protocol.sip.acc1.PROTOCOL_NAME=SIP net.java.sip.communicator.impl.protocol.sip.acc1.SERVER_ADDRESS=sip2sip.info

So i'm now trying to understand what newer properties are missing, updated or incompatible between the two sip-communicator.properties... i hope i won't find any service incompatibilities between my custom Jigasi and the classic docker-jitsi services

Well, that's really harsh... but i feel i'm almost there. Any tips for me ?

And any update of your side for supported implementation of Vosk in the jitsi-docker-meet repo ?

Thx again for your contribution

nshmyrev commented 3 years ago

sip server connection must be caused by docker port mapping, not very straighforward to configure that.

we need to keep google part for merge, just add vosk. I just add vosk.yaml to docker compose and thats it. Maybe language has to be configured from settings:

version: '3'

services:
    # Transcription server
    vosk:
        image: alphacep/kaldi-en:latest
        restart: ${RESTART_POLICY}
        expose:
            - '2700'
        networks:
            meet.jitsi:

for transcription errors you'd better check logs for more details.

damienlaine commented 3 years ago

Ok, i managed to make it work. I have transcripts.

The mis-configuration was in the bosh / http-bind section where i had a duplicate. So i was finaly able to run my custom new jigasi docker within the docker-jitsi set of services after seriously tweaking .env and sip-communicator properties.

Now that's working, i can confirm that the user-experience is broken (i saw some issues around) at the moment, the partial / logic pause / final transcript are displayed very wrongly, there's a lot of "duplicate lines" in the chat (the partials)... well that a good starting point. I guess the frontend code needs to get tweaked as well...

nshmyrev commented 3 years ago

@damienlaine ok, great. For duplicate line fix see https://github.com/alphacep/vosk-server/issues/89#issuecomment-783713983

siddhartha2000v commented 3 years ago

@nshmyrev just wanted to check if there is an update on this task, I am doing a fresh install of jigasi, would be good to know whats the best way to proceed currently. Basis the 2 threads as mentioned below: 1) Using @damienlaine's approach of setting up a separate dockerfile in this thread 2) Using the approach shonjs used of recompiling jigasi from source as per this - https://community.jitsi.org/t/custom-transcriptionservice-vosk-not-instantiable/86562/5 Which would be the better approach to do this currently? Thanks for suggesting.

nshmyrev commented 3 years ago

@siddhartha2000v not yet, but I'll try to look. There are still reports about duplicated lines.

alexk2037 commented 3 years ago

Hello, I am having some trouble getting Jigasi to connect with Vosk for transcription (no docker).

Background:

I have Jitsi and Jigasi and all the core components (prosody, jvb, jicofo, jigasi, etc) running on a Debian 10 server (not with docker; I followed the Jitsi self-hosting guide). I have Vosk running as a daemon on a different Ubuntu server.

I have set up the UI configuration so that I can see the ‘Start Subtitles’ button while in a video conference. I have also set up the prosody configuration with a transcriber user, so that when I click the subtitles button the Transcriber user joins the conference (as seen in the image below)

And in /etc/jitsi/jigasi/sip-communicator.properties I have:

where 172.31.251.237 is the IP address of the server running Vosk.

Problem:

However, behind the scenes, this is what happens when I click the ‘Start Subtitles button’ (relevant excerpt from /var/log/jitsi/jigasi.log)

2021-07-20 22:40:08.726 SEVERE: [123] org.jitsi.jigasi.transcription.VoskTranscriptionService.log() Error to finalize websocket connection for participant correctstaplerbatteryhorse/68bf9d29
java.lang.NullPointerException
        at org.jitsi.jigasi.transcription.VoskTranscriptionService$VoskWebsocketStreamingSession.end(VoskTranscriptionService.java:265)
        at org.jitsi.jigasi.transcription.Participant.left(Participant.java:541)
        at org.jitsi.jigasi.transcription.Transcriber.participantLeft(Transcriber.java:404)
        at org.jitsi.jigasi.TranscriptionGatewaySession.notifyChatRoomMemberLeft(TranscriptionGatewaySession.java:292)
        at org.jitsi.jigasi.JvbConference.memberPresenceChanged(JvbConference.java:1218)
        at net.java.sip.communicator.impl.protocol.jabber.ChatRoomJabberImpl.fireMemberPresenceEvent(ChatRoomJabberImpl.java:1953)
        at net.java.sip.communicator.impl.protocol.jabber.ChatRoomJabberImpl.leave(ChatRoomJabberImpl.java:1047)
        at net.java.sip.communicator.impl.protocol.jabber.ChatRoomJabberImpl.leave(ChatRoomJabberImpl.java:961)
        at org.jitsi.jigasi.JvbConference.leaveConferenceRoom(JvbConference.java:1133)
        at org.jitsi.jigasi.JvbConference.stop(JvbConference.java:582)
        at org.jitsi.jigasi.TranscriptionGatewaySession.lambda$notifyChatRoomMemberUpdated$0(TranscriptionGatewaySession.java:347)
        at java.base/java.lang.Thread.run(Thread.java:829)
2021-07-20 22:40:08.727 INFO: [123] org.jitsi.jigasi.JvbConference.memberPresenceChanged().1219 [ctx=16268351198411693964866] Member left : OWNER klffq_iape3ywol6@rnd1.dev2dev.net/NzBszygN
2021-07-20 22:40:08.729 WARNING: [123] org.jitsi.jigasi.transcription.Transcriber.log() correctstaplerbatteryhorse@conference.rnd1.dev2dev.net: participant with identifier 2651404c left while it did not exist
2021-07-20 22:40:08.730 INFO: [123] org.jitsi.jigasi.JvbConference.memberPresenceChanged().1219 [ctx=16268351198411693964866] Member left : MEMBER transcriber@recorder.rnd1.dev2dev.net
2021-07-20 22:40:08.747 SEVERE: [92] org.jitsi.impl.neomedia.RTPConnectorOutputStream.log() Failed to send a packet to target /172.31.234.2:10000:java.io.IOException: No active socket.
2021-07-20 22:40:08.750 SEVERE: [91] org.jitsi.impl.neomedia.RTPConnectorOutputStream.log() Failed to send a packet to target /172.31.234.2:10000:java.io.IOException: No active socket.
2021-07-20 22:40:08.760 INFO: [123] org.jitsi.jigasi.transcription.AbstractTranscriptPublisher.log() Wrote final transcript to /tmp/jigasi/transcripts/2021-07-21T02:38:42.024409Z_cc6a9b5c-4902-4d98-92da-a3d87ea350a4/transcript_2021-07-21T02:38:42.025080Z_8c416af8-1639-4d8d-bf6f-92c15f05a639.txt
2021-07-20 22:40:08.762 INFO: [123] org.jitsi.jigasi.AbstractGateway.notifyCallEnded().129 [ctx=16268351198411693964866] Removed session for call. Sessions:0
2021-07-20 22:40:08.765 INFO: [91] impl.protocol.jabber.OperationSetBasicTelephonyJabberImpl.registrationStateChanged().134 Jingle : OFF
2021-07-20 22:40:08.766 INFO: [91] org.jitsi.jigasi.CallManager$HangupCallThread.run().663 [ctx=16268351198411693964866] Removing account Jabber:2651404c@rnd1.dev2dev.net/2651404c
2021-07-20 22:40:08.766 INFO: [91] impl.protocol.jabber.OperationSetBasicTelephonyJabberImpl.registrationStateChanged().134 Jingle : OFF

As far as I can tell from the logs, Jigasi is trying to send information to 172.31.234.2:10000. However, I would like to configure the Jigasi/Jitsi server to send packets to 172.31.234.2:2700 instead. Please let me know if there is a good way to resolve this.

Any help or advice would be appreciated, thank you.

nshmyrev commented 3 years ago

Hello, I am having some trouble getting Jigasi to connect with Vosk for transcription (no docker).

as a quick solution you can probably run vosk server on port 10000 on that remote server.

alexk2037 commented 3 years ago

Thank you @nshmyrev , your solution worked. Now the vosk server is receiving a connection from the jigasi server.

alexk2037 commented 3 years ago

Hello, I am stumped on a problem connecting Jigasi and Vosk.

Similar to my previous post, I am running Vosk Server (python WebSocket) and Jigasi, but this time on one AWS machine (Debian 10).

I started a Jitsi meeting and clicked start subtitles. I said some words into the microphone. Then I stopped subtitles. These are the Vosk logs during those events:

Jul 29 18:53:50 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:asyncio:Using selector: EpollSelector
Jul 29 18:53:50 lmjmsjigasi.dev2dev.net python3.8[3456]: INFO:root:Listening on 0.0.0.0:2700
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: server - state = CONNECTING
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server - state = CONNECTING
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: server - event = connection_made(<_SelectorSocketTransport fd=7 read=idle write=<idle, bufsize=0>>)
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server - event = connection_made(<_SelectorSocketTransport fd=7 read=idle write=<idle, bufsize=0>>)
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: server - event = data_received(<255 bytes>)
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server - event = data_received(<255 bytes>)
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: server < GET / HTTP/1.1
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.server:server < GET / HTTP/1.1
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: server < Headers([('Accept-Encoding', 'gzip'), ('User-Agent', 'Jetty/9.4.15.v20190215'), ('Upgrade', 'websocket'), ('Connection', 'Upgrade'), ('Sec-WebSocket-Key', 'wAaJMPIxTPBW7JjN1dpRLQ=='), ('Sec-WebSocket-Version', '13'), ('Pragma', 'no-cache'), ('Cache-Control', 'no-cache'), ('Host', 'localhost:2700')])
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.server:server < Headers([('Accept-Encoding', 'gzip'), ('User-Agent', 'Jetty/9.4.15.v20190215'), ('Upgrade', 'websocket'), ('Connection', 'Upgrade'), ('Sec-WebSocket-Key', 'wAaJMPIxTPBW7JjN1dpRLQ=='), ('Sec-WebSocket-Version', '13'), ('Pragma', 'no-cache'), ('Cache-Control', 'no-cache'), ('Host', 'localhost:2700')])
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: server > HTTP/1.1 101 Switching Protocols
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.server:server > HTTP/1.1 101 Switching Protocols
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: server > Headers([('Upgrade', 'websocket'), ('Connection', 'Upgrade'), ('Sec-WebSocket-Accept', 'Td2+WNTa+GpKp6QGe4UNhCRqyx0='), ('Date', 'Thu, 29 Jul 2021 22:54:37 GMT'), ('Server', 'Python/3.8 websockets/9.1')])
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.server:server > Headers([('Upgrade', 'websocket'), ('Connection', 'Upgrade'), ('Sec-WebSocket-Accept', 'Td2+WNTa+GpKp6QGe4UNhCRqyx0='), ('Date', 'Thu, 29 Jul 2021 22:54:37 GMT'), ('Server', 'Python/3.8 websockets/9.1')])
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: server - state = OPEN
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server - state = OPEN
Jul 29 18:54:37 lmjmsjigasi.dev2dev.net python3.8[3456]: INFO:root:Connection from ('127.0.0.1', 41972)
Jul 29 18:54:57 lmjmsjigasi.dev2dev.net python3.8[3456]: server > Frame(fin=True, opcode=<Opcode.PING: 9>, data=b'\x9eY\x88\xac', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:54:57 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server > Frame(fin=True, opcode=<Opcode.PING: 9>, data=b'\x9eY\x88\xac', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:54:57 lmjmsjigasi.dev2dev.net python3.8[3456]: server - event = data_received(<10 bytes>)
Jul 29 18:54:57 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server - event = data_received(<10 bytes>)
Jul 29 18:54:57 lmjmsjigasi.dev2dev.net python3.8[3456]: server < Frame(fin=True, opcode=<Opcode.PONG: 10>, data=b'\x9eY\x88\xac', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:54:57 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server < Frame(fin=True, opcode=<Opcode.PONG: 10>, data=b'\x9eY\x88\xac', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:54:57 lmjmsjigasi.dev2dev.net python3.8[3456]: server - received solicited pong: 9e5988ac
Jul 29 18:54:57 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server - received solicited pong: 9e5988ac
Jul 29 18:55:17 lmjmsjigasi.dev2dev.net python3.8[3456]: server > Frame(fin=True, opcode=<Opcode.PING: 9>, data=b'\xd1\xb7\x99\xf4', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:55:17 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server > Frame(fin=True, opcode=<Opcode.PING: 9>, data=b'\xd1\xb7\x99\xf4', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:55:17 lmjmsjigasi.dev2dev.net python3.8[3456]: server - event = data_received(<10 bytes>)
Jul 29 18:55:17 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server - event = data_received(<10 bytes>)
Jul 29 18:55:17 lmjmsjigasi.dev2dev.net python3.8[3456]: server < Frame(fin=True, opcode=<Opcode.PONG: 10>, data=b'\xd1\xb7\x99\xf4', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:55:17 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server < Frame(fin=True, opcode=<Opcode.PONG: 10>, data=b'\xd1\xb7\x99\xf4', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:55:17 lmjmsjigasi.dev2dev.net python3.8[3456]: server - received solicited pong: d1b799f4
Jul 29 18:55:17 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server - received solicited pong: d1b799f4
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: server - event = data_received(<17 bytes>)
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server - event = data_received(<17 bytes>)
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: server < Frame(fin=True, opcode=<Opcode.TEXT: 1>, data=b'{"eof" : 1}', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server < Frame(fin=True, opcode=<Opcode.TEXT: 1>, data=b'{"eof" : 1}', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: server > Frame(fin=True, opcode=<Opcode.TEXT: 1>, data=b'{"text": ""}', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server > Frame(fin=True, opcode=<Opcode.TEXT: 1>, data=b'{"text": ""}', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: server - state = CLOSING
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server - state = CLOSING
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: server > Frame(fin=True, opcode=<Opcode.CLOSE: 8>, data=b'\x03\xe8', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server > Frame(fin=True, opcode=<Opcode.CLOSE: 8>, data=b'\x03\xe8', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: server - event = data_received(<8 bytes>)
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server - event = data_received(<8 bytes>)
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: server < Frame(fin=True, opcode=<Opcode.CLOSE: 8>, data=b'\x03\xe8', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server < Frame(fin=True, opcode=<Opcode.CLOSE: 8>, data=b'\x03\xe8', rsv1=False, rsv2=False, rsv3=False)
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: server - event = eof_received()
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server - event = eof_received()
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: server x half-closing TCP connection
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server x half-closing TCP connection
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: server - event = connection_lost(None)
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server - event = connection_lost(None)
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: server - state = CLOSED
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server - state = CLOSED
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: server x code = 1000, reason = [no reason]
Jul 29 18:55:24 lmjmsjigasi.dev2dev.net python3.8[3456]: DEBUG:websockets.protocol:server x code = 1000, reason = [no reason]

It appears as though Vosk and Jigasi are connected. However, there is no transcription happening. Any help explaining why that may be would be greatly appreciated.

I have Jigasi and Vosk running on a local virtual machine (Ubuntu 18.04). I performed the same procedure (start meeting, talk, end meeting) and transcription works well. A sample Vosk log for the working setup on my local machine is:

Jul 28 23:31:31 meet.kappen.net python3[3593]: DEBUG:asyncio:Using selector: EpollSelector
Jul 28 23:31:31 meet.kappen.net python3[3593]: INFO:root:Listening on 0.0.0.0:2700
Jul 28 23:32:00 meet.kappen.net python3[3593]: server - state = CONNECTING
Jul 28 23:32:00 meet.kappen.net python3[3593]: DEBUG:websockets.protocol:server - state = CONNECTING
Jul 28 23:32:00 meet.kappen.net python3[3593]: server - event = connection_made(<_SelectorSocketTransport fd=7 read=idle write=<idle, bufsize=0>>)
Jul 28 23:32:00 meet.kappen.net python3[3593]: DEBUG:websockets.protocol:server - event = connection_made(<_SelectorSocketTransport fd=7 read=idle write=<idle, bufsize=0>>)
Jul 28 23:32:00 meet.kappen.net python3[3593]: server - event = data_received(<255 bytes>)
Jul 28 23:32:00 meet.kappen.net python3[3593]: DEBUG:websockets.protocol:server - event = data_received(<255 bytes>)
Jul 28 23:32:00 meet.kappen.net python3[3593]: server < GET / HTTP/1.1
Jul 28 23:32:00 meet.kappen.net python3[3593]: DEBUG:websockets.server:server < GET / HTTP/1.1
Jul 28 23:32:00 meet.kappen.net python3[3593]: server < Headers([('Accept-Encoding', 'gzip'), ('User-Agent', 'Jetty/9.4.15.v20190215'), ('Upgrade', 'websocket'), ('Connection', 'Upgrade'), ('Sec-WebSocket-Key', 'M8Bdixft4NEHOI/eRSSwZw=='), ('Sec-WebSocket-Version', '13'), ('Pragma', 'no-cache'), ('Cache-Control', 'no-cache'), ('Host', 'localhost:2700')])
Jul 28 23:32:00 meet.kappen.net python3[3593]: DEBUG:websockets.server:server < Headers([('Accept-Encoding', 'gzip'), ('User-Agent', 'Jetty/9.4.15.v20190215'), ('Upgrade', 'websocket'), ('Connection', 'Upgrade'), ('Sec-WebSocket-Key', 'M8Bdixft4NEHOI/eRSSwZw=='), ('Sec-WebSocket-Version', '13'), ('Pragma', 'no-cache'), ('Cache-Control', 'no-cache'), ('Host', 'localhost:2700')])
Jul 28 23:32:00 meet.kappen.net python3[3593]: server > HTTP/1.1 101 Switching Protocols
Jul 28 23:32:00 meet.kappen.net python3[3593]: DEBUG:websockets.server:server > HTTP/1.1 101 Switching Protocols
Jul 28 23:32:00 meet.kappen.net python3[3593]: server > Headers([('Upgrade', 'websocket'), ('Connection', 'Upgrade'), ('Sec-WebSocket-Accept', 'uagNQJRBkub9uT5XYETTIEARuYU='), ('Date', 'Wed, 28 Jul 2021 23:32:00 GMT'), ('Server', 'Python/3.8 websockets/9.1')])
Jul 28 23:32:00 meet.kappen.net python3[3593]: DEBUG:websockets.server:server > Headers([('Upgrade', 'websocket'), ('Connection', 'Upgrade'), ('Sec-WebSocket-Accept', 'uagNQJRBkub9uT5XYETTIEARuYU='), ('Date', 'Wed, 28 Jul 2021 23:32:00 GMT'), ('Server', 'Python/3.8 websockets/9.1')])
Jul 28 23:32:00 meet.kappen.net python3[3593]: server - state = OPEN
Jul 28 23:32:00 meet.kappen.net python3[3593]: DEBUG:websockets.protocol:server - state = OPEN
Jul 28 23:32:00 meet.kappen.net python3[3593]: INFO:root:Connection from ('127.0.0.1', 54730)
Jul 28 23:32:02 meet.kappen.net python3[3593]: server - event = data_received(<45 bytes>)
Jul 28 23:32:02 meet.kappen.net python3[3593]: DEBUG:websockets.protocol:server - event = data_received(<45 bytes>)
Jul 28 23:32:02 meet.kappen.net python3[3593]: server < Frame(fin=True, opcode=<Opcode.TEXT: 1>, data=b'{"config" : {"sample_rate" : 48000.0 }}', rsv1=False, rsv2=False, rsv3=False)
Jul 28 23:32:02 meet.kappen.net python3[3593]: DEBUG:websockets.protocol:server < Frame(fin=True, opcode=<Opcode.TEXT: 1>, data=b'{"config" : {"sample_rate" : 48000.0 }}', rsv1=False, rsv2=False, rsv3=False)
Jul 28 23:32:02 meet.kappen.net python3[3593]: INFO:root:Config {'sample_rate': 48000.0}
Jul 28 23:32:02 meet.kappen.net python3[3593]: server - event = data_received(<32768 bytes>)
Jul 28 23:32:02 meet.kappen.net python3[3593]: DEBUG:websockets.protocol:server - event = data_received(<32768 bytes>)
Jul 28 23:32:02 meet.kappen.net python3[3593]: server - event = data_received(<15240 bytes>)
Jul 28 23:32:02 meet.kappen.net python3[3593]: DEBUG:websockets.protocol:server - event = data_received(<15240 bytes>)
Jul 28 23:32:02 meet.kappen.net python3[3593]: server < Frame(fin=True, opcode=<Opcode.BINARY: 2>, data=b'\xfe\xff\xfe\xff\xff\xff\xfe\xff\xfe\xff\xff\xff\xfc\xff\xfe\xff\x00\x00\xfe\xff\xfd\xff\xfd\xff\xfe\xff\x01\x00\xff\xff\xff\xff\x01\x00\x00\x00\xff\xff\x02\x00\x04\x00\x04\x00\x01\x00\x01\x00\x04\x00\x04\x00\x04\x00\x02\x00\x01\x00\x03\x00\x06\x00\x04\x00\x01\x00\x03\x00\x04\x00\x03\x00\x03\x00\x02\x00\xff\xff\x00\x00\x02\x00\xfd\xff\xf9\xff\xfe\xff\x01\x00\xfb\x
Jul 28 23:32:02 meet.kappen.net python3[3593]: x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x02\x00\x01\x00\x01\x00\x00\x00\xff\xff\xfd\xff\xfc\xff\xfb\xff\xfb\xff\xfb\xff\xfb\xff\xfb\xff\xfb\xff\xfb\xff\xfb\xff\xfb\xff\xfc\xff\xfc\xff\xfd\xff\xfd\xff\xfd\xff\xfe\xff\xfe\xff\xfe\xff\xff\xff\xff\xff\xff\xff\x00\x00\x01\x00\x02\x00\x03\x00\x04\x00\x04\x00\x05\x00\x05\x00\x05\x00\x06\x00\x06\x00\x06\x00\x06\x00\x05\x00\x05\x00\x04\x00\x03\x00\x01\x00\x01\x00\x01\x00\x
Jul 28 23:32:02 meet.kappen.net python3[3593]: x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\x00\x00\x01\x00\x01\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xfe\xff\xfd\xff\xfd\xff\xfd\xff\xfd\xff\xfd\xff\xfd\xff\xfd\xff\xfe\xff\xfe\xff\xff\xff\x00\x00\x00\x00\x01\x00\x02\x00\x02\x00\x02\x00\x02\x00\x02\x00\x02\x00\x02\x00\x02\x00\x02\x00\x03\x00\x03\x00\x03\x00\x02\x00\x02\x00\x01\x00\x00\x00\x00\x00\x00\x00\x
Jul 28 23:32:02 meet.kappen.net python3[3593]: x02\x00\x02\x00\x01\x00\x01\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x00\x00\xff\xff\xff\xff\xfe\xff\xfe\xff\xfe\xff\xff\xff\xfe\xff\xfe\xff\xfe\xff\xfe\xff\xfe\xff\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x
Jul 28 23:32:02 meet.kappen.net python3[3593]: DEBUG:websockets.protocol:server < Frame(fin=True, opcode=<Opcode.BINARY: 2>, data=b'\xfe\xff\xfe\xff\xff\xff\xfe\xff\xfe\xff\xff\xff\xfc\xff\xfe\xff\x00\x00\xfe\xff\xfd\xff\xfd\xff\xfe\xff\x01\x00\xff\xff\xff\xff\x01\x00\x00\x00\xff\xff\x02\x00\x04\x00\x04\x00\x01\x00\x01\x00\x04\x00\x04\x00\x04\x00\x02\x00\x01\x00\x03\x00\x06\x00\x04\x00\x01\x00\x03\x00\x04\x00\x03\x00\x03\x00\x02\x00\xff\xff\x00\x00\x02\x00\xfd\xff\xf9
Jul 28 23:32:02 meet.kappen.net python3[3593]: 0\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x02\x00\x01\x00\x01\x00\x00\x00\xff\xff\xfd\xff\xfc\xff\xfb\xff\xfb\xff\xfb\xff\xfb\xff\xfb\xff\xfb\xff\xfb\xff\xfb\xff\xfb\xff\xfc\xff\xfc\xff\xfd\xff\xfd\xff\xfd\xff\xfe\xff\xfe\xff\xfe\xff\xff\xff\xff\xff\xff\xff\x00\x00\x01\x00\x02\x00\x03\x00\x04\x00\x04\x00\x05\x00\x05\x00\x05\x00\x06\x00\x06\x00\x06\x00\x06\x00\x05\x00\x05\x00\x04\x00\x03\x00
Jul 28 23:32:02 meet.kappen.net python3[3593]: 0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\x00\x00\x01\x00\x01\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xfe\xff\xfd\xff\xfd\xff\xfd\xff\xfd\xff\xfd\xff\xfd\xff\xfd\xff\xfe\xff\xfe\xff\xff\xff\x00\x00\x00\x00\x01\x00\x02\x00\x02\x00\x02\x00\x02\x00\x02\x00\x02\x00\x02\x00\x02\x00\x02\x00\x03\x00\x03\x00\x03\x00\x02\x00\x02\x00\x01\x00
Jul 28 23:32:02 meet.kappen.net python3[3593]: 0\x03\x00\x03\x00\x02\x00\x02\x00\x02\x00\x01\x00\x01\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x01\x00\x00\x00\xff\xff\xff\xff\xfe\xff\xfe\xff\xfe\xff\xff\xff\xfe\xff\xfe\xff\xfe\xff\xfe\xff\xfe\xff\xfe\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
Jul 28 23:32:02 meet.kappen.net python3[3593]: server - event = data_received(<48008 bytes>)
Jul 28 23:32:02 meet.kappen.net python3[3593]: DEBUG:websockets.protocol:server - event = data_received(<48008 bytes>)

...

Jul 28 23:32:26 meet.kappen.net python3[3593]: server x half-closing TCP connection
Jul 28 23:32:26 meet.kappen.net python3[3593]: DEBUG:websockets.protocol:server x half-closing TCP connection
Jul 28 23:32:26 meet.kappen.net python3[3593]: server - event = eof_received()
Jul 28 23:32:26 meet.kappen.net python3[3593]: DEBUG:websockets.protocol:server - event = eof_received()
Jul 28 23:32:26 meet.kappen.net python3[3593]: server - event = connection_lost(None)
Jul 28 23:32:26 meet.kappen.net python3[3593]: DEBUG:websockets.protocol:server - event = connection_lost(None)
Jul 28 23:32:26 meet.kappen.net python3[3593]: server - state = CLOSED
Jul 28 23:32:26 meet.kappen.net python3[3593]: DEBUG:websockets.protocol:server - state = CLOSED
Jul 28 23:32:26 meet.kappen.net python3[3593]: server x code = 1000, reason = [no reason]
Jul 28 23:32:26 meet.kappen.net python3[3593]: DEBUG:websockets.protocol:server x code = 1000, reason = [no reason]

I'm not sure why my local transcription setup works, while the AWS transcription setup doesn't. For both machines:

I setup up vosk server in exactly the same way (I used python3.8, a virtual environment, and Vosk Server runs as a system daemon)
I set up and configured Jitsi in the same way
I configured Vosk to listen to port 2700

Some differences I can think of:

Each machine runs a different linux distro (Debian 10 vs Ubuntu 18.04)
I don't have access to configure ports on the AWS machine, so maybe there is some port issue, but I'm not sure

Any help would be greatly appreciated, thank you. Please let me know if you would like more information.

nshmyrev commented 3 years ago

Hello, I am stumped on a problem connecting Jigasi and Vosk.

You'd better open a new issue.

From the log it seems the problem is on Jigasi side, maybe some SIP issue or something. The server receives messages and sends back messages, it just doesn't receive the data. You'd better check Jigasi logs.

alphacep / vosk-api

Integrate vosk-server in Jigasi #113