Add support for Streaming STT (realtime)

[Swimburger]

Currently, we don't have plans to add support for Streaming STT. We will reconsider this decision if there is enough demand for Streaming STT in Ruby. Add a 👍 to this issue if you want Streaming STT support or leave a comment.

[son1112]

How open are you to public contribution toward this effort?

[Swimburger]

@son1112, we'd be happy to accept it as a contribution. The only condition is that it should be clear that we don't officially support it, either via its name or some other way to mark it as experimental.

[Michael9311]

Is anyone working on this?

I'm just going to dump my client here in case anyone wants it. I'm working with Twilio so it defaults to mulaw. Remember to buffer, unpack and pack your binary.

require "faye/websocket" require "eventmachine" require "json"

class AssemblyClient ASSEMBLYAI_API_URL = "https://api.assemblyai.com/v2/realtime/token" ASSEMBLYAI_WS_BASE_URL = "wss://api.assemblyai.com/v2/realtime/ws"

attr_reader :token, :ws_url, :websocket attr_accessor :on_transcript

def initialize(api_key: ENV["ASSEMBLY_API_KEY"], sample_rate: 8000, encoding: "pcm_mulaw", expires_in: 3600) @api_key = api_key @sample_rate = sample_rate @encoding = encoding @expires_in = expires_in @token = fetch_token @ws_url = build_ws_url @on_transcript = nil end

def fetch_token headers = { "authorization" => @api_key, "content-type" => "application/json" } body = {"expires_in" => @expires_in}.to_json response = HTTP.headers(headers).post(ASSEMBLYAI_API_URL, body: body) if response.status.success? JSON.parse(response.body.to_s)["token"] else raise "Error fetching token: #{response.body}" end end

def build_ws_url "#{ASSEMBLYAI_WS_BASE_URL}?sample_rate=#{@sample_rate}&encoding=#{@encoding}&token=#{@token}" end

def start EM.run do puts @ws_url @websocket = Faye::WebSocket::Client.new(@ws_url)

  @websocket.on :open do |_event|
    puts "Assembly WebSocket connection opened"
  end

  @websocket.on :message do |event|
    handle_message(event.data)
  end

  @websocket.on :error do |event|
    puts "Assembly WebSocket error: #{event.message}"
  end

  @websocket.on :close do |event|
    puts "Assembly WebSocket closed with code: #{event.code}, reason: #{event.reason}"
    EM.stop
  end
end

end

def send_audio(binary_audio_data) if @websocket && @websocket.ready_state == Faye::WebSocket::API::OPEN

binary_audio_data = Base64.decode64(audio_data).bytes

  @websocket.send(binary_audio_data.pack("C*").bytes)
else
  puts "Calling Assembly before Open"
  # raise "WebSocket is not open. Call start() before sending audio."
end

end

def stop if @websocket @websocket.send(JSON.generate({"terminate_session" => true})) @websocket.close end end

private

def handle_message(message) puts message data = JSON.parse(message) if ["PartialTranscript", "FinalTranscript"].include?(data["message_type"]) transcript = data["text"] if @on_transcript @on_transcript.call(transcript, data["message_type"]) else puts "Transcription (#{data["message_type"]}): #{transcript}" end end end end