search

wkok / openai-clojure

Clojure functions to drive the OpenAI API
https://cljdoc.org/d/net.clojars.wkok/openai-clojure
MIT License
208 stars 28 forks source link

Could I have an example for how to call the transcriptions on Azure? #71

Closed HanjoPurebuddha closed 1 month ago

HanjoPurebuddha commented 2 months ago

I've been playing with the new version and not yet figured out how to provide all of the correct parameters.

For reference, here's my error:

; Failed to transcribe audio with error: Interceptor Exception: Could not coerce value to schema: {:deployment-id (not (instance? java.lang.String nil))} And my code:

with multipart/form-data content type
        request {:multipart [{:name "file" ;; (15) File upload for multipart/form-data
                              :content audio-file ;; (16) Audio file as content
                              :filename (.getName audio-file) ;; (17) File name for the upload
                              :mime-type "audio/m4a"} ;; (18) MIME type for the audio file
                             {:name "model" ;; (19) Model specification
                              :content model}
                             {:name "language" ;; (20) Language specification
                              :content language}
                             {:name "response_format" ;; (21) Response format specification
                              :content response-format}
                             {:name "temperature" ;; (22) Temperature specification
                              :content (str temperature)} ;; (23) Temperature as string
                             (when prompt ;; (24) Include prompt if provided
                               {:name "prompt"
                                :content prompt})]}]
(let [response (api/create-transcription request ;; (26) Send request via Azure
                                               {:impl :azure ;; (29) Specify Azure implementation
                                                :api-key azure-key ;; (30) Use Azure API key
                                                :api-endpoint azure-endpoint})]

I think I'm missing something! I figured the deployment-id was just the model parameter. Maybe my Azure deployment isn't set-up right, but the model name matches the deployment name.

Thanks for any help.

wkok commented 2 months ago

Could you try adding the :model parameter (the deployment id) to the request map directly?

Example in the test: https://github.com/wkok/openai-clojure/blob/main/test%2Fwkok%2Fopenai_clojure%2Fapi_test.clj#L145

Disclaimer - I can't actually test / confirm a working example unfortunately as I don't have access to an Azure instance

HanjoPurebuddha commented 2 months ago

I tried some simple code and couldn't seem to get it to work. Let me know if I'm doing it right!

Code:

(api/create-transcription
     {{:name "file" ;; (15) File upload for multipart/form-data
       :content (io/file (str "D:/Downloads/file_example_MP3_700KB.mp3")) ;; (16) Audio file as content
       :filename "example60kf" ;; (17) File name for the upload
       :mime-type "audio/mp3"}

      {:model "whisper-1"}} ;; Specify the model for the transcription
     {:impl :azure ;; Use Azure implementation
      :api-key (System/getenv "AZURE_OPENAI_API_KEY_SOUTHINDIA") ;; Fetch the API key from environment variable
      :api-endpoint (System/getenv "AZURE_OPENAI_API_ENDPOINT_SOUTHINDIA")})

Error:

; #object[java.io.File 0x772b21b9 D:\Downloads\file_example_MP3_700KB.mp3]
; An error occurred during transcription:  Interceptor Exception: Could not coerce value to schema: {:deployment-id (not (instance? java.lang.String nil))}
; {:execution-id #uuid "4d56557f-144b-4ee5-ba5f-1ff11c06eb8b", :stage :enter, :interceptor :martian.interceptors/url, :type :schema-tools.coerce/error, :exception #error {
;  :cause Could not coerce value to schema: {:deployment-id (not (instance? java.lang.String nil))}
;  :data {:type :schema-tools.coerce/error, :schema {:deployment-id java.lang.String}, :value {:api-version 2024-06-01, :deployment-id nil, :martian.core/body {{:name file, :content #object[java.io.File 0x1420222e D:\Downloads\file_example_MP3_700KB.mp3], :filename example60kf, :mime-type audio/mp3} {:model whisper-1}}, :wkok.openai-clojure.core/options {:impl :azure, :api-key <<REDACTED>>, :api-endpoint https://south-india-sophia.openai.azure.com}}, :error {:deployment-id (not (instance? java.lang.String nil))}}
;  :via
;  [{:type clojure.lang.ExceptionInfo
;    :message Could not coerce value to schema: {:deployment-id (not (instance? java.lang.String nil))}
;    :data {:type :schema-tools.coerce/error, :schema {:deployment-id java.lang.String}, :value {:api-version 2024-06-01, :deployment-id nil, :martian.core/body {{:name file, :content #object[java.io.File 0x1420222e D:\Downloads\file_example_MP3_700KB.mp3], :filename example60kf, :mime-type audio/mp3} {:model whisper-1}}, :wkok.openai-clojure.core/options {:impl :azure, :api-key <<REDACTED>>, :api-endpoint https://south-india-sophia.openai.azure.com}}, :error {:deployment-id (not (instance? java.lang.String nil))}}
;    :at [schema_tools.coerce$coerce_or_error_BANG_ invokeStatic coerce.cljc 24]}]
;  :trace
;  [[schema_tools.coerce$coerce_or_error_BANG_ invokeStatic coerce.cljc 24]
;   [schema_tools.coerce$coerce_or_error_BANG_ invoke coerce.cljc 20]
;   [schema_tools.coerce$coercer$fn__10522 invoke coerce.cljc 157]
;   [schema_tools.coerce$coerce invokeStatic coerce.cljc 168]
;   [schema_tools.coerce$coerce invoke coerce.cljc 159]
;   [schema_tools.coerce$coerce invokeStatic coerce.cljc 166]
;   [schema_tools.coerce$coerce invoke coerce.cljc 159]
;   [martian.schema$coerce_data invokeStatic schema.cljc 50]
;   [martian.schema$coerce_data doInvoke schema.cljc 39]
;   [clojure.lang.RestFn invoke RestFn.java 464]
;   [martian.interceptors$coerce_data invokeStatic interceptors.cljc 39]
;   [martian.interceptors$coerce_data invoke interceptors.cljc 38]
;   [martian.core$url_for invokeStatic core.cljc 92]
;   [martian.core$url_for invoke core.cljc 86]
;   [martian.core$url_for invokeStatic core.cljc 88]
;   [martian.core$url_for invoke core.cljc 86]
;   [clojure.core$partial$fn__5908 invoke core.clj 2642]
;   [martian.interceptors$fn__12172 invokeStatic interceptors.cljc 36]
;   [martian.interceptors$fn__12172 invoke interceptors.cljc 35]
;   [tripod.context$try_f invokeStatic context.cljc 32]
;   [tripod.context$try_f invoke context.cljc 22]
;   [tripod.context$enter_all_with_binding invokeStatic context.cljc 101]
;   [tripod.context$enter_all_with_binding invoke context.cljc 82]
;   [tripod.context$enter_all$fn__8316 invoke context.cljc 115]
;   [clojure.lang.AFn applyToHelper AFn.java 152]
;   [clojure.lang.AFn applyTo AFn.java 144]
;   [clojure.core$apply invokeStatic core.clj 667]
;   [clojure.core$with_bindings_STAR_ invokeStatic core.clj 1990]
;   [clojure.core$with_bindings_STAR_ doInvoke core.clj 1990]
;   [clojure.lang.RestFn invoke RestFn.java 425]
;   [tripod.context$enter_all invokeStatic context.cljc 114]
;   [tripod.context$enter_all invoke context.cljc 108]
;   [tripod.context$execute invokeStatic context.cljc 199]
;   [tripod.context$execute invoke context.cljc 198]
;   [martian.core$response_for invokeStatic core.cljc 120]
;   [martian.core$response_for invoke core.cljc 114]
;   [wkok.openai_clojure.core$response_for invokeStatic core.clj 36]
;   [wkok.openai_clojure.core$response_for invoke core.clj 16]
;   [wkok.openai_clojure.api$create_transcription invokeStatic api.clj 197]
;   [wkok.openai_clojure.api$create_transcription invoke api.clj 181]
;   [openai_transcribe_clojure.core$handle_transcription_error invokeStatic core.clj 121]
;   [openai_transcribe_clojure.core$handle_transcription_error invoke core.clj 115]
;   [openai_transcribe_clojure.core$eval26119 invokeStatic NO_SOURCE_FILE 162]
;   [openai_transcribe_clojure.core$eval26119 invoke NO_SOURCE_FILE 162]
;   [clojure.lang.Compiler eval Compiler.java 7201]
;   [clojure.lang.Compiler eval Compiler.java 7156]
;   [clojure.core$eval invokeStatic core.clj 3216]
;   [clojure.core$eval invoke core.clj 3212]
;   [nrepl.middleware.interruptible_eval$evaluate$fn__1359$fn__1360 invoke interruptible_eval.clj 87]
;   [clojure.lang.AFn applyToHelper AFn.java 152]
;   [clojure.lang.AFn applyTo AFn.java 144]
;   [clojure.core$apply invokeStatic core.clj 667]
;   [clojure.core$with_bindings_STAR_ invokeStatic core.clj 1990]
;   [clojure.core$with_bindings_STAR_ doInvoke core.clj 1990]
;   [clojure.lang.RestFn invoke RestFn.java 425]
;   [nrepl.middleware.interruptible_eval$evaluate$fn__1359 invoke interruptible_eval.clj 87]
;   [clojure.main$repl$read_eval_print__9206$fn__9209 invoke main.clj 437]
;   [clojure.main$repl$read_eval_print__9206 invoke main.clj 437]
;   [clojure.main$repl$fn__9215 invoke main.clj 458]
;   [clojure.main$repl invokeStatic main.clj 458]
;   [clojure.main$repl doInvoke main.clj 368]
;   [clojure.lang.RestFn invoke RestFn.java 1523]
;   [nrepl.middleware.interruptible_eval$evaluate invokeStatic interruptible_eval.clj 84]
;   [nrepl.middleware.interruptible_eval$evaluate invoke interruptible_eval.clj 56]
;   [nrepl.middleware.interruptible_eval$interruptible_eval$fn__1392$fn__1396 invoke interruptible_eval.clj 152]
;   [clojure.lang.AFn run AFn.java 22]
;   [nrepl.middleware.session$session_exec$main_loop__1462$fn__1466 invoke session.clj 218]
;   [nrepl.middleware.session$session_exec$main_loop__1462 invoke session.clj 217]
;   [clojure.lang.AFn run AFn.java 22]
;   [java.lang.Thread run Thread.java 1589]]}, :schema {:deployment-id java.lang.String}, :value {:api-version 2024-06-01, :deployment-id nil, :martian.core/body {{:name file, :content #object[java.io.File 0x1420222e D:\Downloads\file_example_MP3_700KB.mp3], :filename example60kf, :mime-type audio/mp3} {:model whisper-1}}, :wkok.openai-clojure.core/options {:impl :azure, :api-key <<REDACTED>>, :api-endpoint https://south-india-sophia.openai.azure.com}}, :error {:deployment-id (not (instance? java.lang.String nil))}}
wkok commented 2 months ago

Try this

(api/create-transcription {:file  (io/file "D:/Downloads/file_example_MP3_700KB.mp3")
                           :model "whisper-1"}
                          {:impl :azure})
HanjoPurebuddha commented 2 months ago

I receive "missing audio file" if I don't pass the additional file infos. Here's my code I'm using:

(api/create-transcription {:file  (io/file "D:/Downloads/file_example_MP3_700KB.mp3")
                               :model "whisper-1"}
                              {:impl :azure ;; Use Azure implementation
                               :api-key (System/getenv "AZURE_OPENAI_API_KEY_SOUTHINDIA") ;; Fetch the API key from environment variable
                               :api-endpoint (System/getenv "AZURE_OPENAI_API_ENDPOINT_SOUTHINDIA")})

Here's the error:

; #object[java.io.File 0x91471d9 D:\Downloads\file_example_MP3_700KB.mp3]
; An error occurred during transcription:  Interceptor Exception: status: 400
; {:request-time 868, :request {:user-info nil, :as :text, :headers {Accept application/json, api-key <<REDACTED>>, accept-encoding gzip, deflate}, :server-port nil, :url https://south-india-sophia.openai.azure.com/openai/deployments/whisper-1/audio/transcriptions, :http-request #object[jdk.internal.net.http.HttpRequestImpl 0x3a41ada3 https://south-india-sophia.openai.azure.com/openai/deployments/whisper-1/audio/transcriptions?api-version=2024-06-01 POST], :uri /openai/deployments/whisper-1/audio/transcriptions, :server-name south-india-sophia.openai.azure.com, :version :http-1.1, :query-string api-version=2024-06-01, :scheme :https, :request-method :post}, :stage :leave, :interceptor :wkok.openai-clojure.sse/perform-sse-capable-request, :http-client #object[jdk.internal.net.http.HttpClientFacade 0x51d402f7 jdk.internal.net.http.HttpClientImpl@588b9a0e(13)], :type clojure.lang.ExceptionInfo, :headers {x-content-type-options nosniff, content-length 0, x-ms-region South India, strict-transport-security max-age=31536000; includeSubDomains; preload, ms-azureml-model-error-reason Missing audio file, apim-request-id 3023e2e2-7497-4bfd-85aa-f8e6a41439dd, date Fri, 30 Aug 2024 09:48:13 GMT, ms-azureml-model-error-statuscode 400, x-ratelimit-remaining-requests 1}, :status 400, :execution-id #uuid "a9ff5171-8541-4aa9-8e54-b73706505b82", :exception #error {
;  :cause status: 400
;  :data {:uri https://south-india-sophia.openai.azure.com/openai/deployments/whisper-1/audio/transcriptions?api-version=2024-06-01, :status 400, :body , :headers {x-content-type-options nosniff, content-length 0, x-ms-region South India, strict-transport-security max-age=31536000; includeSubDomains; preload, ms-azureml-model-error-reason Missing audio file, apim-request-id 3023e2e2-7497-4bfd-85aa-f8e6a41439dd, date Fri, 30 Aug 2024 09:48:13 GMT, ms-azureml-model-error-statuscode 400, x-ratelimit-remaining-requests 1}, :version :http-1.1, :http-client #object[jdk.internal.net.http.HttpClientFacade 0x51d402f7 jdk.internal.net.http.HttpClientImpl@588b9a0e(13)], :request {:user-info nil, :as :text, :headers {Accept application/json, api-key <<REDACTED>>, accept-encoding gzip, deflate}, :server-port nil, :url https://south-india-sophia.openai.azure.com/openai/deployments/whisper-1/audio/transcriptions, :http-request #object[jdk.internal.net.http.HttpRequestImpl 0x3a41ada3 https://south-india-sophia.openai.azure.com/openai/deployments/whisper-1/audio/transcriptions?api-version=2024-06-01 POST], :uri /openai/deployments/whisper-1/audio/transcriptions, :server-name south-india-sophia.openai.azure.com, :version :http-1.1, :query-string api-version=2024-06-01, :scheme :https, :request-method :post}, :request-time 868}
;  :via
;  [{:type clojure.lang.ExceptionInfo
;    :message status: 400
;    :data {:uri https://south-india-sophia.openai.azure.com/openai/deployments/whisper-1/audio/transcriptions?api-version=2024-06-01, :status 400, :body , :headers {x-content-type-options nosniff, content-length 0, x-ms-region South India, strict-transport-security max-age=31536000; includeSubDomains; preload, ms-azureml-model-error-reason Missing audio file, apim-request-id 3023e2e2-7497-4bfd-85aa-f8e6a41439dd, date Fri, 30 Aug 2024 09:48:13 GMT, ms-azureml-model-error-statuscode 400, x-ratelimit-remaining-requests 1}, :version :http-1.1, :http-client #object[jdk.internal.net.http.HttpClientFacade 0x51d402f7 jdk.internal.net.http.HttpClientImpl@588b9a0e(13)], :request {:user-info nil, :as :text, :headers {Accept application/json, api-key <<REDACTED>>, accept-encoding gzip, deflate}, :server-port nil, :url https://south-india-sophia.openai.azure.com/openai/deployments/whisper-1/audio/transcriptions, :http-request #object[jdk.internal.net.http.HttpRequestImpl 0x3a41ada3 https://south-india-sophia.openai.azure.com/openai/deployments/whisper-1/audio/transcriptions?api-version=2024-06-01 POST], :uri /openai/deployments/whisper-1/audio/transcriptions, :server-name south-india-sophia.openai.azure.com, :version :http-1.1, :query-string api-version=2024-06-01, :scheme :https, :request-method :post}, :request-time 868}
;    :at [hato.middleware$exceptions_response invokeStatic middleware.clj 149]}]
;  :trace
;  [[hato.middleware$exceptions_response invokeStatic middleware.clj 149]
;   [hato.middleware$exceptions_response invoke middleware.clj 140]
;   [hato.middleware$wrap_exceptions$fn__12992 invoke middleware.clj 158]
;   [hato.middleware$wrap_accept$fn__13100 invoke middleware.clj 315]
;   [hato.middleware$wrap_accept_encoding$fn__13107 invoke middleware.clj 337]
;   [hato.middleware$wrap_multipart$fn__13284 invoke middleware.clj 707]
;   [hato.middleware$wrap_content_type$fn__13094 invoke middleware.clj 298]
;   [hato.middleware$wrap_form_params$fn__13205 invoke middleware.clj 536]
;   [hato.middleware$wrap_nested_params$fn__13278 invoke middleware.clj 687]
;   [hato.middleware$wrap_method$fn__13163 invoke middleware.clj 470]
;   [hato.client$request invokeStatic client.clj 327]
;   [hato.client$request doInvoke client.clj 321]
;   [clojure.lang.RestFn invoke RestFn.java 410]
;   [wkok.openai_clojure.sse$fn__23804 invokeStatic sse.clj 159]
;   [wkok.openai_clojure.sse$fn__23804 invoke sse.clj 133]
;   [tripod.context$try_f invokeStatic context.cljc 32]
;   [tripod.context$try_f invoke context.cljc 22]
;   [tripod.context$leave_all_with_binding invokeStatic context.cljc 136]
;   [tripod.context$leave_all_with_binding invoke context.cljc 120]
;   [tripod.context$leave_all$fn__8322 invoke context.cljc 149]
;   [clojure.lang.AFn applyToHelper AFn.java 152]
;   [clojure.lang.AFn applyTo AFn.java 144]
;   [clojure.core$apply invokeStatic core.clj 667]
;   [clojure.core$with_bindings_STAR_ invokeStatic core.clj 1990]
;   [clojure.core$with_bindings_STAR_ doInvoke core.clj 1990]
;   [clojure.lang.RestFn invoke RestFn.java 425]
;   [tripod.context$leave_all invokeStatic context.cljc 148]
;   [tripod.context$leave_all invoke context.cljc 142]
;   [tripod.context$execute invokeStatic context.cljc 199]
;   [tripod.context$execute invoke context.cljc 198]
;   [martian.core$response_for invokeStatic core.cljc 120]
;   [martian.core$response_for invoke core.cljc 114]
;   [wkok.openai_clojure.core$response_for invokeStatic core.clj 36]
;   [wkok.openai_clojure.core$response_for invoke core.clj 16]
;   [wkok.openai_clojure.api$create_transcription invokeStatic api.clj 197]
;   [wkok.openai_clojure.api$create_transcription invoke api.clj 181]
;   [openai_transcribe_clojure.core$handle_transcription_error invokeStatic core.clj 121]
;   [openai_transcribe_clojure.core$handle_transcription_error invoke core.clj 115]
;   [openai_transcribe_clojure.core$eval26203 invokeStatic NO_SOURCE_FILE 158]
;   [openai_transcribe_clojure.core$eval26203 invoke NO_SOURCE_FILE 158]
;   [clojure.lang.Compiler eval Compiler.java 7201]
;   [clojure.lang.Compiler eval Compiler.java 7156]
;   [clojure.core$eval invokeStatic core.clj 3216]
;   [clojure.core$eval invoke core.clj 3212]
;   [nrepl.middleware.interruptible_eval$evaluate$fn__1359$fn__1360 invoke interruptible_eval.clj 87]
;   [clojure.lang.AFn applyToHelper AFn.java 152]
;   [clojure.lang.AFn applyTo AFn.java 144]
;   [clojure.core$apply invokeStatic core.clj 667]
;   [clojure.core$with_bindings_STAR_ invokeStatic core.clj 1990]
;   [clojure.core$with_bindings_STAR_ doInvoke core.clj 1990]
;   [clojure.lang.RestFn invoke RestFn.java 425]
;   [nrepl.middleware.interruptible_eval$evaluate$fn__1359 invoke interruptible_eval.clj 87]
;   [clojure.main$repl$read_eval_print__9206$fn__9209 invoke main.clj 437]
;   [clojure.main$repl$read_eval_print__9206 invoke main.clj 437]
;   [clojure.main$repl$fn__9215 invoke main.clj 458]
;   [clojure.main$repl invokeStatic main.clj 458]
;   [clojure.main$repl doInvoke main.clj 368]
;   [clojure.lang.RestFn invoke RestFn.java 1523]
;   [nrepl.middleware.interruptible_eval$evaluate invokeStatic interruptible_eval.clj 84]
;   [nrepl.middleware.interruptible_eval$evaluate invoke interruptible_eval.clj 56]
;   [nrepl.middleware.interruptible_eval$interruptible_eval$fn__1392$fn__1396 invoke interruptible_eval.clj 152]
;   [clojure.lang.AFn run AFn.java 22]
;   [nrepl.middleware.session$session_exec$main_loop__1462$fn__1466 invoke session.clj 218]
;   [nrepl.middleware.session$session_exec$main_loop__1462 invoke session.clj 217]
;   [clojure.lang.AFn run AFn.java 22]
;   [java.lang.Thread run Thread.java 1589]]}, :uri https://south-india-sophia.openai.azure.com/openai/deployments/whisper-1/audio/transcriptions?api-version=2024-06-01, :version :http-1.1, :body }
; Status code: 400
; Headers: {x-content-type-options nosniff, content-length 0, x-ms-region South India, strict-transport-security max-age=31536000; includeSubDomains; preload, ms-azureml-model-error-reason Missing audio file, apim-request-id 3023e2e2-7497-4bfd-85aa-f8e6a41439dd, date Fri, 30 Aug 2024 09:48:13 GMT, ms-azureml-model-error-statuscode 400, x-ratelimit-remaining-requests 1}
; Response body: 
; Finished processing transcription request.
wkok commented 2 months ago

In their spec there's a discrepancy between the file type and the description - how can the type be string but it's an audio file object that need to be provided? :man_shrugging:

wkok commented 2 months ago

I've made a few changes to the azure api to hopefully better support multipart file uploads for transcriptions and translations.

The changes are in this branch

If you update your deps.edn to the following, then you could maybe try it out and let me know?

net.clojars.wkok/openai-clojure {:git/url "https://github.com/wkok/openai-clojure.git"
                                 :sha     "f4f1fe1e1f0a4e2beb4fa3c4cbea6d0812adc4d4"}

usage is the same as in the code example in your last attempt above.

HanjoPurebuddha commented 1 month ago

This works perfectly! Thank you.

Sorry I couldn't get back to this sooner - I was on vacation.