No serializer found for class java.io.ByteArrayInputStream on OpenAiAudioTranscriptionModel.call

[clivelewis]

Bug description Exception is thrown when attempting to send a request to OpenAI transcription service. Please note that the same logic was working approximately 1-2 months ago.

Environment Kotlin + Java 21 Spring Boot 3.3.1 Spring AI 1.0.0-M1 (Also tried on 1.0.0-SNAPSHOT)

Steps to reproduce Simply execute .call method from OpenAiAudioTranscriptionModel with a ByteArrayResource set in the Prompt. Here's my method:

fun transcribeAudio(audio: ByteArray): String {

        val resource = ByteArrayResource(audio)
        val options = OpenAiAudioTranscriptionOptions.builder()
            .withResponseFormat(OpenAiAudioApi.TranscriptResponseFormat.TEXT)
            .build()
        val prompt = AudioTranscriptionPrompt(resource, options)
        val response = client.call(prompt)
        val output = response.result.output
        log.info { "[Transcription] Result - ${response.result}" }
        return output
    }

Expected behavior No exception. Executed method returns transcribed audio as text.

Some of the things that I tried to fix the issue

• Removed all custom beans that could affect the serialization - ObjectMapper and MappingJackson2HttpMessageConverter
• Custom ObjectMapper bean with disabled SerializationFeature.FAIL_ON_EMPTY_BEANS

  @Bean
  fun objectMapper(): ObjectMapper {
      return ObjectMapper().apply {
          disable(SerializationFeature.FAIL_ON_EMPTY_BEANS);
      }
  }

• Custom MappingJackson2HttpMessageConverter bean

  @Bean
  fun mappingJackson2HttpMessageConverter(objectMapper: ObjectMapper): MappingJackson2HttpMessageConverter {
      objectMapper.configure(SerializationFeature.FAIL_ON_EMPTY_BEANS, false)
      val converter = MappingJackson2HttpMessageConverter(objectMapper)
      return converter
  }

Stack trace

org.springframework.http.converter.HttpMessageConversionException: Type definition error: [simple type, class java.io.ByteArrayInputStream]
    at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:489)
    at org.springframework.http.converter.AbstractGenericHttpMessageConverter$1.writeTo(AbstractGenericHttpMessageConverter.java:94)
    at org.springframework.http.client.HttpComponentsClientHttpRequest$BodyEntity.writeTo(HttpComponentsClientHttpRequest.java:155)
    at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:253)
    at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:141)
    at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:218)
    at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:717)
    at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:216)
    at org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainClientExec.java:116)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:188)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
    at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:87)
    at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at org.apache.hc.client5.http.classic.HttpClient.executeOpen(HttpClient.java:183)
    at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:99)
    at org.springframework.http.client.AbstractStreamingClientHttpRequest.executeInternal(AbstractStreamingClientHttpRequest.java:70)
    at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66)
    at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.exchangeInternal(DefaultRestClient.java:492)
    at org.springframework.web.client.DefaultRestClient$DefaultRequestBodyUriSpec.retrieve(DefaultRestClient.java:460)
    at org.springframework.ai.openai.api.OpenAiAudioApi.createTranscription(OpenAiAudioApi.java:670)
    at org.springframework.ai.openai.OpenAiAudioTranscriptionModel.lambda$call$0(OpenAiAudioTranscriptionModel.java:153)
    at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:344)
    at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:217)
    at org.springframework.ai.openai.OpenAiAudioTranscriptionModel.call(OpenAiAudioTranscriptionModel.java:124)
    at io.github.clivelewis.assistant.chat.impl.openai.OpenAiTranscriptionService.transcribeAudio(OpenAiTranscriptionService.kt:24)
    at io.github.clivelewis.assistant.telegram.commands.ConversationCommand.transcribeVoiceMessage(ConversationCommand.kt:187)
    at io.github.clivelewis.assistant.telegram.commands.ConversationCommand.access$transcribeVoiceMessage(ConversationCommand.kt:28)
    at io.github.clivelewis.assistant.telegram.commands.ConversationCommand$transcribeVoiceMessage$1.invokeSuspend(ConversationCommand.kt)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
    at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:111)
    at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:99)
    at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:811)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:715)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:702)
Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class java.io.ByteArrayInputStream and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS) (through reference chain: org.springframework.util.LinkedMultiValueMap["file"]->java.util.ArrayList[0]->org.springframework.ai.openai.api.OpenAiAudioApi$1["inputStream"])
    at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:77)
    at com.fasterxml.jackson.databind.SerializerProvider.reportBadDefinition(SerializerProvider.java:1330)
    at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:414)
    at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.failForEmpty(UnknownSerializer.java:53)
    at com.fasterxml.jackson.databind.ser.impl.UnknownSerializer.serialize(UnknownSerializer.java:30)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:732)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:770)
    at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:183)
    at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
    at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
    at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
    at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:808)
    at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeWithoutTypeInfo(MapSerializer.java:764)
    at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:720)
    at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:35)
    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:502)
    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:422)
    at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1570)
    at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1061)
    at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:483)
    ... 45 common frames omitted

[clivelewis]

I cloned the project, checked out to tags/v1.0.0-M1 and ran tests from OpenAiTranscriptionModelIT.java

All tests succeeded, so it's issue on my side. Either because of Kotlin, or because of my code...

[clivelewis]

I played around with different custom Bean definitions and somehow defining OpenAiAudioTranscriptionModel solved the issue for me...

@Bean
fun openAiAudioApi(@Value("\${spring.ai.openai.api-key}") apiKey: String) = OpenAiAudioApi(apiKey)

@Bean
fun openAiAudioTranscriptionModel(openAiAudioApi: OpenAiAudioApi) = OpenAiAudioTranscriptionModel(openAiAudioApi)

I'm closing the issue even though I have no idea why this helped. Will appreciate if somehow could explain.