search

spring-cloud / stream-applications

Functions and Spring Cloud Stream Applications for data driven microservices
https://spring.io/projects/spring-cloud-stream-applications
247 stars 106 forks source link

s3 source not working with aws sts assume role #522

Closed songhohoon closed 4 months ago

songhohoon commented 5 months ago

I'm trying to deploy a stream with s3 source. my environment is EKS(aws kubernetes) and I'm using IRSA for aws sts assume role.

I tried several methods

does spring-cloud/stream-application support for EKS IRSA (aws sts)??

artembilan commented 5 months ago

Any chances to see some logs and/or error to be sure that there is really something we are missing? Or if to be more straight: any proofs that it doesn't work? Thanks

songhohoon commented 5 months ago

@artembilan thank you for reply. this is full application startup log

(⎈ |plat-dev:scdf) hohoon.song  ~/Workspace/app  k logs -f s3-test-s3-source-main-v4-d7957dfff-xwc4f
Setting Active Processor Count to 4
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx391582K -XX:MaxMetaspaceSize=144993K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 1G, Thread Count: 250, Loaded Class Count: 23185, Headroom: 0%)
Enabling Java Native Memory Tracking
Using readonly truststore: /tmp/truststore
Adding 137 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
NOTE: Picked up JDK_JAVA_OPTIONS: -Dsun.jnu.encoding
Picked up JAVA_TOOL_OPTIONS: -Duser.timezone=Asia/Seoul -Djava.security.properties=/layers/paketo-buildpacks_bellsoft-liberica/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=4 -XX:MaxDirectMemorySize=10M -Xmx391582K -XX:MaxMetaspaceSize=144993K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Djavax.net.ssl.trustStore=/tmp/truststore -Dorg.springframework.cloud.bindings.boot.enable=true
Standard Commons Logging discovery in action with spring-jcl: please remove commons-logging.jar from classpath in order to avoid potential conflicts

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.1.8)

2024-03-06T08:36:50.294+09:00  INFO 1 --- [           main] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at : http://localhost:8888
2024-03-06T08:36:51.389+09:00  INFO 1 --- [           main] c.c.c.ConfigServicePropertySourceLocator : Exception on Url - http://localhost:8888:org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://localhost:8888/s3-source/default": Connection refused. Will be trying the next url if available
2024-03-06T08:36:51.390+09:00  WARN 1 --- [           main] c.c.c.ConfigServicePropertySourceLocator : Could not locate PropertySource: I/O error on GET request for "http://localhost:8888/s3-source/default": Connection refused
2024-03-06T08:36:51.404+09:00  INFO 1 --- [           main] o.s.c.s.a.s.s.k.S3SourceKafkaApplication : No active profile set, falling back to 1 default profile: "default"
2024-03-06T08:36:59.700+09:00  INFO 1 --- [           main] faultConfiguringBeanFactoryPostProcessor : No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
2024-03-06T08:36:59.709+09:00  INFO 1 --- [           main] faultConfiguringBeanFactoryPostProcessor : No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
2024-03-06T08:37:00.622+09:00  INFO 1 --- [           main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=82170e26-73de-3b70-a039-98b3ab42cc64
2024-03-06T08:37:01.690+09:00  INFO 1 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.stream.app.postprocessor.ContentTypeHeaderBeanPostProcessorAutoConfiguration' of type [org.springframework.cloud.stream.app.postprocessor.ContentTypeHeaderBeanPostProcessorAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-03-06T08:37:04.002+09:00  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2024-03-06T08:37:04.103+09:00  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2024-03-06T08:37:04.103+09:00  INFO 1 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.18]
2024-03-06T08:37:04.723+09:00  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2024-03-06T08:37:04.725+09:00  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 13118 ms
Standard Commons Logging discovery in action with spring-jcl: please remove commons-logging.jar from classpath in order to avoid potential conflicts
2024-03-06T08:37:20.176+09:00  INFO 1 --- [tor-tcp-epoll-2] i.m.p.rsocket.PrometheusRSocketClient    : Connected to RSocket Proxy!
2024-03-06T08:37:23.186+09:00  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 3 endpoint(s) beneath base path '/actuator'
2024-03-06T08:37:23.593+09:00  INFO 1 --- [           main] o.s.s.web.DefaultSecurityFilterChain     : Will secure any request with [org.springframework.security.web.session.DisableEncodeUrlFilter@2bb118ae, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@5e3a5f0c, org.springframework.security.web.context.SecurityContextHolderFilter@31c31fa6, org.springframework.security.web.header.HeaderWriterFilter@1dbb9a4a, org.springframework.security.web.csrf.CsrfFilter@20040c6e, org.springframework.security.web.authentication.logout.LogoutFilter@73d7e7aa, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@60655642, org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter@4cbd17b3, org.springframework.security.web.authentication.ui.DefaultLogoutPageGeneratingFilter@5afaae7e, org.springframework.security.web.authentication.www.BasicAuthenticationFilter@1e9d6c78, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@3d5e3f58, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@29b2a94c, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@7364073b, org.springframework.security.web.access.ExceptionTranslationFilter@78a0d664, org.springframework.security.web.access.intercept.AuthorizationFilter@2ef3efcc]
2024-03-06T08:37:24.598+09:00  INFO 1 --- [           main] ctiveUserDetailsServiceAutoConfiguration :

Using generated security password: 9e9e5f88-e71e-4b92-ac81-658643c32b50

2024-03-06T08:37:24.607+09:00  INFO 1 --- [tor-tcp-epoll-2] i.m.p.rsocket.PrometheusRSocketClient    : Connected to RSocket Proxy!
2024-03-06T08:37:24.886+09:00  WARN 1 --- [           main] ocalVariableTableParameterNameDiscoverer : Using deprecated '-debug' fallback for parameter name resolution. Compile the affected code with '-parameters' instead or avoid its introspection: org.springframework.cloud.fn.filter.FilterFunctionConfiguration
2024-03-06T08:37:24.894+09:00  WARN 1 --- [           main] ocalVariableTableParameterNameDiscoverer : Using deprecated '-debug' fallback for parameter name resolution. Compile the affected code with '-parameters' instead or avoid its introspection: org.springframework.cloud.fn.spel.SpelFunctionConfiguration
2024-03-06T08:37:25.090+09:00  WARN 1 --- [           main] ocalVariableTableParameterNameDiscoverer : Using deprecated '-debug' fallback for parameter name resolution. Compile the affected code with '-parameters' instead or avoid its introspection: org.springframework.cloud.fn.task.launch.request.TaskLaunchRequestFunctionConfiguration
2024-03-06T08:37:25.891+09:00  INFO 1 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2024-03-06T08:37:25.892+09:00  INFO 1 --- [           main] o.s.i.channel.PublishSubscribeChannel    : Channel 'application-1.errorChannel' has 1 subscriber(s).
2024-03-06T08:37:25.893+09:00  INFO 1 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : started bean '_org.springframework.integration.errorLogger'
2024-03-06T08:37:25.994+09:00  INFO 1 --- [           main] o.s.i.endpoint.ReactiveStreamsConsumer   : started bean 's3Supplier_integrationflow.org.springframework.integration.config.ConsumerEndpointFactoryBean#0'
2024-03-06T08:37:25.996+09:00  INFO 1 --- [           main] o.s.c.s.binder.DefaultBinderFactory      : Creating binder: scdf
2024-03-06T08:37:25.996+09:00  INFO 1 --- [           main] o.s.c.s.binder.DefaultBinderFactory      : Constructing binder child context for scdf
2024-03-06T08:37:28.093+09:00  INFO 1 --- [           main] o.s.c.s.binder.DefaultBinderFactory      : Caching the binder: scdf
2024-03-06T08:37:28.387+09:00  INFO 1 --- [           main] o.s.c.s.b.k.p.KafkaTopicProvisioner      : Using kafka topic for outbound: s3-test.s3-source-main
2024-03-06T08:37:28.391+09:00  INFO 1 --- [           main] o.a.k.clients.admin.AdminClientConfig    : AdminClientConfig values:
    auto.include.jmx.reporter = true
    bootstrap.servers = [b-1.scdfdev.p11u7d.c2.kafka.ap-northeast-2.amazonaws.com:9092, b-2.scdfdev.p11u7d.c2.kafka.ap-northeast-2.amazonaws.com:9092]
    client.dns.lookup = use_all_dns_ips
    client.id =
    connections.max.idle.ms = 300000
    default.api.timeout.ms = 60000
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 30000
    retries = 2147483647
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.connect.timeout.ms = null
    sasl.login.read.timeout.ms = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.login.retry.backoff.max.ms = 10000
    sasl.login.retry.backoff.ms = 100
    sasl.mechanism = GSSAPI
    sasl.oauthbearer.clock.skew.seconds = 30
    sasl.oauthbearer.expected.audience = null
    sasl.oauthbearer.expected.issuer = null
    sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
    sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
    sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
    sasl.oauthbearer.jwks.endpoint.url = null
    sasl.oauthbearer.scope.claim.name = scope
    sasl.oauthbearer.sub.claim.name = sub
    sasl.oauthbearer.token.endpoint.url = null
    security.protocol = PLAINTEXT
    security.providers = null
    send.buffer.bytes = 131072
    socket.connection.setup.timeout.max.ms = 30000
    socket.connection.setup.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
    ssl.endpoint.identification.algorithm = https
    ssl.engine.factory.class = null
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.certificate.chain = null
    ssl.keystore.key = null
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLSv1.3
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.certificates = null
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS

2024-03-06T08:37:29.089+09:00  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 3.4.1
2024-03-06T08:37:29.090+09:00  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: 8a516edc2755df89
2024-03-06T08:37:29.090+09:00  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1709681849088
2024-03-06T08:37:30.093+09:00  INFO 1 --- [| adminclient-1] o.a.kafka.common.utils.AppInfoParser     : App info kafka.admin.client for adminclient-1 unregistered
2024-03-06T08:37:30.187+09:00  INFO 1 --- [| adminclient-1] o.apache.kafka.common.metrics.Metrics    : Metrics scheduler closed
2024-03-06T08:37:30.187+09:00  INFO 1 --- [| adminclient-1] o.apache.kafka.common.metrics.Metrics    : Closing reporter org.apache.kafka.common.metrics.JmxReporter
2024-03-06T08:37:30.187+09:00  INFO 1 --- [| adminclient-1] o.apache.kafka.common.metrics.Metrics    : Metrics reporters closed
2024-03-06T08:37:30.291+09:00  INFO 1 --- [           main] o.a.k.clients.producer.ProducerConfig    : Idempotence will be disabled because acks is set to 1, not set to 'all'.
2024-03-06T08:37:30.292+09:00  INFO 1 --- [           main] o.a.k.clients.producer.ProducerConfig    : ProducerConfig values:
    acks = 1
    auto.include.jmx.reporter = true
    batch.size = 16384
    bootstrap.servers = [b-1.scdfdev.p11u7d.c2.kafka.ap-northeast-2.amazonaws.com:9092, b-2.scdfdev.p11u7d.c2.kafka.ap-northeast-2.amazonaws.com:9092]
    buffer.memory = 33554432
    client.dns.lookup = use_all_dns_ips
    client.id = producer-1
    compression.type = none
    connections.max.idle.ms = 540000
    delivery.timeout.ms = 120000
    enable.idempotence = false
    interceptor.classes = []
    key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
    linger.ms = 0
    max.block.ms = 60000
    max.in.flight.requests.per.connection = 5
    max.request.size = 1048576
    metadata.max.age.ms = 300000
    metadata.max.idle.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    partitioner.adaptive.partitioning.enable = true
    partitioner.availability.timeout.ms = 0
    partitioner.class = null
    partitioner.ignore.keys = false
    receive.buffer.bytes = 32768
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 30000
    retries = 2147483647
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.connect.timeout.ms = null
    sasl.login.read.timeout.ms = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.login.retry.backoff.max.ms = 10000
    sasl.login.retry.backoff.ms = 100
    sasl.mechanism = GSSAPI
    sasl.oauthbearer.clock.skew.seconds = 30
    sasl.oauthbearer.expected.audience = null
    sasl.oauthbearer.expected.issuer = null
    sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
    sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
    sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
    sasl.oauthbearer.jwks.endpoint.url = null
    sasl.oauthbearer.scope.claim.name = scope
    sasl.oauthbearer.sub.claim.name = sub
    sasl.oauthbearer.token.endpoint.url = null
    security.protocol = PLAINTEXT
    security.providers = null
    send.buffer.bytes = 131072
    socket.connection.setup.timeout.max.ms = 30000
    socket.connection.setup.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
    ssl.endpoint.identification.algorithm = https
    ssl.engine.factory.class = null
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.certificate.chain = null
    ssl.keystore.key = null
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLSv1.3
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.certificates = null
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
    transaction.timeout.ms = 60000
    transactional.id = null
    value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer

2024-03-06T08:37:30.495+09:00  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka version: 3.4.1
2024-03-06T08:37:30.495+09:00  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka commitId: 8a516edc2755df89
2024-03-06T08:37:30.496+09:00  INFO 1 --- [           main] o.a.kafka.common.utils.AppInfoParser     : Kafka startTimeMs: 1709681850495
2024-03-06T08:37:30.587+09:00  INFO 1 --- [ad | producer-1] org.apache.kafka.clients.Metadata        : [Producer clientId=producer-1] Cluster ID: XdPay31LRMenDOXmYjQ4Qg
2024-03-06T08:37:30.798+09:00  INFO 1 --- [           main] o.s.c.s.m.DirectWithAttributesChannel    : Channel 'application-1.output' has 1 subscriber(s).
2024-03-06T08:37:30.800+09:00  INFO 1 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : Adding {transformer} as a subscriber to the 's3SupplierFlow.channel#1' channel
2024-03-06T08:37:30.800+09:00  INFO 1 --- [           main] o.s.integration.channel.DirectChannel    : Channel 'application-1.s3SupplierFlow.channel#1' has 1 subscriber(s).
2024-03-06T08:37:30.800+09:00  INFO 1 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : started bean 's3SupplierFlow.org.springframework.integration.config.ConsumerEndpointFactoryBean#1'; defined in: 'org.springframework.cloud.fn.supplier.s3.AwsS3SupplierConfiguration$SynchronizingConfiguration'; from source: 'bean method s3SupplierFlow'
2024-03-06T08:37:30.892+09:00  INFO 1 --- [           main] o.s.i.endpoint.ReactiveStreamsConsumer   : started bean 's3SupplierFlow.org.springframework.integration.config.ConsumerEndpointFactoryBean#0'; defined in: 'org.springframework.cloud.fn.supplier.s3.AwsS3SupplierConfiguration$SynchronizingConfiguration'; from source: 'bean method s3SupplierFlow'
2024-03-06T08:37:30.993+09:00  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2024-03-06T08:37:31.387+09:00  INFO 1 --- [           main] o.s.c.s.a.s.s.k.S3SourceKafkaApplication : Started S3SourceKafkaApplication in 51.494 seconds (process running for 54.193)
2024-03-06T08:37:31.896+09:00  INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2024-03-06T08:37:31.896+09:00  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2024-03-06T08:37:31.899+09:00  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms
2024-03-06T08:37:34.200+09:00 DEBUG 1 --- [oundedElastic-3] i.awspring.cloud.s3.CrossRegionS3Client  : Exception when requesting S3 for bucket: bucket: details=[PermanentRedirect, The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.], httpcode=301
2024-03-06T08:37:34.200+09:00 DEBUG 1 --- [oundedElastic-3] i.awspring.cloud.s3.CrossRegionS3Client  : Region for bucket was discovered to be us-east-1 and is being cached
2024-03-06T08:37:34.201+09:00 DEBUG 1 --- [oundedElastic-3] i.awspring.cloud.s3.CrossRegionS3Client  : Creating new S3 client for region: us-east-1
2024-03-06T08:37:34.613+09:00  INFO 1 --- [tor-tcp-epoll-2] i.m.p.rsocket.PrometheusRSocketClient    : Connected to RSocket Proxy!
2024-03-06T08:37:35.187+09:00 ERROR 1 --- [oundedElastic-3] o.s.i.util.IntegrationReactiveUtils      : Error from Flux for : org.springframework.integration.aws.inbound.S3InboundFileSynchronizingMessageSource@67e46c15

org.springframework.messaging.MessagingException: Problem occurred while synchronizing 'bucket' to local directory
    at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:348) ~[spring-integration-file-6.1.5.jar:6.1.5]
    at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.doReceive(AbstractInboundFileSynchronizingMessageSource.java:267) ~[spring-integration-file-6.1.5.jar:6.1.5]
    at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizingMessageSource.doReceive(AbstractInboundFileSynchronizingMessageSource.java:69) ~[spring-integration-file-6.1.5.jar:6.1.5]
    at org.springframework.integration.endpoint.AbstractFetchLimitingMessageSource.doReceive(AbstractFetchLimitingMessageSource.java:47) ~[spring-integration-core-6.1.5.jar:6.1.5]
    at org.springframework.integration.endpoint.AbstractMessageSource.receive(AbstractMessageSource.java:142) ~[spring-integration-core-6.1.5.jar:6.1.5]
    at org.springframework.integration.util.IntegrationReactiveUtils.lambda$messageSourceToFlux$0(IntegrationReactiveUtils.java:82) ~[spring-integration-core-6.1.5.jar:6.1.5]
    at reactor.core.publisher.MonoCreate$DefaultMonoSink.onRequest(MonoCreate.java:221) ~[reactor-core-3.5.14.jar:3.5.14]
    at org.springframework.integration.util.IntegrationReactiveUtils.lambda$messageSourceToFlux$1(IntegrationReactiveUtils.java:82) ~[spring-integration-core-6.1.5.jar:6.1.5]
    at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58) ~[reactor-core-3.5.14.jar:3.5.14]
    at reactor.core.publisher.Mono.subscribe(Mono.java:4495) ~[reactor-core-3.5.14.jar:3.5.14]
    at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:126) ~[reactor-core-3.5.14.jar:3.5.14]
    at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) ~[reactor-core-3.5.14.jar:3.5.14]
    at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) ~[reactor-core-3.5.14.jar:3.5.14]
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[na:na]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
    at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
Caused by: org.springframework.messaging.MessagingException: Failed to execute on session
    at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:461) ~[spring-integration-file-6.1.5.jar:6.1.5]
    at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.synchronizeToLocalDirectory(AbstractInboundFileSynchronizer.java:341) ~[spring-integration-file-6.1.5.jar:6.1.5]
    ... 17 common frames omitted
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: YE5HSS3K5NHN6WYV, Extended Request ID: YhyC3MOdyFxN0frGvEQKEI/C1lkR6xa469I4g89nZEbsWIlyE0uGHwu9Cb7qiQ1PYcWHo8RSTsU=)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156) ~[aws-xml-protocol-2.20.63.jar:na]
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108) ~[aws-xml-protocol-2.20.63.jar:na]
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85) ~[aws-xml-protocol-2.20.63.jar:na]
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43) ~[aws-xml-protocol-2.20.63.jar:na]
    at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:95) ~[aws-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:270) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[sdk-core-2.20.63.jar:na]
    at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) ~[aws-core-2.20.63.jar:na]
    at software.amazon.awssdk.services.s3.DefaultS3Client.listObjects(DefaultS3Client.java:6448) ~[s3-2.20.63.jar:na]
    at io.awspring.cloud.s3.crossregion.CrossRegionS3Client.lambda$listObjects$4(CrossRegionS3Client.java:157) ~[spring-cloud-aws-s3-cross-region-client-3.0.2.jar:3.0.2]
    at io.awspring.cloud.s3.crossregion.CrossRegionS3Client.executeInBucketRegion(CrossRegionS3Client.java:126) ~[spring-cloud-aws-s3-cross-region-client-3.0.2.jar:3.0.2]
    at io.awspring.cloud.s3.crossregion.CrossRegionS3Client.listObjects(CrossRegionS3Client.java:157) ~[spring-cloud-aws-s3-cross-region-client-3.0.2.jar:3.0.2]
    at org.springframework.integration.aws.support.S3Session.list(S3Session.java:84) ~[spring-integration-aws-3.0.2.jar:na]
    at org.springframework.integration.aws.support.S3Session.list(S3Session.java:52) ~[spring-integration-aws-3.0.2.jar:na]
    at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.transferFilesFromRemoteToLocal(AbstractInboundFileSynchronizer.java:356) ~[spring-integration-file-6.1.5.jar:6.1.5]
    at org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.lambda$synchronizeToLocalDirectory$0(AbstractInboundFileSynchronizer.java:342) ~[spring-integration-file-6.1.5.jar:6.1.5]
    at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:452) ~[spring-integration-file-6.1.5.jar:6.1.5]
    ... 18 common frames omitted

and I run into pod and there is no dependency to sts

$ pwd
/workspace/BOOT-INF
$ ls lib
HdrHistogram-2.1.12.jar                kotlin-stdlib-jdk7-1.8.22.jar                       spring-beans-6.0.16.jar
LatencyUtils-2.0.3.jar                 kotlin-stdlib-jdk8-1.8.22.jar                       spring-boot-3.1.8.jar
accessors-smart-2.4.11.jar             lang-tag-1.7.jar                            spring-boot-actuator-3.1.8.jar
annotations-13.0.jar                   listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar  spring-boot-actuator-autoconfigure-3.1.8.jar
annotations-2.20.63.jar                log4j-api-2.20.0.jar                        spring-boot-autoconfigure-3.1.8.jar
apache-client-2.20.63.jar              log4j-to-slf4j-2.20.0.jar                       spring-boot-jarmode-layertools-3.1.8.jar
arns-2.20.63.jar                   logback-classic-1.4.14.jar                      spring-cloud-aws-autoconfigure-3.0.2.jar
asm-9.3.jar                    logback-core-1.4.14.jar                         spring-cloud-aws-core-3.0.2.jar
aspectjweaver-1.9.21.jar               lz4-java-1.8.0.jar                          spring-cloud-aws-s3-3.0.2.jar
auth-2.20.63.jar                   metadata-store-common-5.0.0-SNAPSHOT.jar                spring-cloud-aws-s3-cross-region-client-3.0.2.jar
aws-core-2.20.63.jar                   metrics-spi-2.20.63.jar                         spring-cloud-aws-starter-3.0.2.jar
aws-crt-0.21.14.jar                micrometer-commons-1.11.8.jar                       spring-cloud-aws-starter-s3-3.0.2.jar
aws-crt-client-2.20.63.jar             micrometer-core-1.11.8.jar                      spring-cloud-bindings-1.13.0.jar
aws-query-protocol-2.20.63.jar             micrometer-observation-1.11.8.jar                   spring-cloud-commons-4.0.4.jar
aws-s3-common-5.0.0-SNAPSHOT.jar           micrometer-registry-influx-1.11.8.jar                   spring-cloud-config-client-4.0.4.jar
aws-xml-protocol-2.20.63.jar               micrometer-registry-prometheus-1.11.8.jar               spring-cloud-context-4.0.4.jar
bcpkix-jdk18on-1.73.jar                micrometer-registry-wavefront-1.11.8.jar                spring-cloud-function-context-4.0.6.jar
bcprov-jdk18on-1.73.jar                netty-buffer-4.1.105.Final.jar                      spring-cloud-function-core-4.0.6.jar
bcutil-jdk18on-1.73.jar                netty-codec-4.1.105.Final.jar                       spring-cloud-services-config-client-autoconfigure-3.2.0.RELEASE.jar
checker-qual-3.12.0.jar                netty-codec-dns-4.1.105.Final.jar                   spring-cloud-services-starter-config-client-3.2.0.RELEASE.jar
classmate-1.5.1.jar                netty-codec-http-4.1.105.Final.jar                  spring-cloud-starter-4.0.4.jar
commons-codec-1.15.jar                 netty-codec-http2-4.1.105.Final.jar                 spring-cloud-starter-bootstrap-4.0.4.jar
commons-io-2.11.0.jar                  netty-codec-socks-4.1.105.Final.jar                 spring-cloud-starter-config-4.0.4.jar
commons-logging-1.2.jar                netty-common-4.1.105.Final.jar                      spring-cloud-stream-4.0.5.jar
config-common-5.0.0-SNAPSHOT.jar           netty-handler-4.1.105.Final.jar                     spring-cloud-stream-binder-kafka-4.0.5.jar
content-type-2.2.jar                   netty-handler-proxy-4.1.105.Final.jar                   spring-cloud-stream-binder-kafka-core-4.0.5.jar
crt-core-2.20.63.jar                   netty-nio-client-2.20.63.jar                    spring-context-6.0.16.jar
endpoints-spi-2.20.63.jar              netty-resolver-4.1.105.Final.jar                    spring-core-6.0.16.jar
error_prone_annotations-2.11.0.jar         netty-resolver-dns-4.1.105.Final.jar                spring-expression-6.0.16.jar
eventstream-1.0.1.jar                  netty-resolver-dns-classes-macos-4.1.105.Final.jar          spring-integration-aws-3.0.2.jar
failureaccess-1.0.1.jar                netty-resolver-dns-native-macos-4.1.105.Final-osx-x86_64.jar    spring-integration-core-6.1.5.jar
file-common-5.0.0-SNAPSHOT.jar             netty-transport-4.1.105.Final.jar                   spring-integration-file-6.1.5.jar
filter-function-5.0.0-SNAPSHOT.jar         netty-transport-classes-epoll-4.1.105.Final.jar             spring-integration-jmx-6.1.5.jar
guava-31.1-jre.jar                 netty-transport-native-epoll-4.1.105.Final-linux-x86_64.jar     spring-integration-kafka-6.1.5.jar
header-enricher-function-5.0.0-SNAPSHOT.jar    netty-transport-native-unix-common-4.1.105.Final.jar        spring-jcl-6.0.16.jar
hibernate-validator-8.0.1.Final.jar        nimbus-jose-jwt-9.24.4.jar                      spring-kafka-3.0.13.jar
http-client-spi-2.20.63.jar            oauth2-oidc-sdk-9.43.3.jar                      spring-messaging-6.0.16.jar
httpclient-4.5.13.jar                  payload-converter-function-5.0.0-SNAPSHOT.jar               spring-retry-2.0.5.jar
httpclient5-5.2.3.jar                  profiles-2.20.63.jar                        spring-security-config-6.1.6.jar
httpcore-4.4.16.jar                prometheus-rsocket-client-1.5.2.jar                 spring-security-core-6.1.6.jar
httpcore5-5.2.4.jar                prometheus-rsocket-proxy-server-1.5.2.jar               spring-security-crypto-6.1.6.jar
httpcore5-h2-5.2.4.jar                 prometheus-rsocket-spring-1.5.2.jar                 spring-security-oauth2-client-6.1.6.jar
j2objc-annotations-1.3.jar             protobuf-java-3.19.6.jar                        spring-security-oauth2-core-6.1.6.jar
jackson-annotations-2.15.3.jar             protocol-core-2.20.63.jar                       spring-security-rsa-1.0.12.RELEASE.jar
jackson-core-2.15.3.jar                reactive-streams-1.0.4.jar                      spring-security-web-6.1.6.jar
jackson-databind-2.15.3.jar            reactor-core-3.5.14.jar                         spring-tx-6.0.16.jar
jackson-datatype-jdk8-2.15.3.jar           reactor-netty-core-1.1.15.jar                       spring-web-6.0.16.jar
jackson-datatype-jsr310-2.15.3.jar         reactor-netty-http-1.1.15.jar                       spring-webflux-6.0.16.jar
jackson-module-parameter-names-2.15.3.jar      regions-2.20.63.jar                         spring-webmvc-6.0.16.jar
jakarta.annotation-api-2.1.1.jar           rsocket-core-1.1.3.jar                          stream-applications-composite-function-support-5.0.0-SNAPSHOT.jar
jakarta.validation-api-3.0.2.jar           rsocket-micrometer-1.1.3.jar                    stream-applications-micrometer-common-5.0.0-SNAPSHOT.jar
java-cfenv-2.2.5.RELEASE.jar               rsocket-transport-netty-1.1.3.jar                   stream-applications-postprocessor-common-5.0.0-SNAPSHOT.jar
java-cfenv-boot-2.4.0.jar              s3-2.20.63.jar                              stream-applications-security-common-5.0.0-SNAPSHOT.jar
java-cfenv-boot-pivotal-scs-2.2.5.RELEASE.jar  s3-supplier-5.0.0-SNAPSHOT.jar                      t-digest-3.2.jar
java-cfenv-jdbc-2.4.0.jar              sdk-core-2.20.63.jar                        task-launch-request-function-5.0.0-SNAPSHOT.jar
jboss-logging-3.5.3.Final.jar              simpleclient-0.16.0.jar                         third-party-jackson-core-2.20.63.jar
jcip-annotations-1.0-1.jar             simpleclient_common-0.16.0.jar                      tomcat-embed-core-10.1.18.jar
json-path-2.8.0.jar                simpleclient_tracer_common-0.16.0.jar                   tomcat-embed-el-10.1.18.jar
json-smart-2.4.11.jar                  simpleclient_tracer_otel-0.16.0.jar                 tomcat-embed-websocket-10.1.18.jar
json-utils-2.20.63.jar                 simpleclient_tracer_otel_agent-0.16.0.jar               typetools-0.6.2.jar
jsr305-3.0.2.jar                   slf4j-api-2.0.11.jar                        utils-2.20.63.jar
jul-to-slf4j-2.0.11.jar                snakeyaml-1.33.jar                          wavefront-sdk-java-3.1.0.jar
kafka-clients-3.4.1.jar                snappy-java-1.1.10.1.jar                        zstd-jni-1.5.2-1.jar
kotlin-stdlib-1.8.22.jar               spel-function-5.0.0-SNAPSHOT.jar
kotlin-stdlib-common-1.8.22.jar            spring-aop-6.0.16.jar

I read about dependency in this reference https://docs.awspring.io/spring-cloud-aws/docs/3.0.0/reference/html/index.html#stswebidentitytokenfilecredentialsprovider

artembilan commented 5 months ago

I see. We have an software.amazon.awssdk:sts dependency transitively in case of Kinesis Binder, but you use Kafka.

Any chances that you can patch an image adding that dependency as a workaround?

We are going to fix this in the respective function for the next version: https://github.com/spring-cloud/spring-functions-catalog/issues/41

artembilan commented 5 months ago

Here is the doc how to patch stream applications: https://docs.spring.io/stream-applications/docs/current/reference/html/index.html#

songhohoon commented 5 months ago

thank you for reply. I'll try this.

onobc commented 4 months ago

Closed in M1 release.