Open gvart opened 1 year ago
eduanb
commented
1 year ago For now, this works in Spring Boot 3, with 3.0.0-RC1;
@Configuration
@ImportRuntimeHints(AwsRuntimeHints::class)
class AwsRuntimeHintsConfig
class AwsRuntimeHints : RuntimeHintsRegistrar {
override fun registerHints(hints: RuntimeHints, classLoader: ClassLoader?) {
hints.resources().registerPattern("io/awspring/cloud/core/SpringCloudClientConfiguration.properties")
}
}
maciejwalkowiak
commented
1 year ago Thanks @gvart and @eduanb. We are going to add official support for GraalVM native image in 3.1
goafabric
commented
1 year ago @maciejwalkowiak having native support would be really awesome
for s3 we also need "S3ObjectContentTypeResolver.properties" but registering that .. will bump into the next error and the next
is there any timeline or suggestion concerning the 3.1 release ?
@goafabric : remindme
maciejwalkowiak
commented
1 year ago @MatejNedic is working on native support. @MatejNedic do you have an ETA?
MatejNedic
commented
1 year ago @maciejwalkowiak I think it will be faster if we split issue for each integration.
I have not started a work on S3 yet. Will do more work this week but can't say when it will be done.
goafabric
commented
1 year ago @MatejNedic thanks for your work here, unfortunately i just don't think this will be enough for s3 support, though i am unsure if this is in scope here ?
even with native hints incorporated from your PR, at least putobject and listBuckets will break, see below for details I tried to figure this out on my own, with some experience concerning native images, but this one is beyond my capabilities honestly.
In both scenarios there seems to be a problem with the endpoint resolution. The putobject scenario can be mitigated by using "httpS" .. which will resolve correctly but than of course break, when your s3 is not supporting https
---- cut --
putobject)
will bail out with
"Custom endpoint http://localhost:9100 was not a valid URI"
because of a check at: AwsEndpointProviderUtils.valueAsEndpointOrThrow(AwsEndpointProviderUtils.java:135)
listbuckets) will break with
Caused by: java.lang.RuntimeException: no rules inside of tree rule matched—invalid rules (software.amazon.awssdk.services.s3.endpoints.internal.RuleEvaluator$1@592adda7) at software.amazon.awssdk.services.s3.endpoints.internal.RuleEvaluator$1.visitTreeRule(RuleEvaluator.java:134) ~[na:na] at software.amazon.awssdk.services.s3.endpoints.internal.RuleEvaluator$1.visitTreeRule(RuleEvaluator.java:125) ~[na:na] at software.amazon.awssdk.services.s3.endpoints.internal.TreeRule.accept(TreeRule.java:30) ~[na:na] at software.amazon.awssdk.services.s3.endpoints.internal.RuleEvaluator.lambda$handleRule$4(RuleEvaluator.java:125) ~[callee-service:na]
MatejNedic
commented
1 year ago Hey @goafabric , I still have not finished S3 integration fully, tbh not sure when I will have time to fully go through it.
This happens when you are using S3Template?
All of the things that are not coming from Spring Cloud AWS, but rather from AWS Java SDK v2 should be opened to them on their GitHub.
goafabric
commented
1 year ago Hi @MatejNedic thanks for your very fast reply .. it happens when you use s3Client, but from what I see s3Template mostly just delegates to s3Client, and wraps a little of the boiler plate request building (putObject -> store)
For me it's unclear if it is related to to AWS or Spring Cloud. For the Custom Endpoint resolution, i would guess, it's the way that the Endpoint is resolved from SpringProperties and passed to the S3 Engine. I also tried to get around this by using one of the Customizers, but to no avail.
Also from my experience, parties outside of Spring usually do not care much about GraalVM. For Resilience4j i was able to create a simple PR, but this one her is beyond my capabilities.
So it would be nice if we find a solution .. as the bootstrap already passes .. i guess we are already near the finish line Thx
add:
s3Template.store, gives the same error
"Custom endpoint http://localhost:9100 was not a valid URI"
MatejNedic
commented
1 year ago Hey @goafabric , interesting... Could you provide a simple sample of not working S3 native, I will try to see what hints can we add in Spring Cloud AWS to resolve it.
Also, have you tried https://www.graalvm.org/latest/reference-manual/native-image/metadata/AutomaticMetadataCollection/#:~:text=GraalVM%20provides%20a%20Tracing%20Agent,on%20a%20regular%20Java%20VM. ?
The problem is if it is AWS SDK it should be fixed on their side not ours since it can cause issues with compatibility in future releases. Spring Kafka has the same issue you can check their GitHub issues and will find that they don't add hints for kafka specifics.
Edit: I played with the sample a little bit and it worked like a charm but this was for the localstack and controller part. Will check into more details once I have more time on hand.
goafabric
commented
1 year ago Upload Archiv.zip
goafabric
commented
1 year ago Hey @MatejNedic thx again, i've attached an example with gradle. With graalvm(23) installed you can just execute the native-compile command, no need for s3 beeing started.
Honestly i've spent a whole day figuring out what is going on, and vor the first time leveraging the tracing agent you mentioned.
But to now avail .. except for knowing no a lot of S3 Sourcecode :) The problem definitely arises from the custom endpoint => In your example are you also using a custom endpoint with HTTP ? And also with http only, https works.
Debugging inside S3 reveals some funky resolution of the custom endpoint, with visitor pattern, expression evaluation and a lot of classes involved. And in the AwsEndpointProviderUtils.java:135, there is the check where it checks if the Endpoint is of Type "Endpoint" or "Str" => YOu have to uncomment one line in my example, noted with a comment
But the thing is, everything is mostly just pure functional programming .. there is reflection involved later one, when executing the HTTP Call, but not up to that point.
And sure you are right about future versions, but as long as 3rd party libs will not work with the reachability repository, i think this will always be a problem ..
Maybe you will see something i havent, but i have currently no more clues. I assume the problem might not be reflection, proxy or other hints, but maybe some more general problem like code beeing erased during graalvm compile. I can also attach the tracing agent output .. and you propabely see what i mean s3-metadata.zip
MatejNedic
commented
11 months ago Hey @goafabric , Sorry for late reply are you by any chance using crt client?
goafabric
commented
11 months ago @MatejNedic thx 4 reaching out :) no i am not .. at least not that i am aware of i did however now try the latest version of localstack together with the examples you proposed earlier and also with this combination i get the same exceptions as mentioned here
the only "workaround" is to use https instead of http .. so there seems to be some kind of funky url detection going on that fails for native
goafabric
commented
9 months ago Do we have any update on the S3 Native Topic ? I took a brief look into the crt client, but did not get very far btw
klopfdreh
commented
7 months ago Please add a file to META-INF/native-image/io.awspring.cloud.core/resource-config.json in the package that contains the properties file. (https://github.com/awspring/spring-cloud-aws/blob/main/spring-cloud-aws-core/src/main/resources/io/awspring/cloud/core/SpringCloudClientConfiguration.properties)
with the content:
{
"resources": [
{
"pattern": "io/awspring/cloud/core/SpringCloudClientConfiguration.properties"
}
]
}So that we can add the following for the native-build:
native-image ... -H:ResourceConfigurationResources=META-INF/native-image/io.awspring.cloud/spring-cloud-aws/resource-config.json
See: https://www.graalvm.org/latest/reference-manual/native-image/dynamic-features/Resources/
Offtopic: S3 requires: "pattern": "io/awspring/cloud/s3/S3ObjectContentTypeResolver.properties"
With those updates I was able to use the S3PathMatchingResourcePatternResolver and access all files in a S3 bucket.
MatejNedic
commented
7 months ago Hey @klopfdreh , It is already present in PR https://github.com/awspring/spring-cloud-aws/pull/856 . Check file Will try to finish it finally this month have to refactor certain parts.
klopfdreh
commented
7 months ago Hey @MatejNedic,
thanks a lot for the hint. Here are some related issues in
AWS SDK V2: https://github.com/aws/aws-sdk-java-v2/issues/5127 AWS CRT: https://github.com/awslabs/aws-crt-java/issues/780
For native image those issues appear if you use the latest version of the SDK and need also to be fixed in order to use Spring Cloud AWS with S3 TransferManager or just for download resources from S3 with S3PathMatchingResourcePatternResolver.
Type: Bug Component: DynamodDB
Describe the bug When I was trying to use
io.awspring.cloud:spring-cloud-aws-starter-dynamodbstarter with a Native Image for an AWS Lambda, application failed to startup due to a missing resource file in the classpath. Stacktrace:As a solution would be to provide a resource-hint for GraalVM i.e.