search

quarkiverse / quarkus-azure-services

Quarkus extensions for Azure services
Apache License 2.0
14 stars 18 forks source link

Proposal for Quarkus Extension - Azure Storage Queue - Working Prototype Available #150

Open sarxos opened 1 year ago

sarxos commented 1 year ago

Dear Azure Services community of Quarkiverse,

I've been working on implementing a Quarkus extension for Azure Storage Queue and have made good progress. I'd like to share my thoughts on the current state of my implementation and seek your input on the best way we can move forward to merge it into the official release.

Background

I've based my work on the existing Azure Storage Blob extension, which allowed me to quickly build a working prototype extension for the Azure Storage Queue. In most places, the prototype uses a copy of the code from the Blob extension to establish the basic functionality. The working prototype together with working integration tests and a bit of documentation can be found in my fork.

Current Status

The prototype is working rather well and meets the initial requirements for interacting with Azure Storage Queues. However, there are certain aspects of my implementation that are duplicated between the two extensions. In particular, the following components are duplicated in most places:

Some might argue that a functional PoC is sufficient, but due to my strong inclination towards maintaining clean code, the absence of a suitable abstraction for the classes mentioned above is incredibly frustrating to me. 😉

Opportunity

I believe there is a valuable opportunity to deduplicate the code by extracting common components and making them more universally applicable. By doing so, we can achieve cleaner and more maintainable code while reducing redundancy between the Blob and Queue extensions (and most likely a new extension for Storage Table that may potentially be created in the future, since it would also use the same abstraction).

Proposed Refactoring

What I would like to propose would involve extracting the common components mentioned above into a shared dependency (either a new or existing one, e.g. quarkus-azure-core-deployment), which would be utilized by both the Blob and Queue extensions (and most likely Table in future). After performing some preliminary exercises in this approach I found out that this would entail changes to the configuration paths, if we are willing to keep it clear, as well as some structural modifications to accommodate the new shared components.

Challenge

It's important to note that extracting these common components and making mentioned changes would result in a break of backward compatibility within existing configurations. The reason lies mainly in the fact that we have a blob fragment in the connection-string configuration path. When implementing my extension I did use the same approach and placed a queue in the path as well. However, with the assumption that we have Blob, Queue, and most likely would have a Table extension in a future, we will end up with three different configuration paths to carry the same exact value:

quarkus.azure.storage.blob.connection-string = {a storage account connection here}
quarkus.azure.storage.queue.connection-string = {a storage account connection here}
quarkus.azure.storage.table.connection-string = {a storage account connection here}
                      ^- blob, queue and table with DevServicesConfig for each, but exactly the same value

Similarily, we would end up with three, exactly the same Azurite containers running in dev mode, which is a complete waste of resources (ah, just thinking about this with my mere 16 GB of RAM, annoys me as hell). The only difference between these containers would be port number they exposes, specifically:

This sucks.

Solution

What I would like to do is to drop the blob, queue or table from the connection string configuration path, and have it common between the extensions:

quarkus.azure.storage.connection-string = {a storage account connection here}
quarkus.azure.storage.devservices.image-name = {docker image}
                      ^- no blob here

While having a smaller configuration just for the ports, e.g.:

quarkus.azure.storage.devservices.blob.port = {port number for blobs}
quarkus.azure.storage.devservices.queue.port = {port number for queues}
quarkus.azure.storage.devservices.table.port = {port number for tables}

By doing so we can have only one configuration for connection-string and only one Azurite container instead of three separate ones. However, this would break backward compatibility, since, as you can see, the old configurations users use in the wild would become invalid.

While I understand the significance of maintaining compatibility, I also believe in the long-term benefits of cleaner code and easier maintenance, which may outweigh this concern. But I'm at a crossroads.

Community Input

At this juncture, I would greatly appreciate the community's insights and opinions on how we can proceed. I'm seeking your thoughts on the following questions:

Your Feedback

I understand that this decision may have implications for the broader Quarkus Azure Services ecosystem, and I value your opinions on this matter. Please share your thoughts, concerns, or alternative suggestions regarding the direction we can take. I believe that any feedback in regard to this matter will help us make a well-informed decision that benefits the entire user community.

Thank you for your time and consideration.

majguo commented 1 year ago

Hello @sarxos First of all, I'm sorry for late response as I was in vacation last week and just returned back this week. Your proposal is great and generally I like it which targets to enrich the Azure storage Quarkus extensions and purse cleaner code by reducing the duplicated code.

Regarding to your Community Input, I'm going to look into it these days and also discuss with my team before sharing further feedback.

parasjain27031994 commented 4 months ago

Hi @sarxos , I have been trying to generate a native image of my Quarkus application that uses EvenHub & Storage Queue, however I have been running into below exception:

Fatal error: com.oracle.graal.pointsto.util.AnalysisError$ParsingError: Error encountered while parsing com.azure.core.http.vertx.VertxAsyncHttpClientProvider.createInstance(VertxAsyncHttpClientProvider.java:48) 
Parsing context:
   at com.azure.core.http.vertx.VertxAsyncHttpClientProvider.createInstance(VertxAsyncHttpClientProvider.java:57)
   at com.azure.core.implementation.http.HttpClientProviders.lambda$createInstance$0(HttpClientProviders.java:58)
   at com.azure.core.implementation.http.HttpClientProviders$$Lambda/0x00000007c20a0c88.apply(Unknown Source)
   at sun.security.ec.ParametersMap$1.get(ParametersMap.java:78)
   at java.util.Optional.orElseThrow(Optional.java:403)
   at jakarta.enterprise.inject.spi.CDI.getCDIProvider(CDI.java:97)
   at jakarta.enterprise.inject.spi.CDI.current(CDI.java:64)
   at io.smallrye.stork.impl.RoundRobinLoadBalancerProviderLoader.<init>(RoundRobinLoadBalancerProviderLoader.java:20)
   at com.oracle.svm.core.code.FactoryMethodHolder.RoundRobinLoadBalancerProviderLoader_constructor_6e399d40c2ca969b7037c4111991c13965d24fbd(generated:0)
   at static root method.(Unknown Source)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.AnalysisError.parsingError(AnalysisError.java:149)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.createFlowsGraph(MethodTypeFlow.java:184)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureFlowsGraphCreated(MethodTypeFlow.java:153)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.getOrCreateMethodFlowsGraphInfo(MethodTypeFlow.java:111)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.typestate.DefaultVirtualInvokeTypeFlow.onObservedUpdate(DefaultVirtualInvokeTypeFlow.java:114)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.TypeFlow.update(TypeFlow.java:620)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.PointsToAnalysis$1.run(PointsToAnalysis.java:491)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.executeCommand(CompletionExecutor.java:187)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.lambda$executeService$0(CompletionExecutor.java:171)
    at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
Caused by: org.graalvm.compiler.java.BytecodeParser$BytecodeParserError: org.graalvm.compiler.debug.GraalError: com.oracle.svm.core.util.UserError$UserException: Class initialization of com.azure.core.http.vertx.VertxAsyncHttpClientProvider$GlobalVertxHttpClient failed. Use the option 
    '--initialize-at-run-time=com.azure.core.http.vertx.VertxAsyncHttpClientProvider$GlobalVertxHttpClient'
 to explicitly request initialization of this class at run time.
    at parsing com.azure.core.http.vertx.VertxAsyncHttpClientProvider.createInstance(VertxAsyncHttpClientProvider.java:49)
    at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.throwParserError(BytecodeParser.java:2553)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.throwParserError(SharedGraphBuilderPhase.java:182)
    at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3434)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.iterateBytecodesForBlock(SharedGraphBuilderPhase.java:743)
    at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.handleBytecodeBlock(BytecodeParser.java:3386)
    at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBlock(BytecodeParser.java:3228)
    at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.build(BytecodeParser.java:1137)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.build(SharedGraphBuilderPhase.java:162)
    at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.buildRootMethod(BytecodeParser.java:1029)
    at jdk.internal.vm.compiler/org.graalvm.compiler.java.GraphBuilderPhase$Instance.run(GraphBuilderPhase.java:101)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase.run(SharedGraphBuilderPhase.java:116)
    at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.run(Phase.java:49)
    at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:434)
    at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:42)
    at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:38)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.AnalysisParsedGraph.parseBytecode(AnalysisParsedGraph.java:146)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.parseGraph(AnalysisMethod.java:895)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.ensureGraphParsedHelper(AnalysisMethod.java:860)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.ensureGraphParsed(AnalysisMethod.java:843)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.parse(MethodTypeFlowBuilder.java:186)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.apply(MethodTypeFlowBuilder.java:621)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.createFlowsGraph(MethodTypeFlow.java:167)
    ... 13 more
Caused by: org.graalvm.compiler.debug.GraalError: com.oracle.svm.core.util.UserError$UserException: Class initialization of com.azure.core.http.vertx.VertxAsyncHttpClientProvider$GlobalVertxHttpClient failed. Use the option 
    '--initialize-at-run-time=com.azure.core.http.vertx.VertxAsyncHttpClientProvider$GlobalVertxHttpClient'
 to explicitly request initialization of this class at run time.
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.AnalysisFuture.setException(AnalysisFuture.java:49)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:322)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.AnalysisFuture.ensureDone(AnalysisFuture.java:63)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.ensureOnTypeReachableTaskDone(AnalysisType.java:696)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.onReachable(AnalysisType.java:590)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.AtomicUtils.atomicSetAndRun(AtomicUtils.java:49)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.lambda$registerAsReachable$8(AnalysisType.java:562)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.forAllSuperTypes(AnalysisType.java:676)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.forAllSuperTypes(AnalysisType.java:659)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.forAllSuperTypes(AnalysisType.java:655)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.registerAsReachable(AnalysisType.java:562)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisUniverse.lookupAllowUnresolved(AnalysisUniverse.java:368)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.infrastructure.AnalysisConstantPool.lookupField(AnalysisConstantPool.java:42)
    at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.lookupField(BytecodeParser.java:4343)
    at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genGetStatic(BytecodeParser.java:4902)
    at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBytecode(BytecodeParser.java:5410)
    at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3426)
    ... 32 more
Caused by: com.oracle.svm.core.util.UserError$UserException: Class initialization of com.azure.core.http.vertx.VertxAsyncHttpClientProvider$GlobalVertxHttpClient failed. Use the option 
    '--initialize-at-run-time=com.azure.core.http.vertx.VertxAsyncHttpClientProvider$GlobalVertxHttpClient'
 to explicitly request initialization of this class at run time.
    at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.UserError.abort(UserError.java:85)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationSupport.ensureClassInitialized(ClassInitializationSupport.java:195)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.AllowAllHostedUsagesClassInitializationSupport.computeInitKindAndMaybeInitializeClass(AllowAllHostedUsagesClassInitializationSupport.java:191)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.AllowAllHostedUsagesClassInitializationSupport.computeInitKindAndMaybeInitializeClass(AllowAllHostedUsagesClassInitializationSupport.java:129)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationSupport.maybeInitializeAtBuildTime(ClassInitializationSupport.java:161)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationSupport.maybeInitializeAtBuildTime(ClassInitializationSupport.java:150)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.SVMHost.onTypeReachable(SVMHost.java:310)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisUniverse.onTypeReachable(AnalysisUniverse.java:699)
    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisType.lambda$new$0(AnalysisType.java:310)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
    ... 47 more
Caused by: java.lang.ExceptionInInitializerError
    at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized0(Native Method)
    at java.base/jdk.internal.misc.Unsafe.ensureClassInitialized(Unsafe.java:1160)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationSupport.ensureClassInitialized(ClassInitializationSupport.java:177)
    ... 56 more
Caused by: jakarta.enterprise.inject.CreationException: Error creating synthetic bean [LWrsYRaP0Jv92xLS2Ep8aqzdJ1k]: jakarta.enterprise.inject.CreationException: Synthetic bean instance for io.vertx.core.Vertx not initialized yet: io_vertx_core_Vertx_74affa450965de1cd6a7217ce7ce0dbad2c16185
    - a synthetic bean initialized during RUNTIME_INIT must not be accessed during STATIC_INIT
    - RUNTIME_INIT build steps that require access to synthetic beans initialized during RUNTIME_INIT should consume the SyntheticBeansRuntimeInitBuildItem
    at io.vertx.core.Vertx_LWrsYRaP0Jv92xLS2Ep8aqzdJ1k_Synthetic_Bean.doCreate(Unknown Source)
    at io.vertx.core.Vertx_LWrsYRaP0Jv92xLS2Ep8aqzdJ1k_Synthetic_Bean.create(Unknown Source)
    at io.vertx.core.Vertx_LWrsYRaP0Jv92xLS2Ep8aqzdJ1k_Synthetic_Bean.create(Unknown Source)
    at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:119)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:38)
    at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:35)
    at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:32)
    at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
    at io.quarkus.arc.impl.ComputingCacheContextInstances.computeIfAbsent(ComputingCacheContextInstances.java:19)
------------------------------------------------------------------------------------------------------------------------    at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:35)
    at io.vertx.core.Vertx_LWrsYRaP0Jv92xLS2Ep8aqzdJ1k_Synthetic_Bean.get(Unknown Source)
    at io.vertx.core.Vertx_LWrsYRaP0Jv92xLS2Ep8aqzdJ1k_Synthetic_Bean.get(Unknown Source)
    at io.quarkus.arc.impl.ArcContainerImpl.beanInstanceHandle(ArcContainerImpl.java:559)
    at io.quarkus.arc.impl.BeanManagerImpl.getReference(BeanManagerImpl.java:71)
    at io.quarkiverse.azure.core.http.vertx.runtime.QuarkusVertxProvider.createVertx(QuarkusVertxProvider.java:44)
    at com.azure.core.http.vertx.VertxAsyncHttpClientBuilder.getVertx(VertxAsyncHttpClientBuilder.java:260)
    at com.azure.core.http.vertx.VertxAsyncHttpClientBuilder.build(VertxAsyncHttpClientBuilder.java:198)
    at com.azure.core.http.vertx.VertxAsyncHttpClientProvider$GlobalVertxHttpClient.<clinit>(VertxAsyncHttpClientProvider.java:21)
    ... 59 more
Caused by: jakarta.enterprise.inject.CreationException: Synthetic bean instance for io.vertx.core.Vertx not initialized yet: io_vertx_core_Vertx_74affa450965de1cd6a7217ce7ce0dbad2c16185
    - a synthetic bean initialized during RUNTIME_INIT must not be accessed during STATIC_INIT
    - RUNTIME_INIT build steps that require access to synthetic beans initialized during RUNTIME_INIT should consume the SyntheticBeansRuntimeInitBuildItem
    at io.vertx.core.Vertx_LWrsYRaP0Jv92xLS2Ep8aqzdJ1k_Synthetic_Bean.createSynthetic(Unknown Source)
    ... 77 more

I have been referencing the existing Azure Storage Blob extension, and have made necessary changes in my POM.

<dependency>
            <groupId>io.quarkiverse.azureservices</groupId>
            <artifactId>quarkus-azure-http-client-vertx</artifactId>
            <version>1.0.5</version>
        </dependency>
        <dependency>
            <groupId>io.quarkiverse.azureservices</groupId>
            <artifactId>quarkus-azure-core-util</artifactId>
            <version>1.0.5</version>
        </dependency>
<dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-messaging-eventhubs</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>com.azure</groupId>
                    <artifactId>azure-core-http-netty</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-storage-queue</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>com.azure</groupId>
                    <artifactId>azure-core-http-netty</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

Any idea what I might be missing here ? @majguo @sarxos

burl21 commented 1 month ago

Is there any news on this? Thanks!

majguo commented 1 month ago

The initial thought is that both Quarkus extensions for azure-messaging-eventhubs and azure-storage-queue need to be developed to support native image.

burl21 commented 1 week ago

@majguo What do you think about the solution proposed by @sarxos ?

majguo commented 1 week ago

@burl21 The idea proposed by @sarxos is great. @sarxos Are you still available and passionate to contribute your proposed extension for azure-storage-queue along with the refactoring of the existing extension azure-storage-blob to this repo as a PR? If you're ok with my suggestion, I'm going to assign this issue to you.