Azure-Functions broke Storage API

alwyna commented 4 years ago

Expected: Be able to use azure-storage-blob Java API from azure function. Actual: Azure function generates runtime error because of library mismatch.

Azure storage blob API does not work at all from inside azure functions due to netty conflict. To make matter worse, something along the chain uses a default static variable which makes shading impossible, without also impacting how it's being used by the functions.

When the library is used from a cli app, it works fine, but not when inside the Azure function context.

azure-functions-java-library 1.3.1 (current) azure-storage-blob 12.4.0 (current)

Executed 'Functions.TestTimer' (Failed, Id=7055efb2-0332-403e-83f4-61364f20c4c3) [3/9/20 8:11:59 PM] System.Private.CoreLib: Exception while executing function: Functions.TestTimer. System.Private.CoreLib: Result: Failure [3/9/20 8:11:59 PM] Exception: NoSuchMethodError: io.netty.channel.SingleThreadEventLoop.<init>(Lio/netty/channel/EventLoopGroup;Ljava/util/concurrent/Executor;ZLjava/util/Queue;Ljava/util/Queue;Lio/netty/util/concurrent/RejectedExecutionHandler;)V [3/9/20 8:11:59 PM] Stack: java.lang.reflect.InvocationTargetException [3/9/20 8:11:59 PM] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [3/9/20 8:11:59 PM] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [3/9/20 8:11:59 PM] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [3/9/20 8:11:59 PM] at java.lang.reflect.Method.invoke(Method.java:498) [3/9/20 8:11:59 PM] at com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22) [3/9/20 8:11:59 PM] at com.microsoft.azure.functions.worker.broker.JavaMethodExecutor.execute(JavaMethodExecutor.java:54) [3/9/20 8:11:59 PM] at com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:53) [3/9/20 8:11:59 PM] at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:33) [3/9/20 8:11:59 PM] at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10) [3/9/20 8:11:59 PM] at com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:45) [3/9/20 8:11:59 PM] at com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:92) [3/9/20 8:11:59 PM] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [3/9/20 8:11:59 PM] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [3/9/20 8:11:59 PM] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [3/9/20 8:11:59 PM] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [3/9/20 8:11:59 PM] at java.lang.Thread.run(Thread.java:748) [3/9/20 8:11:59 PM] Caused by: java.lang.NoSuchMethodError: io.netty.channel.SingleThreadEventLoop.<init>(Lio/netty/channel/EventLoopGroup;Ljava/util/concurrent/Executor;ZLjava/util/Queue;Ljava/util/Queue;Lio/netty/util/concurrent/RejectedExecutionHandler;)V [3/9/20 8:11:59 PM] at io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:87) [3/9/20 8:11:59 PM] at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:143) [3/9/20 8:11:59 PM] at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35) [3/9/20 8:11:59 PM] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) [3/9/20 8:11:59 PM] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) [3/9/20 8:11:59 PM] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) [3/9/20 8:11:59 PM] at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) [3/9/20 8:11:59 PM] at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:104) [3/9/20 8:11:59 PM] at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:91) [3/9/20 8:11:59 PM] at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:68) [3/9/20 8:11:59 PM] at reactor.netty.resources.DefaultLoopEpoll.newEventLoopGroup(DefaultLoopEpoll.java:64) [3/9/20 8:11:59 PM] at reactor.netty.resources.DefaultLoopResources.cacheNativeServerLoops(DefaultLoopResources.java:234) [3/9/20 8:11:59 PM] at reactor.netty.resources.DefaultLoopResources.cacheNativeClientLoops(DefaultLoopResources.java:248) [3/9/20 8:11:59 PM] at reactor.netty.resources.DefaultLoopResources.onClient(DefaultLoopResources.java:184) [3/9/20 8:11:59 PM] at reactor.netty.tcp.TcpResources.onClient(TcpResources.java:179) [3/9/20 8:11:59 PM] at reactor.netty.http.client.HttpClientConnect$HttpTcpClient.connect(HttpClientConnect.java:134) [3/9/20 8:11:59 PM] at reactor.netty.tcp.TcpClientOperator.connect(TcpClientOperator.java:43) [3/9/20 8:11:59 PM] at reactor.netty.tcp.TcpClientOperator.connect(TcpClientOperator.java:43) [3/9/20 8:11:59 PM] at reactor.netty.tcp.TcpClientOperator.connect(TcpClientOperator.java:43) [3/9/20 8:11:59 PM] at reactor.netty.tcp.TcpClientOperator.connect(TcpClientOperator.java:43) [3/9/20 8:11:59 PM] at reactor.netty.tcp.TcpClientOperator.connect(TcpClientOperator.java:43) [3/9/20 8:11:59 PM] at reactor.netty.tcp.TcpClient.connect(TcpClient.java:187) [3/9/20 8:11:59 PM] at reactor.netty.http.client.HttpClientFinalizer.connect(HttpClientFinalizer.java:68) [3/9/20 8:11:59 PM] at reactor.netty.http.client.HttpClientFinalizer.responseConnection(HttpClientFinalizer.java:85) [3/9/20 8:11:59 PM] at com.azure.core.http.netty.NettyAsyncHttpClient.send(NettyAsyncHttpClient.java:72) [3/9/20 8:11:59 PM] at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:44) [3/9/20 8:11:59 PM] at com.azure.storage.common.policy.ScrubEtagPolicy.process(ScrubEtagPolicy.java:35) [3/9/20 8:11:59 PM] at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:46) [3/9/20 8:11:59 PM] at com.azure.core.http.policy.HttpLoggingPolicy.process(HttpLoggingPolicy.java:84) [3/9/20 8:11:59 PM] at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:46) [3/9/20 8:11:59 PM] at com.azure.storage.common.policy.ResponseValidationPolicyBuilder$ResponseValidationPolicy.process(ResponseValidationPolicyBuilder.java:77) [3/9/20 8:11:59 PM] at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:46) [3/9/20 8:11:59 PM] at com.azure.storage.common.policy.RequestRetryPolicy.attemptAsync(RequestRetryPolicy.java:114) [3/9/20 8:11:59 PM] at com.azure.storage.common.policy.RequestRetryPolicy.process(RequestRetryPolicy.java:48) [3/9/20 8:11:59 PM] at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:46) [3/9/20 8:11:59 PM] at com.azure.storage.common.policy.StorageSharedKeyCredentialPolicy.process(StorageSharedKeyCredentialPolicy.java:41) [3/9/20 8:11:59 PM] at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:46) [3/9/20 8:11:59 PM] at com.azure.core.http.policy.AddDatePolicy.lambda$process$0(AddDatePolicy.java:29) [3/9/20 8:11:59 PM] at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44) [3/9/20 8:11:59 PM] at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) [3/9/20 8:11:59 PM] at reactor.core.publisher.Mono.subscribe(Mono.java:4087) [3/9/20 8:11:59 PM] at reactor.core.publisher.Mono.block(Mono.java:1662) [3/9/20 8:11:59 PM] at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:99) [3/9/20 8:11:59 PM] at com.azure.storage.blob.BlobContainerClient.existsWithResponse(BlobContainerClient.java:195) [3/9/20 8:11:59 PM] at com.azure.storage.blob.BlobContainerClient.exists(BlobContainerClient.java:179)

rochlina commented 4 years ago

Have the same issue with azure-functions-java-library 1.3.1 and azure-storage-blob 12.5.0

sitase commented 4 years ago

Seeing the same with com.microsoft.azure.functions:azure-functions-java-library:1.3.1 and com.azure:azure-security-keyvault-secrets:4.1.0.

gkh70 commented 4 years ago

Same issue with CLI app.
azure-functions-java-library:1.3.1 azure-servicebus:3.1.2 azure-storage-blob:12.3.0 azure-documentdb:2.4.7

following stacktrace when calling upload method of com.azure.storage.blob.BlockBlobClient class:

[2020-03-30 5:24:53 [2020-03-30 5:24:53 [2020-03-30 5:24:53 [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] [2020-03-30 5:24:53 PM] PM] System.Private.CoreLib: Exception while executing function: Functions.pim-ingestion-function. System.Private.CoreLib: Result: Failure PM] Exception: NoSuchMethodError: io.netty.channel.SingleThreadEventLoop.(Lio/netty/channel/EventLoopGroup;Ljava/util/concurrent/Executor;ZLjava/util/Queue;Ljava/util/Queue;Lio/netty/util/concurrent/RejectedExecutionHandler;)V PM] Stack: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22) at com.microsoft.azure.functions.worker.broker.JavaMethodExecutor.execute(JavaMethodExecutor.java:54) at com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:53) at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:33) at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10) at com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:45) at com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:92) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) PM] Caused by: java.lang.NoSuchMethodError: io.netty.channel.SingleThreadEventLoop.(Lio/netty/channel/EventLoopGroup;Ljava/util/concurrent/Executor;ZLjava/util/Queue;Ljava/util/Queue;Lio/netty/util/concurrent/RejectedExecutionHandler;)V at io.netty.channel.kqueue.KQueueEventLoop.(KQueueEventLoop.java:77) at io.netty.channel.kqueue.KQueueEventLoopGroup.newChild(KQueueEventLoopGroup.java:143) at io.netty.channel.kqueue.KQueueEventLoopGroup.newChild(KQueueEventLoopGroup.java:32) at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:84) at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:58) at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:47) at io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:59) at io.netty.channel.kqueue.KQueueEventLoopGroup.(KQueueEventLoopGroup.java:102) at io.netty.channel.kqueue.KQueueEventLoopGroup.(KQueueEventLoopGroup.java:89) at io.netty.channel.kqueue.KQueueEventLoopGroup.(KQueueEventLoopGroup.java:65) at reactor.netty.resources.DefaultLoopKQueue.newEventLoopGroup(DefaultLoopKQueue.java:64) at reactor.netty.resources.DefaultLoopResources.cacheNativeServerLoops(DefaultLoopResources.java:234) at reactor.netty.resources.DefaultLoopResources.cacheNativeClientLoops(DefaultLoopResources.java:248) at reactor.netty.resources.DefaultLoopResources.onClient(DefaultLoopResources.java:184) at reactor.netty.tcp.TcpResources.onClient(TcpResources.java:179) at reactor.netty.http.client.HttpClientConnect$HttpTcpClient.connect(HttpClientConnect.java:134) at reactor.netty.tcp.TcpClientOperator.connect(TcpClientOperator.java:43) at reactor.netty.tcp.TcpClientOperator.connect(TcpClientOperator.java:43) at reactor.netty.tcp.TcpClientOperator.connect(TcpClientOperator.java:43) at reactor.netty.tcp.TcpClientOperator.connect(TcpClientOperator.java:43) at reactor.netty.tcp.TcpClientOperator.connect(TcpClientOperator.java:43) at reactor.netty.tcp.TcpClient.connect(TcpClient.java:187) at reactor.netty.http.client.HttpClientFinalizer.connect(HttpClientFinalizer.java:68) at reactor.netty.http.client.HttpClientFinalizer.responseConnection(HttpClientFinalizer.java:85) at com.azure.core.http.netty.NettyAsyncHttpClient.send(NettyAsyncHttpClient.java:70) at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:44) at com.azure.storage.common.policy.ScrubEtagPolicy.process(ScrubEtagPolicy.java:35) at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:46) at com.azure.core.http.policy.HttpLoggingPolicy.process(HttpLoggingPolicy.java:84) at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:46) at com.azure.storage.common.policy.ResponseValidationPolicyBuilder$ResponseValidationPolicy.process(ResponseValidationPolicyBuilder.java:77) at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:46) at com.azure.storage.common.policy.RequestRetryPolicy.attemptAsync(RequestRetryPolicy.java:114) at com.azure.storage.common.policy.RequestRetryPolicy.process(RequestRetryPolicy.java:48) at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:46) at com.azure.storage.common.policy.StorageSharedKeyCredentialPolicy.process(StorageSharedKeyCredentialPolicy.java:41) at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:46) at com.azure.core.http.policy.AddDatePolicy.lambda$process$0(AddDatePolicy.java:29) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44) at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) at reactor.core.publisher.Mono.subscribe(Mono.java:4087) at reactor.core.publisher.Mono.block(Mono.java:1662) at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:95) at com.azure.storage.blob.specialized.BlockBlobClient.uploadWithResponse(BlockBlobClient.java:236) at com.azure.storage.blob.specialized.BlockBlobClient.upload(BlockBlobClient.java:188) at com.azure.storage.blob.specialized.BlockBlobClient.upload(BlockBlobClient.java:162)

wcc8088 commented 4 years ago

same issue with visual studio code azure-functions-java-library 1.3.1 azure-storage-blob 12.6.0

... [2020. 4. 16. 오전 7:37:13] 400 [parallel-1] DEBUG reactor.netty.resources.DefaultLoopEpoll - Default Epoll support : true [2020. 4. 16. 오전 7:37:13] 401 [parallel-1] DEBUG reactor.netty.resources.DefaultLoopKQueue - Default KQueue support : false [2020. 4. 16. 오전 7:37:13] 406 [parallel-1] ERROR reactor.core.scheduler.Schedulers - Scheduler worker in group main failed with an uncaught exception [2020. 4. 16. 오전 7:37:13] java.lang.NoSuchMethodError: io.netty.channel.SingleThreadEventLoop.(Lio/netty/channel/EventLoopGroup;Ljava/util/concurrent/Executor;ZLjava/util/Queue;Ljava/util/Queue;Lio/netty/util/concurrent/RejectedExecutionHandler;)V [2020. 4. 16. 오전 7:37:13] at io.netty.channel.epoll.EpollEventLoop.(EpollEventLoop.java:90) [2020. 4. 16. 오전 7:37:13] at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:151) [2020. 4. 16. 오전 7:37:13] at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35) [2020. 4. 16. 오전 7:37:13] at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:84) [2020. 4. 16. 오전 7:37:13] at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:58) [2020. 4. 16. 오전 7:37:13] at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:47) [2020. 4. 16. 오전 7:37:13] at io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:59) [2020. 4. 16. 오전 7:37:13] at io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:112) [2020. 4. 16. 오전 7:37:13] at io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:99) [2020. 4. 16. 오전 7:37:13] at io.netty.channel.epoll.EpollEventLoopGroup.(EpollEventLoopGroup.java:76) [2020. 4. 16. 오전 7:37:13] at reactor.netty.resources.DefaultLoopEpoll.newEventLoopGroup(DefaultLoopEpoll.java:64) ...

merric-rocketpartners commented 4 years ago

I'm seeing the same thing, but I only started seeing it when I deployed the Function in a premium plan instead of in the Consumption plan.

anuchandy commented 4 years ago

as a workaround, you can exclude the Epoll dependency

<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core-http-netty</artifactId>
<version>{put-version-here}</version>
<exclusions>
    <exclusion>
        <groupId>io.netty</groupId>
        <artifactId>netty-transport-native-epoll</artifactId>
    </exclusion>
    <exclusion>
        <groupId>io.netty</groupId>
        <artifactId>netty-transport-native-unix-common</artifactId>
    </exclusion>
</exclusions>
</dependency>

tedescomicchidev commented 4 years ago

Hi all - I've just given the "netty depency"-approach described above a try, but unfortunately I get an exception: Exception has occurred: java.lang.NoClassDefFoundError "java.lang.NoClassDefFoundError: Could not initialize class reactor.netty.http.client.HttpClientSecure" error

anuchandy commented 4 years ago

@tedescomicchidev - In my case exclusion worked on "Linux + azure-functions-core-tools-3". Note that this was on dev env, details of env is here. I didn't try deploying to a real Function plan :(

From the error, it looks like you're running into the same issue https://github.com/Azure/azure-functions-java-worker/issues/365, this issue is due to the version conflict between the netty version GRPC brings vs the one Azure SDK depends on.