[FEATURE REQ] Storage.Queues change internal modifier to public for in-memory tests

[maris-jurgenbergs]

Library or service name. Microsoft.Azure.WebJobs.Extensions.Storage.Queues

Is your feature request related to a problem? Please describe. I wanted to create an integration test, that emulates storage in memory as it was in the previous webjobs sdk.

Old webjobs sdk test implementation, that can be faked: https://github.com/Azure/azure-webjobs-sdk/blob/18ddad07187d94f6e890c6ebe1c02ba758d1f490/test/Microsoft.Azure.Webjobs.Extensions.Storage.UnitTests/FakeStorageAccountProvider.cs#L26

    public class FakeStorageAccount : StorageAccount
    {
        private FakeStorage.FakeAccount _account2 = new FakeStorage.FakeAccount();

        public override CloudQueueClient CreateCloudQueueClient()
        {
            return _account2.CreateCloudQueueClient();
        }
        public override CloudBlobClient CreateCloudBlobClient()
        {
            return _account2.CreateCloudBlobClient();
        }

        public override CloudTableClient CreateCloudTableClient()
        {
            return _account2.CreateCloudTableClient();
        }

        public override string Name => _account2.Name;
        public override bool IsDevelopmentStorageAccount() { return true; }
    }

https://github.com/Azure/azure-webjobs-sdk/blob/18ddad07187d94f6e890c6ebe1c02ba758d1f490/test/FakeStorage/Queue/FakeQueueClient.cs

public class FakeQueueClient : CloudQueueClient
    {
...
        public override CloudQueue GetQueueReference(string queueName)
        {
            return new FakeQueue(this, queueName);
        }
...

This is possible because the classes are public and can be used in different projects and not just in local assemblies.

New azure sdk implementation uses internal everywhere: I identified, that the main class, that i could fake is QueueServiceClientProvider. And if i could fake the class, then i could pass a fake QueueServiceClient, that can create a fake QueueClient, that internally would use ConcurrentQueue in order to run tests only in memory and without an azure storage emulator.

    internal class QueueServiceClientProvider : StorageClientProvider<QueueServiceClient, QueueClientOptions>
    {
        public QueueServiceClientProvider(IConfiguration configuration, AzureComponentFactory componentFactory, AzureEventSourceLogForwarder logForwarder)
            : base(configuration, componentFactory, logForwarder) {}

        protected override QueueServiceClient CreateClientFromConnectionString(string connectionString, QueueClientOptions options)
        {
            return new QueueServiceClient(connectionString, options);
        }

        protected override QueueServiceClient CreateClientFromTokenCredential(Uri endpointUri, TokenCredential tokenCredential, QueueClientOptions options)
        {
            return new QueueServiceClient(endpointUri, tokenCredential, options);
        }
    }

Possible solution: Can we turn some classes to public in order to be able to fake some types or do you suggest some other solution how i could test webjobs in memory and without azure storage emulator?

[jsquire]

Thank you for your feedback. Tagging and routing to the team best able to assist.

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[seanmcc-msft]

Hi @maris-jurgenbergs, I'd recommend using a mocking framework like MOQ.

using Azure;
using Azure.Core;
using Azure.Storage.Queues;
using Azure.Storage.Queues.Models;
using Moq;
using NUnit.Framework;
using System;
using System.Collections.Generic;
using System.Diagnostics.CodeAnalysis;
using System.IO;
using System.Threading.Tasks;

namespace ConsoleApp8
{
    class Program
    {
        static async Task Main(string[] args)
        {
            // Arrange
            Mock<QueueClient> queueClient = new Mock<QueueClient>();

            QueueMessage[] queueMessages = new QueueMessage[1];
            queueMessages[0] = QueuesModelFactory.QueueMessage("messageId", "popReceipt", "messageText", 0);

            queueClient.Setup(queueClient => queueClient.ReceiveMessagesAsync()).Returns(Task.FromResult(Response.FromValue(queueMessages, new MockResponse())));

            // Act
            Response<QueueMessage[]> returnedResponse = await queueClient.Object.ReceiveMessagesAsync();

            // Assert
            Assert.AreEqual("messageText", returnedResponse.Value[0].MessageText);
            queueClient.Verify(queueClient => queueClient.ReceiveMessagesAsync());
        }
    }

    class MockResponse : Response
    {
        public override int Status => throw new NotImplementedException();

        public override string ReasonPhrase => throw new NotImplementedException();

        public override Stream ContentStream { get => throw new NotImplementedException(); set => throw new NotImplementedException(); }
        public override string ClientRequestId { get => throw new NotImplementedException(); set => throw new NotImplementedException(); }

        public override void Dispose()
        {
            throw new NotImplementedException();
        }

        protected override bool ContainsHeader(string name)
        {
            throw new NotImplementedException();
        }

        protected override IEnumerable<HttpHeader> EnumerateHeaders()
        {
            throw new NotImplementedException();
        }

        protected override bool TryGetHeader(string name, [NotNullWhen(true)] out string value)
        {
            throw new NotImplementedException();
        }

        protected override bool TryGetHeaderValues(string name, [NotNullWhen(true)] out IEnumerable<string> values)
        {
            throw new NotImplementedException();
        }
    }
}

[maris-jurgenbergs]

Yeah Mock is a good scenario, when I would want to have just a unit test to test something specific. What I wanted was a integration test, that would create the host, the services and the storage specific functionality.

If I could replace one part that was responsible for saving the data in storage with in-memory functionality, then I could create a integration test, that is lightweight compared to a solution, if I would have to use a storage emulator dependency (which means configuring a build agent, that has an emulator configured and up to date).

I encountered, where a problem happens with WebJobs not starting properly and the problem was that after a library update one of the dependent libraries was not fully updated. Everything compiles but I got a runtime error later. With a unit test I might not get that error, because mocks don't use the whole original code.

Trying to have an automated test that checks if WebJobs are fully running.

[kasobol-msft]

@maris-jurgenbergs The fakes present in earlier version of extension has not been carried forward and replaced with Azurite. Functionality wise they were any way overlapping with the Azurite, so we decided to rely on official emulator rather than try to maintain them in the long term (and risk divergence from actual runtime behavior that storage service has). The downside was as you mentioned necessity of maintaining extra process in test runs, but that turned out to be simpler to solve. See below.

AzuriteFixture AzuriteNUnitFixture Agent Instalation