search

Azure / azure-event-hubs-java

☁️ Java client library for Azure Event Hubs
https://azure.microsoft.com/services/event-hubs
MIT License
51 stars 60 forks source link

Support larger payload for Event Hub Standard #463

Closed dgilling closed 4 years ago

dgilling commented 4 years ago

Actual Behavior

  1. Java client throws error that payload is larger than 200kb
  2. com.microsoft.azure.eventhubs.PayloadSizeExceededException: Size of the payload exceeded Maximum message size: 200 kb

Expected Behavior

  1. Support for 1MB payload size (See: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas) Eventhub Standard should support up to 1MB.

Versions

JamesBirdsall commented 4 years ago

What is your event hubs namespace? The Java client gets the max size from the service, and we know it supports 1MB. Also, 200kb is a strange number -- it used to be 256kb and then the default was boosted to 1MB, it was never 200 -- which makes me wonder if there's a config issue on the service.

dgilling commented 4 years ago

I have multiple due to many regions, but here is an example: "moesif-sb-dev”

Derric

On Jan 28, 2020, at 6:04 PM, JamesBirdsall notifications@github.com wrote:

What is your event hubs namespace? The Java client gets the max size from the service, and we know it supports 1MB. Also, 200kb is a strange number -- it used to be 256kb and then the default was boosted to 1MB, it was never 200 -- which makes me wonder if there's a config issue on the service.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Azure/azure-event-hubs-java/issues/463?email_source=notifications&email_token=ACIJPJUAKUEHSLZACXJ3DVLRADP4BA5CNFSM4KMNXBPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKFV5SQ#issuecomment-579559114, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACIJPJQ3YT76HKISMK5IPKTRADP4BANCNFSM4KMNXBPA.

JamesBirdsall commented 4 years ago

I put together a quick scanner app that homes in on the maximum size, whatever it is.

    public static void main(String[] args)
            throws InterruptedException, ExecutionException, EventHubException, IOException {
        final String eventHubConnectionStringWithEventHubName = System.getenv("REPROHUB");
        if (eventHubConnectionStringWithEventHubName == null) {
            System.out.println("Set REPROHUB");
            return;
        } else if (!eventHubConnectionStringWithEventHubName.contains("EntityPath=")) {
            System.out.println("REPROHUB does not contain event hub name");
            return;
        }

        ScheduledExecutorService ses = Executors.newScheduledThreadPool(8);
        EventHubClient ehc = EventHubClient.createFromConnectionStringSync(eventHubConnectionStringWithEventHubName, ses);

        int highestGoodScanner = 0;
        for (int increment = 100000; increment > 0; increment /= 10) {
            for (int scanner = highestGoodScanner + increment; scanner < 2000000; scanner += increment) {
                byte bodydata[] = new byte[scanner];
                EventData ed = EventData.create(bodydata);
                System.out.print("Sending " + bodydata.length + "... ");
                try {
                    ehc.sendSync(ed);
                    System.out.println("OK");
                    highestGoodScanner = scanner;
                }
                catch (PayloadSizeExceededException psee) {
                    System.out.println("TOO BIG");
                    break;
                }
            }
        }

        ehc.closeSync();
        ses.shutdown();
    }

This is the output when run against an event hub that I created for this investigation:

Sending 100000... OK Sending 200000... OK Sending 300000... OK Sending 400000... OK Sending 500000... OK Sending 600000... OK Sending 700000... OK Sending 800000... OK Sending 900000... OK Sending 1000000... OK Sending 1100000... TOO BIG Sending 1010000... OK Sending 1020000... OK Sending 1030000... OK Sending 1040000... OK Sending 1050000... TOO BIG Sending 1041000... OK Sending 1042000... OK Sending 1043000... OK Sending 1044000... OK Sending 1045000... OK Sending 1046000... OK Sending 1047000... TOO BIG Sending 1046100... OK Sending 1046200... OK Sending 1046300... OK Sending 1046400... OK Sending 1046500... OK Sending 1046600... TOO BIG Sending 1046510... OK Sending 1046520... OK Sending 1046530... TOO BIG Sending 1046521... TOO BIG

I would be interested to see the output of running against one of your namespaces. I checked the config of moesif-sb-dev and did not find anything that should change the max message size.

dgilling commented 4 years ago

It’s on the “Standard” plan which is supposed to be 1MB. Maybe because I created the Servicebus before Azure supported 1MB? Is there a way to check with Azure team, do they need to enable something on their end?

Derric

On Jan 29, 2020, at 2:29 PM, JamesBirdsall notifications@github.com wrote:

I put together a quick scanner app that homes in on the maximum size, whatever it is.

public static void main(String[] args)
        throws InterruptedException, ExecutionException, EventHubException, IOException {
    final String eventHubConnectionStringWithEventHubName = System.getenv("REPROHUB");
    if (eventHubConnectionStringWithEventHubName == null) {
        System.out.println("Set REPROHUB");
        return;
    } else if (!eventHubConnectionStringWithEventHubName.contains("EntityPath=")) {
        System.out.println("REPROHUB does not contain event hub name");
        return;
    }

    ScheduledExecutorService ses = Executors.newScheduledThreadPool(8);
    EventHubClient ehc = EventHubClient.createFromConnectionStringSync(eventHubConnectionStringWithEventHubName, ses);

    int highestGoodScanner = 0;
    for (int increment = 100000; increment > 0; increment /= 10) {
        for (int scanner = highestGoodScanner + increment; scanner < 2000000; scanner += increment) {
            byte bodydata[] = new byte[scanner];
            EventData ed = EventData.create(bodydata);
            System.out.print("Sending " + bodydata.length + "... ");
            try {
                ehc.sendSync(ed);
                System.out.println("OK");
                highestGoodScanner = scanner;
            }
            catch (PayloadSizeExceededException psee) {
                System.out.println("TOO BIG");
                break;
            }
        }
    }

    ehc.closeSync();
    ses.shutdown();
}

This is the output when run against an event hub that I created for this investigation:

Sending 100000... OK Sending 200000... OK Sending 300000... OK Sending 400000... OK Sending 500000... OK Sending 600000... OK Sending 700000... OK Sending 800000... OK Sending 900000... OK Sending 1000000... OK Sending 1100000... TOO BIG Sending 1010000... OK Sending 1020000... OK Sending 1030000... OK Sending 1040000... OK Sending 1050000... TOO BIG Sending 1041000... OK Sending 1042000... OK Sending 1043000... OK Sending 1044000... OK Sending 1045000... OK Sending 1046000... OK Sending 1047000... TOO BIG Sending 1046100... OK Sending 1046200... OK Sending 1046300... OK Sending 1046400... OK Sending 1046500... OK Sending 1046600... TOO BIG Sending 1046510... OK Sending 1046520... OK Sending 1046530... TOO BIG Sending 1046521... TOO BIG

I would be interested to see the output of running against one of your namespaces. I checked the config of moesif-sb-dev and did not find anything that should change the max message size.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Azure/azure-event-hubs-java/issues/463?email_source=notifications&email_token=ACIJPJVVC2C7DNRKVJZKPGTRAH7LXA5CNFSM4KMNXBPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKI7RPA#issuecomment-579991740, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACIJPJWBFA4LHWTL2MEGRBTRAH7LXANCNFSM4KMNXBPA.

JamesBirdsall commented 4 years ago

This SDK is maintained directly by the Azure Event Hubs service team. The maximum message size for Event Hubs is set at the cluster level and only depends on the Basic/Standard difference. The namespace you gave me is definitely Standard in our system, so it should get a message size of a megabyte, and the age of your namespace should make no difference. If it does, or you are getting the wrong maximum size for some other reason, that is something we would very much like to determine and fix.

If you could run the little scanner app and post the results, that would help us get started with the investigation. I forgot to mention, in order to run it you will need to set the environment variable REPROHUB with the connection string for an event hub, and since the scanner sends messages it should be a test event hub.

Thanks!

dgilling commented 4 years ago

I ran the provided scanner app (Also printed out exception message just for sanity). Weird that in this case it shows maximum message is 1022 kb: Do you think it's due to my app being a scala app or something else? When I saw the error max of 200kb, we were using the following dep:

libraryDependencies += "com.microsoft.azure" % "azure-eventhubs" % "3.0.2"

Sending 100000... OK Sending 200000... OK Sending 300000... OK Sending 400000... OK Sending 500000... OK Sending 600000... OK Sending 700000... OK Sending 800000... OK Sending 900000... OK Sending 1000000... OK Sending 1100000... com.microsoft.azure.eventhubs.PayloadSizeExceededException: Entity(collector): size of the payload exceeded Maximum message size: 1022 kb TOO BIG Sending 1010000... OK Sending 1020000... OK Sending 1030000... OK Sending 1040000... OK Sending 1050000... com.microsoft.azure.eventhubs.PayloadSizeExceededException: Entity(collector): size of the payload exceeded Maximum message size: 1022 kb TOO BIG Sending 1041000... OK Sending 1042000... OK Sending 1043000... OK Sending 1044000... OK Sending 1045000... OK Sending 1046000... OK Sending 1047000... com.microsoft.azure.eventhubs.PayloadSizeExceededException: Entity(collector): size of the payload exceeded Maximum message size: 1022 kb TOO BIG Sending 1046100... OK Sending 1046200... OK Sending 1046300... OK Sending 1046400... OK Sending 1046500... OK Sending 1046600... com.microsoft.azure.eventhubs.PayloadSizeExceededException: Entity(collector): size of the payload exceeded Maximum message size: 1022 kb TOO BIG Sending 1046510... OK Sending 1046520... OK Sending 1046530... com.microsoft.azure.eventhubs.PayloadSizeExceededException: Entity(collector): size of the payload exceeded Maximum message size: 1022 kb TOO BIG Sending 1046521... com.microsoft.azure.eventhubs.PayloadSizeExceededException: Entity(collector): size of the payload exceeded Maximum message size: 1022 kb TOO BIG

dgilling commented 4 years ago

I see the issue, I must have set the message size for the batch via the following:

val options = new BatchOptions() options.maxMessageSize = 200* 1024

Why would you even require setting the message size for batched messages when you poll it automatically?