search

vert-x3 / vertx-mail-client

Apache License 2.0
35 stars 34 forks source link

Sending mail with large attachement consumes lots of memory #93

Open jurajw opened 6 years ago

jurajw commented 6 years ago

Observed on 3.5.0.

Sending a mail with a single 3_000_000 bytes attachment requires ~ 27_000_000 bytes of heap. Based on my observation following is happening:

  1. An AttachmentPart instance is created for the attachment. In the constructor the buffer is base64 encoded into a ~ 4_000_000 characters long string. On JDK8 this is ~ 8_000_000 bytes in the heap.

  2. A MultiPart instance is created for the AttachmentPart. a. The AttachmentPart header is concatenated with the base64 data. b. The copy is appended into a StringBuilder. c. StringBuilder#toString is called and the result is set to MultiPart#part.

  3. MultiPart#headers is set.

  4. MultiPart#toString is called.

In 2a, 2b, 2c and 4 the base64 data is copied.

jurajw commented 6 years ago

I have been prototyping changes such that:

  1. base64 encoding is performed lazily line-by-line (by exposing an Iterator).
  2. No copies are created (iterators are concatenated).

Is there interest in a patch?

pmlopes commented 6 years ago

@jurajw I think it would be interesting to have such encoder not just for mail but for json in Vertx too. Since we support a few extra RFC for the json implementation which cover temporal types and base64 I can imagine that such encoder would reduce the memory usage there too.

Perhaps you could provide a pull request and we could continue the conversation there?

vietj commented 6 years ago

there has been contributions recently in vertx-web-client for this, perhaps some of its classes should be moved to vertx-core ?

jurajw commented 6 years ago

@vietj Can you point me to the relevant vertx-web-client changes?

vietj commented 6 years ago

see https://github.com/vert-x3/vertx-web/commits/master/vertx-web-client https://github.com/vert-x3/vertx-web/commits/master/vertx-web-client

the 10 last commits

On 3 May 2018, at 17:37, Juraj Wagner notifications@github.com wrote:

@vietj https://github.com/vietj Can you point me to the relevant vertx-web-client changes?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/vert-x3/vertx-mail-client/issues/93#issuecomment-386338367, or mute the thread https://github.com/notifications/unsubscribe-auth/AANxiiydL-D7WJJvYfpPi0WSGPJ8ioZ9ks5tuyQfgaJpZM4S_goD.