Possible memory leak in TLS(Wrap?)

[aredridel]

I'm diagnosing a problem when I run paypal's npm proxy https://github.com/krakenjs/kappa when run on iojs. After 300 requests for a 50kb tarball, memory allocated grows from 250 mb base to 450 mb.

I've run it under valgrind, and there's some things that look a bit suspicious around tlswrap, smalloc and some other crypto related bits. The interesting bits of the valgrind report are:

==31542== 827,216 bytes in 656 blocks are still reachable in loss record 1,214 of 1,223
==31542==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31542==    by 0x7450D8: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x6C868B: asn1_enc_save (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x6C2866: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x6C14FF: asn1_template_noexp_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x6C1BAB: asn1_template_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x6C2F0F: ASN1_item_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x67CC9E: ssl3_get_server_certificate (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x681323: ssl3_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x67958B: ssl23_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x679E1A: ssl23_write (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xCDBCEF: _ZN4node7TLSWrap7ClearInEv.part.43 (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542== 
==31542== 1,343,488 bytes in 328 blocks are still reachable in loss record 1,215 of 1,223
==31542==    at 0x4C2AC27: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31542==    by 0xCD825D: node::NodeBIO::PeekWritable(unsigned long*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xCD8EEC: node::TLSWrap::OnAllocImpl(unsigned long, uv_buf_t*, void*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xD031D0: uv__read (stream.c:1089)
==31542==    by 0xD03967: uv__stream_io (stream.c:1219)
==31542==    by 0xD08CEC: uv__io_poll (linux-core.c:319)
==31542==    by 0xCFA955: uv_run (core.c:324)
==31542==    by 0xC87650: node::Start(int, char**) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x5C8E76C: (below main) (libc-start.c:226)
==31542== 
==31542== 2,303,504 bytes in 131 blocks are still reachable in loss record 1,216 of 1,223
==31542==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31542==    by 0x7450D8: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x67BBAF: ssl3_setup_write_buffer (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x67BCB1: ssl3_setup_buffers (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x679000: ssl23_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x679CBA: ssl23_read (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xCDCE78: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xCDD2B4: node::TLSWrap::Start(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x801781: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x828E02: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x1C66A79060BA: ???
==31542==    by 0x1C66A7CDC7AD: ???
==31542== 
==31542== 2,323,416 bytes in 131 blocks are still reachable in loss record 1,217 of 1,223
==31542==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31542==    by 0x7450D8: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x67BD5F: ssl3_setup_buffers (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x679000: ssl23_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x679CBA: ssl23_read (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xCDCE78: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xCDD2B4: node::TLSWrap::Start(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x801781: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x828E02: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x1C66A79060BA: ???
==31542==    by 0x1C66A7CDC7AD: ???
==31542==    by 0x1C66A7CDC562: ???
==31542== 
==31542== 3,464,048 bytes in 197 blocks are still reachable in loss record 1,218 of 1,223
==31542==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31542==    by 0x7450D8: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x67BBAF: ssl3_setup_write_buffer (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x67BCB1: ssl3_setup_buffers (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x679000: ssl23_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x679CBA: ssl23_read (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xCDCE78: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xCDD2B4: node::TLSWrap::Start(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x1C66A7A43345: ???
==31542==    by 0x1C66A7EA169A: ???
==31542==    by 0x1C66A7CDC562: ???
==31542==    by 0x1C66A7924BC5: ???
==31542== 
==31542== 3,493,992 bytes in 197 blocks are still reachable in loss record 1,219 of 1,223
==31542==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31542==    by 0x7450D8: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x67BD5F: ssl3_setup_buffers (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x679000: ssl23_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x679CBA: ssl23_read (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xCDCE78: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xCDD2B4: node::TLSWrap::Start(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x1C66A7A43345: ???
==31542==    by 0x1C66A7EA169A: ???
==31542==    by 0x1C66A7CDC562: ???
==31542==    by 0x1C66A7924BC5: ???
==31542==    by 0x1C66A7C08177: ???
==31542== 
==31542== 5,373,952 bytes in 328 blocks are still reachable in loss record 1,220 of 1,223
==31542==    at 0x4C2AC27: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31542==    by 0xCD8325: node::NodeBIO::Commit(unsigned long) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xCDD745: node::TLSWrap::OnReadImpl(long, uv_buf_t const*, uv_handle_type, void*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xD0335E: uv__read (stream.c:1156)
==31542==    by 0xD03967: uv__stream_io (stream.c:1219)
==31542==    by 0xD08CEC: uv__io_poll (linux-core.c:319)
==31542==    by 0xCFA955: uv_run (core.c:324)
==31542==    by 0xC87650: node::Start(int, char**) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x5C8E76C: (below main) (libc-start.c:226)
==31542== 
==31542== 7,422,722 bytes in 131 blocks are still reachable in loss record 1,221 of 1,223
==31542==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31542==    by 0xCAD6A2: node::smalloc::Alloc(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x801781: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x828E02: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x1C66A79060BA: ???
==31542==    by 0x1C66A79B20B8: ???
==31542==    by 0x1C66A791EA54: ???
==31542==    by 0x1C66A791EDA0: ???
==31542==    by 0x1C66A7D088BE: ???
==31542==    by 0x1C66A7D084D4: ???
==31542==    by 0x1C66A7D08272: ???
==31542==    by 0x1C66A7924BC5: ???
==31542== 
==31542== 11,162,414 bytes in 197 blocks are still reachable in loss record 1,222 of 1,223
==31542==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31542==    by 0xCAD6A2: node::smalloc::Alloc(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x801781: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x828E02: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x1C66A79060BA: ???
==31542==    by 0x1C66A79B20B8: ???
==31542==    by 0x1C66A791EA54: ???
==31542==    by 0x1C66A791EDA0: ???
==31542==    by 0x1C66A7EA5607: ???
==31542==    by 0x1C66A7D08272: ???
==31542==    by 0x1C66A7924BC5: ???
==31542==    by 0x1C66A7C08177: ???
==31542== 
==31542== 18,591,935 bytes in 20,677 blocks are still reachable in loss record 1,223 of 1,223
==31542==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==31542==    by 0xCD8E91: node::TLSWrap::OnAllocSelf(unsigned long, uv_buf_t*, void*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xCDCEA0: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xCDD7B7: node::TLSWrap::OnReadImpl(long, uv_buf_t const*, uv_handle_type, void*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0xD0335E: uv__read (stream.c:1156)
==31542==    by 0xD03967: uv__stream_io (stream.c:1219)
==31542==    by 0xD08CEC: uv__io_poll (linux-core.c:319)
==31542==    by 0xCFA955: uv_run (core.c:324)
==31542==    by 0xC87650: node::Start(int, char**) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==31542==    by 0x5C8E76C: (below main) (libc-start.c:226)
==31542== 
==31542== LEAK SUMMARY:
==31542==    definitely lost: 10,464 bytes in 327 blocks
==31542==    indirectly lost: 63,532 bytes in 3,258 blocks
==31542==      possibly lost: 4,568 bytes in 64 blocks
==31542==    still reachable: 69,011,541 bytes in 206,575 blocks
==31542==         suppressed: 0 bytes in 0 blocks
==31542== 
==31542== For counts of detected and suppressed errors, rerun with: -v
==31542== ERROR SUMMARY: 42 errors from 42 contexts (suppressed: 2 from 2)

Any help or suggestions for what to look at to track this down would be wonderful.

Node 0.12 and 0.10 don't show this behavior with the same code (we're stress-testing io.js in parallel to shake stuff like this loose)

[rvagg]

ping @trevnorris @indutny @bnoordhuis

[bnoordhuis]

I wonder if there is a handle leak. I noticed that TLSWrap::OnReadSelf() creates a JS buffer object without a HandleScope. Does this patch make a difference?

diff --git a/src/tls_wrap.cc b/src/tls_wrap.cc
index 1647ab6..2b00d9e 100644
--- a/src/tls_wrap.cc
+++ b/src/tls_wrap.cc
@@ -641,6 +641,7 @@ void TLSWrap::OnReadSelf(ssize_t nread,
                          uv_handle_type pending,
                          void* ctx) {
   TLSWrap* wrap = static_cast<TLSWrap*>(ctx);
+  HandleScope handle_scope(wrap->env()->isolate());
   Local<Object> buf_obj;
   if (buf != nullptr)
     buf_obj = Buffer::Use(wrap->env(), buf->base, buf->len);

[jasisk]

Providing a tiny bit of additional context: this is a graph of memory usage across 6 instances of our application from a few weeks back (4x node v0.10.3?, 1x node v0.12.0 in orange, 1x iojs v1.3.0 in red). Continuing to see the same behavior in 1.4.x.

The application is a very simple proxy, seeing highs of ~350 reqs/sec though more consistently ~80-100. Each incoming request results in 1 to 4 outbound requests.

[indutny]

Looking.

[indutny]

It seems to be leaking TLSWrap instances, wtf.

[indutny]

Partial fix here: https://github.com/iojs/io.js/pull/1078

[GeoffreyPlitt]

I've got a test here that reproduces the problem, comparing HTTP and HTTPS on 0.10, 0.12, and iojs.

https://github.com/GeoffreyPlitt/iojs_tls_bug

I'm going to try the partial fix above and see if it fixes my tests.

[indutny]

It seems like these lines are partly responsible for this problem: https://github.com/iojs/io.js/blob/b27931b0fedc5fad638d637525aba84b2754ed5f/lib/_tls_wrap.js#L290-L305

[indutny]

Found a JSStream leak, but so far don't have much progress on the main issue. It seems that cross-reference between StreamWrap and TLSWrap creates some non-collectible cycle, which should not be really possible. Most likely some bug in my code, but I can't say what exactly it is yet. Will continue tomorrow.

[Qard]

Any particular reason that existence check/apply in the proxied methods runs on every call? Can it not be:

// Proxy HandleWrap, PipeWrap and TCPWrap methods
proxiedMethods.forEach(function(name) {
  res[name] = (handle[name] || function () {}).bind(handle);
});

[indutny]

@Qard no reason, all of these methods are usually called once per handle lifetime.

[indutny]

Looks like my patch is fixing the issue.

[indutny]

Should be fixed now, please confirm! ;)

[gm112]

This bug has been causing me a lot of issues. I'll certainly build with this change and let you know how goes the memory usage.

[rvagg]

https://iojs.org/download/nightly/v1.4.4-nightly20150306dee07e2983/

Nightly with this change is there, please test, would love to have some feedback by the time 1.5.0 goes out later today.

[jasisk]

Deployed the nightly. Will beat on it for a couple of hours and report back.

(fwiw, I know @aredridel reran her benchmark and saw the issue resolved as well.)

[gm112]

I can confirm that for me, my memory usage is certainly much more stable. Before I would climb to several hundred megs with no problems.

I look forward to there being a full blown release.

[jasisk]

Really early observations on the 1.4.4-nightly suggest there may be more. @aredridel is profiling it now.

blue: 0.10, purple: 0.12, green: 1.4.4-nightly

[aredridel]

Without valgrind, it seems to have leveled things off somewhat.

However, it seems to grow still once I take a deeper look. Here's the larger chunks of valgrind's output:

==27619== 327,760 bytes in 482 blocks are still reachable in loss record 1,244 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0x6F6004: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x6E8E9E: EVP_CipherInit_ex (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x6608F6: tls1_change_cipher_state (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x642A6A: ssl3_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63B52F: ssl23_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63BD81: ssl23_write (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0D1CF: node::TLSWrap::ClearIn() (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0D727: node::TLSWrap::DoRead(long, uv_buf_t const*, uv_handle_type) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xBECBDA: node::StreamWrap::OnRead(uv_stream_s*, long, uv_buf_t const*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC3235D: uv__read (stream.c:1156)
==27619==    by 0xC32E8A: uv__stream_io (stream.c:1219)
==27619== 
==27619== 332,520 bytes in 489 blocks are still reachable in loss record 1,245 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0x6F6004: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x6E8E9E: EVP_CipherInit_ex (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x6608F6: tls1_change_cipher_state (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x648150: ssl3_do_change_cipher_spec (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x648F5E: ssl3_read_bytes (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63D657: ssl3_get_message (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63CFDC: ssl3_get_finished (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x642CAA: ssl3_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x64807F: ssl3_write_bytes (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0D1CF: node::TLSWrap::ClearIn() (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0D727: node::TLSWrap::DoRead(long, uv_buf_t const*, uv_handle_type) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619== 
==27619== 356,000 bytes in 8,900 blocks are still reachable in loss record 1,246 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0x6F6004: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x66BAC8: c2i_ASN1_OBJECT (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67C7FA: asn1_ex_c2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67CBE4: asn1_d2i_ex_primitive (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67DA2B: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67DE2E: asn1_template_noexp_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67E063: asn1_template_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67D4E4: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67DCED: asn1_template_noexp_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67DFD5: asn1_template_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67D4E4: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619== 
==27619== 366,592 bytes in 358 blocks are still reachable in loss record 1,247 of 1,265
==27619==    at 0x4C2AC27: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0xC0A7C3: node::NodeBIO::Write(char const*, unsigned long) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0A033: node::NodeBIO::Write(bio_st*, char const*, int) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x689237: BIO_write (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63BE9C: ssl23_write_bytes (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63AF21: ssl23_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63BC21: ssl23_read (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0CAFD: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0CA1E: node::TLSWrap::Start(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x3E39A4743605: ???
==27619==    by 0x3E39A4B880FE: ???
==27619==    by 0x3E39A4A3DE02: ???
==27619== 
==27619== 379,640 bytes in 2,264 blocks are still reachable in loss record 1,248 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0x6F6004: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x6A3501: BUF_MEM_grow (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x682B03: x509_name_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67D1CB: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67DE2E: asn1_template_noexp_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67E063: asn1_template_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67D4E4: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67DE2E: asn1_template_noexp_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67E063: asn1_template_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67D4E4: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67E193: ASN1_item_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619== 
==27619== 418,760 bytes in 10,469 blocks are still reachable in loss record 1,249 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0x6F6004: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x66BAC8: c2i_ASN1_OBJECT (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67C7FA: asn1_ex_c2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67CBE4: asn1_d2i_ex_primitive (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67DA2B: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67DE2E: asn1_template_noexp_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67E063: asn1_template_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67D4E4: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67DCED: asn1_template_noexp_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67E063: asn1_template_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67D92E: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619== 
==27619== 419,840 bytes in 410 blocks are still reachable in loss record 1,250 of 1,265
==27619==    at 0x4C2AC27: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0xC0A7C3: node::NodeBIO::Write(char const*, unsigned long) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0D652: node::TLSWrap::DoWrite(node::WriteWrap*, uv_buf_t*, unsigned long, uv_stream_s*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xBEBADC: int node::StreamBase::WriteString<(node::encoding)4>(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0E9A4: void node::StreamBase::JSMethod<node::TLSWrap, &(int node::StreamBase::WriteString<(node::encoding)4>(v8::FunctionCallbackInfo<v8::Value> const&))>(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x7A0145: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x7C7F47: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x3E39A46060BA: ???
==27619==    by 0x3E39A4B343B0: ???
==27619==    by 0x3E39A490E90E: ???
==27619==    by 0x3E39A49DE737: ???
==27619==    by 0x3E39A4624BC5: ???
==27619== 
==27619== 471,600 bytes in 393 blocks are still reachable in loss record 1,251 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0x6F6004: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x64539F: ssl3_new (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x662200: tls1_new (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x65735A: SSL_new (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0B472: node::TLSWrap::TLSWrap(node::Environment*, node::crypto::SSLWrap<node::TLSWrap>::Kind, node::StreamBase*, node::crypto::SecureContext*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0C59F: node::TLSWrap::Wrap(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x7A0145: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x7C7F47: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x3E39A46060BA: ???
==27619==    by 0x3E39A49F0239: ???
==27619==    by 0x3E39A49EFD6A: ???
==27619== 
==27619== 524,280 bytes in 1 blocks are still reachable in loss record 1,252 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0x77C42A: v8::internal::Malloced::New(unsigned long) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x8ED0A3: v8::internal::HeapProfiler::DefineWrapperClass(unsigned short, v8::RetainedObjectInfo* (*)(unsigned short, v8::Handle<v8::Value>)) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xBC7A13: node::Binding(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x7A0145: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x7C7F47: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x3E39A46060BA: ???
==27619==    by 0x3E39A4670CC4: ???
==27619==    by 0x3E39A461EA54: ???
==27619==    by 0x3E39A466B52D: ???
==27619==    by 0x3E39A466AD78: ???
==27619==    by 0x3E39A466E272: ???
==27619== 
==27619== 559,838 bytes in 8,900 blocks are still reachable in loss record 1,253 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0x6F6004: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x671A3C: ASN1_STRING_set (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67C991: asn1_ex_c2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67CBE4: asn1_d2i_ex_primitive (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67DA2B: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67DE2E: asn1_template_noexp_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67E063: asn1_template_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67D4E4: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67DCED: asn1_template_noexp_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67DFD5: asn1_template_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67D4E4: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619== 
==27619== 739,392 bytes in 30,808 blocks are still reachable in loss record 1,254 of 1,265
==27619==    at 0x4C2B1C7: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0xBE4BB5: node::smalloc::Alloc(node::Environment*, v8::Handle<v8::Object>, char*, unsigned long, v8::ExternalArrayType) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xBCC4EA: node::Buffer::Use(node::Environment*, char*, unsigned int) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0B8D8: node::TLSWrap::OnReadSelf(long, uv_buf_t const*, uv_handle_type, void*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0CBAD: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0D72F: node::TLSWrap::DoRead(long, uv_buf_t const*, uv_handle_type) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xBECBDA: node::StreamWrap::OnRead(uv_stream_s*, long, uv_buf_t const*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC3235D: uv__read (stream.c:1156)
==27619==    by 0xC32E8A: uv__stream_io (stream.c:1219)
==27619==    by 0xC37F01: uv__io_poll (linux-core.c:319)
==27619==    by 0xC29D09: uv_run (core.c:324)
==27619==    by 0xBC9220: node::Start(int, char**) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619== 
==27619== 938,448 bytes in 114 blocks are still reachable in loss record 1,255 of 1,265
==27619==    at 0x4C2B1C7: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0x8E9506: v8::internal::GlobalHandles::Create(v8::internal::Object*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xBE4BCA: node::smalloc::Alloc(node::Environment*, v8::Handle<v8::Object>, char*, unsigned long, v8::ExternalArrayType) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xBCC4EA: node::Buffer::Use(node::Environment*, char*, unsigned int) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0B8D8: node::TLSWrap::OnReadSelf(long, uv_buf_t const*, uv_handle_type, void*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0CBAD: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0D72F: node::TLSWrap::DoRead(long, uv_buf_t const*, uv_handle_type) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xBECBDA: node::StreamWrap::OnRead(uv_stream_s*, long, uv_buf_t const*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC3235D: uv__read (stream.c:1156)
==27619==    by 0xC32E8A: uv__stream_io (stream.c:1219)
==27619==    by 0xC37F01: uv__io_poll (linux-core.c:319)
==27619==    by 0xC29D09: uv_run (core.c:324)
==27619== 
==27619== 1,215,604 bytes in 964 blocks are still reachable in loss record 1,256 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0x6F6004: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x680F2F: asn1_enc_save (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67DAD1: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67DE2E: asn1_template_noexp_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67E063: asn1_template_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67D4E4: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x67E193: ASN1_item_d2i (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63E927: ssl3_get_server_certificate (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x642EF1: ssl3_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63B52F: ssl23_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63BD81: ssl23_write (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619== 
==27619== 2,002,944 bytes in 489 blocks are still reachable in loss record 1,257 of 1,265
==27619==    at 0x4C2AC27: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0xC0AD91: node::NodeBIO::PeekWritable(unsigned long*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0B829: node::TLSWrap::OnAllocImpl(unsigned long, uv_buf_t*, void*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xBECB0C: node::StreamWrap::OnAlloc(uv_handle_s*, unsigned long, uv_buf_t*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC320CA: uv__read (stream.c:1089)
==27619==    by 0xC32E8A: uv__stream_io (stream.c:1219)
==27619==    by 0xC37F01: uv__io_poll (linux-core.c:319)
==27619==    by 0xC29D09: uv_run (core.c:324)
==27619==    by 0xBC9220: node::Start(int, char**) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x5C8E76C: (below main) (libc-start.c:226)
==27619== 
==27619== 2,303,504 bytes in 131 blocks are still reachable in loss record 1,258 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0x6F6004: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63DB07: ssl3_setup_write_buffer (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63DB6E: ssl3_setup_buffers (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63B02B: ssl23_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63BC21: ssl23_read (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0CAFD: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0CA1E: node::TLSWrap::Start(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x7A0145: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x7C7F47: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x3E39A46060BA: ???
==27619==    by 0x3E39A4A3E0A7: ???
==27619== 
==27619== 2,323,416 bytes in 131 blocks are still reachable in loss record 1,259 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0x6F6004: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63D9DD: ssl3_setup_read_buffer (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63DB60: ssl3_setup_buffers (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63B02B: ssl23_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63BC21: ssl23_read (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0CAFD: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0CA1E: node::TLSWrap::Start(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x7A0145: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x7C7F47: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x3E39A46060BA: ???
==27619==    by 0x3E39A4A3E0A7: ???
==27619== 
==27619== 6,295,072 bytes in 358 blocks are still reachable in loss record 1,260 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0x6F6004: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63DB07: ssl3_setup_write_buffer (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63DB6E: ssl3_setup_buffers (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63B02B: ssl23_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63BC21: ssl23_read (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0CAFD: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0CA1E: node::TLSWrap::Start(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x3E39A4743605: ???
==27619==    by 0x3E39A4B880FE: ???
==27619==    by 0x3E39A4A3DE02: ???
==27619==    by 0x3E39A4624BC5: ???
==27619== 
==27619== 6,349,488 bytes in 358 blocks are still reachable in loss record 1,261 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0x6F6004: CRYPTO_malloc (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63D9DD: ssl3_setup_read_buffer (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63DB60: ssl3_setup_buffers (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63B02B: ssl23_connect (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x63BC21: ssl23_read (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0CAFD: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0CA1E: node::TLSWrap::Start(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x3E39A4743605: ???
==27619==    by 0x3E39A4B880FE: ???
==27619==    by 0x3E39A4A3DE02: ???
==27619==    by 0x3E39A4624BC5: ???
==27619== 
==27619== 7,422,722 bytes in 131 blocks are still reachable in loss record 1,262 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0xBE4AA8: node::smalloc::Alloc(node::Environment*, v8::Handle<v8::Object>, unsigned long, v8::ExternalArrayType) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xBE49D3: node::smalloc::Alloc(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x7A0145: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x7C7F47: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x3E39A46060BA: ???
==27619==    by 0x3E39A46B2198: ???
==27619==    by 0x3E39A461EA54: ???
==27619==    by 0x3E39A461EDA0: ???
==27619==    by 0x3E39A4A54D9E: ???
==27619==    by 0x3E39A4A549B4: ???
==27619==    by 0x3E39A4A54752: ???
==27619== 
==27619== 8,011,776 bytes in 489 blocks are still reachable in loss record 1,263 of 1,265
==27619==    at 0x4C2AC27: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0xC0AE7A: node::NodeBIO::Commit(unsigned long) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0D6FE: node::TLSWrap::DoRead(long, uv_buf_t const*, uv_handle_type) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xBECBDA: node::StreamWrap::OnRead(uv_stream_s*, long, uv_buf_t const*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC3235D: uv__read (stream.c:1156)
==27619==    by 0xC32E8A: uv__stream_io (stream.c:1219)
==27619==    by 0xC37F01: uv__io_poll (linux-core.c:319)
==27619==    by 0xC29D09: uv_run (core.c:324)
==27619==    by 0xBC9220: node::Start(int, char**) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x5C8E76C: (below main) (libc-start.c:226)
==27619== 
==27619== 20,284,996 bytes in 358 blocks are still reachable in loss record 1,264 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0xBE4AA8: node::smalloc::Alloc(node::Environment*, v8::Handle<v8::Object>, unsigned long, v8::ExternalArrayType) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xBE49D3: node::smalloc::Alloc(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x7A0145: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x7C7F47: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x3E39A46060BA: ???
==27619==    by 0x3E39A46B2198: ???
==27619==    by 0x3E39A461EA54: ???
==27619==    by 0x3E39A461EDA0: ???
==27619==    by 0x3E39A4B8B0C7: ???
==27619==    by 0x3E39A4A54752: ???
==27619==    by 0x3E39A4624BC5: ???
==27619== 
==27619== 27,708,244 bytes in 30,808 blocks are still reachable in loss record 1,265 of 1,265
==27619==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27619==    by 0xC0B881: node::TLSWrap::OnAllocSelf(unsigned long, uv_buf_t*, void*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0CB69: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC0D72F: node::TLSWrap::DoRead(long, uv_buf_t const*, uv_handle_type) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xBECBDA: node::StreamWrap::OnRead(uv_stream_s*, long, uv_buf_t const*) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0xC3235D: uv__read (stream.c:1156)
==27619==    by 0xC32E8A: uv__stream_io (stream.c:1219)
==27619==    by 0xC37F01: uv__io_poll (linux-core.c:319)
==27619==    by 0xC29D09: uv_run (core.c:324)
==27619==    by 0xBC9220: node::Start(int, char**) (in /home/aredridel/t/node_modules/iojs-bin/node_modules/iojs-linux-x64/iojs-v1.4.2-linux-x64/bin/iojs)
==27619==    by 0x5C8E76C: (below main) (libc-start.c:226)
==27619== 
==27619== LEAK SUMMARY:
==27619==    definitely lost: 15,616 bytes in 488 blocks
==27619==    indirectly lost: 94,945 bytes in 4,868 blocks
==27619==      possibly lost: 4,369 bytes in 54 blocks
==27619==    still reachable: 101,753,796 bytes in 297,899 blocks
==27619==         suppressed: 0 bytes in 0 blocks
==27619== 
==27619== For counts of detected and suppressed errors, rerun with: -v
==27619== ERROR SUMMARY: 42 errors from 42 contexts (suppressed: 2 from 2)

[aredridel]

(ignore the v1.4.2 in the paths -- the actual binary being run is compiled from the pr/1078 branch)

[indutny]

@aredridel this is bad news, as it could be related to https://code.google.com/p/v8/issues/detail?id=3949&thanks=3949&ts=1425654431 which I was experiencing with JSStream during debugging.

Being not able to reproduce this issue myself, I've pushed suggested patch to the: https://github.com/indutny/io.js/tree/test/paypal-leak . May I ask you to give it a try?

Thanks a lot!

[bnoordhuis]

@aredridel It's a crude hack but can you try this patch? If it keeps running, great. If it aborts, there's still a handle leak somewhere.

[aredridel]

Building now!

The actual instrumentation I'm using is the kappa package, configured to talk to a private (empty) npm registry over http, then to https://registry.npmjs.org/, then logging via good. The bulk of the work is that proxy, try each, moving on if there's a 404. I just tested with ab -c 10 -n 1000 on it. Pretty simple overall, doing https as a client from that hapi proxy.

[indutny]

@bnoordhuis I think it could be v8 just not keeping up with it's partial GCs. Investigating.

[indutny]

@aredridel pushed one more patch to that branch for you, please test ;)

@bnoordhuis what do you think about https://github.com/indutny/io.js/commit/8ef701cb13ba904453d08740c5d8d7f69e2128a9 ?

[indutny]

@aredridel also, 1000 requests might not suffice. It doesn't seem like v8 is collecting any TLSWrap instances after 1000 connections, the whole thing happens much later.

[aredridel]

With the assertions: keeps running. So there's that. Trying the next patch on the branch.

[aredridel]

And without valgrind, that revision indutny@8ef701cb13b seems to level off after 1200 or so requests. I'm gonna run that under valgrind and see if we've got any improvement there.

[aredridel]

With valgrind, it levels off at about the same place -- 1200 requests -- and the heap used after exit report from it is much smaller (13mb, not 100mb). The largest chunks left allocated and reachable, too, are smaller -- 2mb or so.

==575== 196,608 bytes in 48 blocks are still reachable in loss record 1,225 of 1,236
==575==    at 0x4C2AC27: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==575==    by 0xC0AEF7: node::NodeBIO::TryAllocateForWrite(unsigned long) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0AF84: node::NodeBIO::PeekWritable(unsigned long*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0B9B9: node::TLSWrap::OnAllocImpl(unsigned long, uv_buf_t*, void*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xBECCAC: node::StreamWrap::OnAlloc(uv_handle_s*, unsigned long, uv_buf_t*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC322CA: uv__read (stream.c:1089)
==575==    by 0xC3308A: uv__stream_io (stream.c:1219)
==575==    by 0xC38101: uv__io_poll (linux-core.c:319)
==575==    by 0xC29F09: uv_run (core.c:324)
==575==    by 0xBC9280: node::Start(int, char**) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x5C8E76C: (below main) (libc-start.c:226)
==575== 
==575== 198,912 bytes in 777 blocks are still reachable in loss record 1,226 of 1,236
==575==    at 0x4C2AC27: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==575==    by 0xC0D04A: node::TLSWrap::EncOut() (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0DA9B: non-virtual thunk to node::TLSWrap::DoShutdown(node::ShutdownWrap*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xBE95AE: node::StreamBase::Shutdown(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0E724: void node::StreamBase::JSMethod<node::TLSWrap, &(node::StreamBase::Shutdown(v8::FunctionCallbackInfo<v8::Value> const&))>(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x7A0145: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x7C7F47: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x2B95418060BA: ???
==575==    by 0x2B9541DCB0B3: ???
==575==    by 0x2B9541E5953B: ???
==575==    by 0x2B9541E20F4B: ???
==575==    by 0x2B9541E594D7: ???
==575== 
==575== 199,168 bytes in 778 blocks are still reachable in loss record 1,227 of 1,236
==575==    at 0x4C2AC27: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==575==    by 0xC0D04A: node::TLSWrap::EncOut() (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0CC16: node::TLSWrap::Start(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x7A0145: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x7C7F47: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x2B95418060BA: ???
==575==    by 0x2B9541D935EF: ???
==575==    by 0x2B9541C33F62: ???
==575==    by 0x2B9541824BC5: ???
==575==    by 0x2B9541B0ADB7: ???
==575==    by 0x2B9541E594D7: ???
==575==    by 0x2B9541DC3B2D: ???
==575== 
==575== 306,708 (44,000 direct, 262,708 indirect) bytes in 1,375 blocks are definitely lost in loss record 1,228 of 1,236
==575==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==575==    by 0x6F6004: CRYPTO_malloc (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x7294AC: sk_new_null (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x67DECD: asn1_template_noexp_d2i (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x67E063: asn1_template_ex_d2i (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x67D92E: ASN1_item_ex_d2i (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x67E193: ASN1_item_d2i (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC083E9: node::crypto::X509ToObject(node::Environment*, x509_st*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC03ECB: node::crypto::SSLWrap<node::TLSWrap>::GetPeerCertificate(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x7A0145: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x7C7F47: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x2B95418060BA: ???
==575== 
==575== 524,280 bytes in 1 blocks are still reachable in loss record 1,229 of 1,236
==575==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==575==    by 0x77C42A: v8::internal::Malloced::New(unsigned long) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x8ED0A3: v8::internal::HeapProfiler::DefineWrapperClass(unsigned short, v8::RetainedObjectInfo* (*)(unsigned short, v8::Handle<v8::Value>)) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xBC7A83: node::Binding(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x7A0145: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x7C7F47: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x2B95418060BA: ???
==575==    by 0x2B954187108C: ???
==575==    by 0x2B954181EA54: ???
==575==    by 0x2B954186B52D: ???
==575==    by 0x2B954186AD78: ???
==575==    by 0x2B954186E272: ???
==575== 
==575== 710,400 bytes in 2,775 blocks are still reachable in loss record 1,230 of 1,236
==575==    at 0x4C2AC27: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==575==    by 0xC0D04A: node::TLSWrap::EncOut() (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0D927: node::TLSWrap::DoRead(long, uv_buf_t const*, uv_handle_type) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xBECD7A: node::StreamWrap::OnRead(uv_stream_s*, long, uv_buf_t const*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC3255D: uv__read (stream.c:1156)
==575==    by 0xC3308A: uv__stream_io (stream.c:1219)
==575==    by 0xC38101: uv__io_poll (linux-core.c:319)
==575==    by 0xC29F09: uv_run (core.c:324)
==575==    by 0xBC9280: node::Start(int, char**) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x5C8E76C: (below main) (libc-start.c:226)
==575== 
==575== 737,280 bytes in 45 blocks are still reachable in loss record 1,231 of 1,236
==575==    at 0x4C2AC27: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==575==    by 0xC0AEF7: node::NodeBIO::TryAllocateForWrite(unsigned long) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0AFEB: node::NodeBIO::Commit(unsigned long) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0D8EE: node::TLSWrap::DoRead(long, uv_buf_t const*, uv_handle_type) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xBECD7A: node::StreamWrap::OnRead(uv_stream_s*, long, uv_buf_t const*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC3255D: uv__read (stream.c:1156)
==575==    by 0xC3308A: uv__stream_io (stream.c:1219)
==575==    by 0xC38101: uv__io_poll (linux-core.c:319)
==575==    by 0xC29F09: uv_run (core.c:324)
==575==    by 0xBC9280: node::Start(int, char**) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x5C8E76C: (below main) (libc-start.c:226)
==575== 
==575== 799,036 bytes in 893 blocks are still reachable in loss record 1,232 of 1,236
==575==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==575==    by 0xC0BA11: node::TLSWrap::OnAllocSelf(unsigned long, uv_buf_t*, void*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0CD59: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0D91F: node::TLSWrap::DoRead(long, uv_buf_t const*, uv_handle_type) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xBECD7A: node::StreamWrap::OnRead(uv_stream_s*, long, uv_buf_t const*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC3255D: uv__read (stream.c:1156)
==575==    by 0xC3308A: uv__stream_io (stream.c:1219)
==575==    by 0xC38101: uv__io_poll (linux-core.c:319)
==575==    by 0xC29F09: uv_run (core.c:324)
==575==    by 0xBC9280: node::Start(int, char**) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x5C8E76C: (below main) (libc-start.c:226)
==575== 
==575== 879,200 bytes in 50 blocks are still reachable in loss record 1,233 of 1,236
==575==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==575==    by 0x6F6004: CRYPTO_malloc (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x63DB07: ssl3_setup_write_buffer (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x63DB6E: ssl3_setup_buffers (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x63B02B: ssl23_connect (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x63BC21: ssl23_read (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0CCED: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0CC0E: node::TLSWrap::Start(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x7A0145: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x7C7F47: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x2B95418060BA: ???
==575==    by 0x2B9541D935EF: ???
==575== 
==575== 886,800 bytes in 50 blocks are still reachable in loss record 1,234 of 1,236
==575==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==575==    by 0x6F6004: CRYPTO_malloc (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x63D9DD: ssl3_setup_read_buffer (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x63DB60: ssl3_setup_buffers (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x63B02B: ssl23_connect (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x63BC21: ssl23_read (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0CCED: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0CC0E: node::TLSWrap::Start(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x7A0145: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x7C7F47: v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x2B95418060BA: ???
==575==    by 0x2B9541D935EF: ???
==575== 
==575== 1,613,472 bytes in 196 blocks are still reachable in loss record 1,235 of 1,236
==575==    at 0x4C2B1C7: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==575==    by 0x8E9506: v8::internal::GlobalHandles::Create(v8::internal::Object*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xBE4CEA: node::smalloc::Alloc(node::Environment*, v8::Handle<v8::Object>, char*, unsigned long, v8::ExternalArrayType) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xBCC55A: node::Buffer::Use(node::Environment*, char*, unsigned int) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0BA68: node::TLSWrap::OnReadSelf(long, uv_buf_t const*, uv_handle_type, void*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0CD9D: node::TLSWrap::ClearOut() (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC0D91F: node::TLSWrap::DoRead(long, uv_buf_t const*, uv_handle_type) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xBECD7A: node::StreamWrap::OnRead(uv_stream_s*, long, uv_buf_t const*) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xC3255D: uv__read (stream.c:1156)
==575==    by 0xC3308A: uv__stream_io (stream.c:1219)
==575==    by 0xC38101: uv__io_poll (linux-core.c:319)
==575==    by 0xC29F09: uv_run (core.c:324)
==575== 
==575== 2,436,466 bytes in 43 blocks are still reachable in loss record 1,236 of 1,236
==575==    at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==575==    by 0xBE4BC8: node::smalloc::Alloc(node::Environment*, v8::Handle<v8::Object>, unsigned long, v8::ExternalArrayType) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0xBE4AF3: node::smalloc::Alloc(v8::FunctionCallbackInfo<v8::Value> const&) (in /home/aredridel/t/node_modules/.bin/iojs)
==575==    by 0x2B95419A75E5: ???
==575==    by 0x2B9541D96BB5: ???
==575==    by 0x2B9541C4A772: ???
==575==    by 0x2B9541824BC5: ???
==575==    by 0x2B9541B0ADB7: ???
==575==    by 0x2B9541E594D7: ???
==575==    by 0x2B9541D29DD1: ???
==575==    by 0x2B954181EA54: ???
==575==    by 0x2B9541C49AD9: ???
==575== 
==575== LEAK SUMMARY:
==575==    definitely lost: 44,000 bytes in 1,375 blocks
==575==    indirectly lost: 262,708 bytes in 13,461 blocks
==575==      possibly lost: 5,528 bytes in 110 blocks
==575==    still reachable: 13,063,533 bytes in 49,091 blocks
==575==         suppressed: 0 bytes in 0 blocks
==575== 
==575== For counts of detected and suppressed errors, rerun with: -v
==575== ERROR SUMMARY: 42 errors from 42 contexts (suppressed: 2 from 2)

[indutny]

I guess this is a good news, right? :)

[jasisk]

I deployed indutny/iojs@8ef701cb13ba904453d08740c5d8d7f69e2128a9 but our traffic is ramping down a bit (Friday afternoon). Similar to @aredridel, initial observations are good but I'm driving more traffic to it to be sure.

[jasisk]

I may have spoken too soon. It's more subtle (could be a result of traffic slowing), but still seems to be there (red: 1.4.4, blue: 0.10, orange 0.12):

[indutny]

@jasisk may I ask you to keep it running for some time just to make sure that it does not stabilize over longer time?

[jasisk]

You got it. :) Will update the thread when a trend appears.

[gm112]

Hey is anyone in this thread observing memory growth when there's a higher volume of socket hangups as well? (http/s)

[indutny]

Oh, this is interesting. @gm112 how can I reproduce it?

[indutny]

@jasisk @aredridel one more patch for you:

diff --git a/src/node_crypto.cc b/src/node_crypto.cc
index 9123207..a650d4f 100644
--- a/src/node_crypto.cc
+++ b/src/node_crypto.cc
@@ -1132,6 +1132,7 @@ static bool SafeX509ExtPrint(BIO* out, X509_EXTENSION* ext) {
       X509V3_EXT_val_prn(out, nval, 0, 0);
     }
   }
+  sk_GENERAL_NAME_pop_free(names, GENERAL_NAME_free);

   return true;
 }

This should fix definite leaks.

[jasisk]

Great. Will deploy in the morning and report back. Thanks for all the work!

[indutny]

Cool, that patch is landed in v1.x, btw: 3b57819

[indutny]

Ok, I finally installed kappa and I do observe growing RSS, going to spend some time with valgrind...

[indutny]

Here is the heapsnapshot after 20k requests: https://cloudup.com/cSFN1VaukuF

[indutny]

I think I know where the leak goes from :) 5 minutes, working on PR.

[indutny]

This should fix it: https://github.com/iojs/io.js/pull/1090

[jasisk]

Build from yesterday spiked overnight. Just deployed 3b57819 + 50c7421 + 0dee2ad.

[indutny]

Should be fixed in latest master, and might be improved by https://github.com/iojs/io.js/commit/50c74217af1a53f3f25533436153efcc49d2f8df . Please verify @aredridel @jasisk

[rvagg]

See release proposal v1.5.1 in #1096, I'd love to get feedback from everyone involved here as to whether it's worth shipping a new release to clear this up. Even incremental progress is good of course.

RC with the latest HEAD in v1.x is here: https://iojs.org/download/nightly/v1.5.1-nightly20150308fe36076c78/

[gm112]

With the build before 50c7421 after 4 hours my server would sit at 150MB. After the same amount of time with similar traffic it sits at 60MB, which is also where the baseline is as well. Just thought I would provide some input.

[jasisk]

Been running the build I mentioned above (included #1085 and #1090) for about nine hours, driving double the usual traffic. Not enough consistency in weekend traffic to have it level off, but it certainly looks a lot better.

purple: iojs, green: 0.10, red: 0.12

Deploying the rc just in case, will report if anything looks different.

[rvagg]

@indutny @bnoordhuis can you comment on the shape of the graph for 0.12 vs io.js at all? the lack of increase is great but the delta is kind of large.

[indutny]

@rvagg no comments yet, need to investigate. But it might be that v8's GC has became overall less aggressive.