nodules / asker

http.request wrapper with gzip, request retries and http.Agent tuning
MIT License
93 stars 11 forks source link

Cache DNS lookups #95

Closed 1999 closed 10 years ago

1999 commented 10 years ago

As far as I can see, asker doesn't cache dns lookups, and due to dns module documentation http.get / http.request will call dns.lookup each time request is made. It seems like adding dns.lookup caching will speed up requests. The only new thing will be Host header. What do you think about it?

The problem I faced was getaddrinfo ENOTFOUND, so I tried to investigate it a bit and came to conclusion that chanhing localhost to 127.0.0.1 fixed the problem a bit (Error: connect EMFILE appears, but that's a problem of http.agent setting probably).

code: 905,
  codeName: 'HTTP_CLIENT_REQUEST_ERROR',
  message: 'http.clientRequest error for request  in 23~240 ms http://localhost:5984/products/4974bec06d84996dbc276cb41c052e6cc81fb80b',
  data: 
   { timings: 'in 23~240 ms',
     url: 'http://localhost:5984/products/4974bec06d84996dbc276cb41c052e6cc81fb80b',
     requestId: '' },
  originalError: { [Error: getaddrinfo ENOTFOUND] code: 'ENOTFOUND', errno: 'ENOTFOUND', syscall: 'getaddrinfo' } 
1999 commented 10 years ago

https://github.com/mikeal/request/issues/803 is probably relating issue

kaero commented 10 years ago

As the feature request it looks fine.

But why you have a lot of DNS lookups? Lets enable keep alive on the target HTTP server and configure agents on your app side: while you have the incoming load connections will be reused from the agent pool.

1999 commented 10 years ago

I can't configure target HTTP server - this is CouchDB and AFAIK it doesn't allow keep-alive connections. Btw, what happens when all agent pool sockets are busy? Does asker reject the request or puts request in some kind of queue?

1999 commented 10 years ago

Btw, bem-node (which is used in tv.yandex.ru) has some kind of this dns caching layer.

1999 commented 10 years ago

We talked about this feature and came to conclusion: dns requests should not be cached at application level. Instead you should use special tools for that, for example bind9 or any other dns caching server. bem-node code which resolves hostname by making request to / is a great mistake.