search

node-modules / urllib

Request HTTP(s) URLs in a complex world.
MIT License
726 stars 117 forks source link

πŸ“¦ NEW: Support timing and socket info #400

Closed fengmk2 closed 1 year ago

fengmk2 commented 1 year ago 
NODE_DEBUG=urllib:* node examples/httpclient.cjs
URLLIB:HTTPCLIENT 23661: Request#1 GET https://nodejs.org/, headers: {"user-agent":"node-urllib/3.0.0 Node.js/18.7.0 (OS X; arm64)"}, headersTimeout: 10000, bodyTimeout: 10000
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:create] Request#1 GET https://nodejs.org
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:client:connected] Socket#1 connected
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:client:sendHeaders] Request#1 send headers on Socket#1 (handled 1 requests)
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:bodySent] Request#1 send body
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:headers] Request#1 get response headers on Socket#1 (handled 1 responses)
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:trailers] Request#1 get response body and trailers
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:create] Request#1 GET https://nodejs.org
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:client:connected] Socket#2 connected
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:client:sendHeaders] Request#1 send headers on Socket#2 (handled 1 requests)
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:bodySent] Request#1 send body
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:headers] Request#1 get response headers on Socket#2 (handled 1 responses)
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:trailers] Request#1 get response body and trailers
urllib {
  status: 200,
  statusCode: 200,
  headers: {
    date: 'Sun, 25 Sep 2022 17:12:24 GMT',
    'content-type': 'text/html',
    'transfer-encoding': 'chunked',
    connection: 'keep-alive',
    'last-modified': 'Sun, 25 Sep 2022 16:05:56 GMT',
    'cache-control': 'max-age=14400',
    'cf-cache-status': 'HIT',
    age: '3737',
    'strict-transport-security': 'max-age=31536000; includeSubDomains; preload',
    'x-content-type-options': 'nosniff',
    server: 'cloudflare',
    'cf-ray': '750562711ba46432-SJC'
  },
  size: 0,
  aborted: false,
  rt: 967.27,
  keepAliveSocket: true,
  requestUrls: [ 'https://nodejs.org/', 'https://nodejs.org/en/' ],
  timing: {
    queuing: 492.781,
    connected: 799.974,
    requestHeadersSent: 801.528,
    requestSent: 803.296,
    waiting: 962.664,
    contentDownload: 966.34
  },
  socket: {
    id: 2,
    localAddress: '192.168.3.211',
    localPort: 63992,
    remoteAddress: '104.20.23.46',
    remotePort: 443,
    remoteFamily: 'IPv4',
    bytesWritten: 123,
    bytesRead: 15327,
    handledRequests: 1,
    handledResponses: 1
  }
}
URLLIB:HTTPCLIENT 23661: Request#2 GET https://nodejs.org/, headers: {"user-agent":"node-urllib/3.0.0 Node.js/18.7.0 (OS X; arm64)"}, headersTimeout: 10000, bodyTimeout: 10000
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:create] Request#2 GET https://nodejs.org
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:client:sendHeaders] Request#2 send headers on Socket#1 (handled 2 requests)
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:bodySent] Request#2 send body
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:headers] Request#2 get response headers on Socket#1 (handled 2 responses)
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:trailers] Request#2 get response body and trailers
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:create] Request#2 GET https://nodejs.org
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:client:sendHeaders] Request#2 send headers on Socket#2 (handled 2 requests)
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:bodySent] Request#2 send body
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:headers] Request#2 get response headers on Socket#2 (handled 2 responses)
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:trailers] Request#2 get response body and trailers
urllib {
  status: 200,
  statusCode: 200,
  headers: {
    date: 'Sun, 25 Sep 2022 17:12:24 GMT',
    'content-type': 'text/html',
    'transfer-encoding': 'chunked',
    connection: 'keep-alive',
    'last-modified': 'Sun, 25 Sep 2022 16:05:56 GMT',
    'cache-control': 'max-age=14400',
    'cf-cache-status': 'HIT',
    age: '3737',
    'strict-transport-security': 'max-age=31536000; includeSubDomains; preload',
    'x-content-type-options': 'nosniff',
    server: 'cloudflare',
    'cf-ray': '750562731f386432-SJC'
  },
  size: 0,
  aborted: false,
  rt: 320.27,
  keepAliveSocket: true,
  requestUrls: [ 'https://nodejs.org/', 'https://nodejs.org/en/' ],
  timing: {
    queuing: 154.396,
    connected: 0,
    requestHeadersSent: 154.613,
    requestSent: 155.065,
    waiting: 316.475,
    contentDownload: 320.046
  },
  socket: {
    id: 2,
    localAddress: '192.168.3.211',
    localPort: 63992,
    remoteAddress: '104.20.23.46',
    remotePort: 443,
    remoteFamily: 'IPv4',
    bytesWritten: 246,
    bytesRead: 30654,
    handledRequests: 2,
    handledResponses: 2
  }
}
URLLIB:HTTPCLIENT 23661: Request#3 GET https://nodejs.org/, headers: {"user-agent":"node-urllib/3.0.0 Node.js/18.7.0 (OS X; arm64)"}, headersTimeout: 10000, bodyTimeout: 10000
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:create] Request#3 GET https://nodejs.org
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:client:sendHeaders] Request#3 send headers on Socket#1 (handled 3 requests)
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:bodySent] Request#3 send body
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:headers] Request#3 get response headers on Socket#1 (handled 3 responses)
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:trailers] Request#3 get response body and trailers
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:create] Request#3 GET https://nodejs.org
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:client:sendHeaders] Request#3 send headers on Socket#2 (handled 3 requests)
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:bodySent] Request#3 send body
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:headers] Request#3 get response headers on Socket#2 (handled 3 responses)
URLLIB:DIAGNOSTICSCHANNEL 23661: [undici:request:trailers] Request#3 get response body and trailers
urllib {
  status: 200,
  statusCode: 200,
  headers: {
    date: 'Sun, 25 Sep 2022 17:12:25 GMT',
    'content-type': 'text/html',
    'transfer-encoding': 'chunked',
    connection: 'keep-alive',
    'last-modified': 'Sun, 25 Sep 2022 16:05:56 GMT',
    'cache-control': 'max-age=14400',
    'cf-cache-status': 'HIT',
    age: '3738',
    'strict-transport-security': 'max-age=31536000; includeSubDomains; preload',
    'x-content-type-options': 'nosniff',
    server: 'cloudflare',
    'cf-ray': '750562752a906432-SJC'
  },
  size: 0,
  aborted: false,
  rt: 316.453,
  keepAliveSocket: true,
  requestUrls: [ 'https://nodejs.org/', 'https://nodejs.org/en/' ],
  timing: {
    queuing: 154.699,
    connected: 0,
    requestHeadersSent: 154.851,
    requestSent: 155.32,
    waiting: 314.762,
    contentDownload: 316.19
  },
  socket: {
    id: 2,
    localAddress: '192.168.3.211',
    localPort: 63992,
    remoteAddress: '104.20.23.46',
    remotePort: 443,
    remoteFamily: 'IPv4',
    bytesWritten: 369,
    bytesRead: 45981,
    handledRequests: 3,
    handledResponses: 3
  }
}
codecov[bot] commented 1 year ago

Codecov Report

Merging #400 (099940f) into master (414692b) will increase coverage by 1.06%. The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #400      +/-   ##
==========================================
+ Coverage   97.34%   98.40%   +1.06%     
==========================================
  Files           4        6       +2     
  Lines         753      940     +187     
  Branches      168      210      +42     
==========================================
+ Hits          733      925     +192     
+ Misses         20       15       -5
Impacted Files Coverage Ξ”
src/HttpClient.ts 98.07% <100.00%> (+1.06%) :arrow_up:
src/diagnosticsChannel.ts 100.00% <100.00%> (ΓΈ)
src/symbols.ts 100.00% <100.00%> (ΓΈ)
src/utils.ts 97.90% <100.00%> (+0.24%) :arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

fengmk2 commented 1 year ago

性能不受影响

[bench:run] > CONNECTIONS=1 node --experimental-wasm-simd benchmarks/benchmark.js; CONNECTIONS=50 node --experimental-wasm-simd benchmarks/benchmark.js
[bench:run] 
[bench:run] β”‚ Tests                           β”‚ Samples β”‚        Result β”‚ Tolerance β”‚ Difference with slowest β”‚
[bench:run] |─────────────────────────────────|─────────|───────────────|───────────|─────────────────────────|
[bench:run] β”‚ http - no keepalive             β”‚       1 β”‚  6.60 req/sec β”‚  Β± 0.00 % β”‚                       - β”‚
[bench:run] |─────────────────────────────────|─────────|───────────────|───────────|─────────────────────────|
[bench:run] β”‚ http - keepalive                β”‚       1 β”‚  7.02 req/sec β”‚  Β± 0.00 % β”‚                + 6.46 % β”‚
[bench:run] |─────────────────────────────────|─────────|───────────────|───────────|─────────────────────────|
[bench:run] β”‚ urllib2 - request               β”‚       1 β”‚ 14.13 req/sec β”‚  Β± 0.00 % β”‚              + 114.26 % β”‚
[bench:run] |─────────────────────────────────|─────────|───────────────|───────────|─────────────────────────|
[bench:run] β”‚ undici - fetch                  β”‚       1 β”‚ 35.47 req/sec β”‚  Β± 0.00 % β”‚              + 437.66 % β”‚
[bench:run] |─────────────────────────────────|─────────|───────────────|───────────|─────────────────────────|
[bench:run] β”‚ undici - request                β”‚       1 β”‚ 48.24 req/sec β”‚  Β± 0.00 % β”‚              + 631.28 % β”‚
[bench:run] |─────────────────────────────────|─────────|───────────────|───────────|─────────────────────────|
[bench:run] β”‚ undici - pipeline               β”‚       1 β”‚ 49.07 req/sec β”‚  Β± 0.00 % β”‚              + 643.78 % β”‚
[bench:run] |─────────────────────────────────|─────────|───────────────|───────────|─────────────────────────|
[bench:run] β”‚ undici - stream                 β”‚       1 β”‚ 49.65 req/sec β”‚  Β± 0.00 % β”‚              + 652.65 % β”‚
[bench:run] |─────────────────────────────────|─────────|───────────────|───────────|─────────────────────────|
[bench:run] β”‚ urllib3 enable timing - request β”‚       1 β”‚ 50.12 req/sec β”‚  Β± 0.00 % β”‚              + 659.77 % β”‚
[bench:run] |─────────────────────────────────|─────────|───────────────|───────────|─────────────────────────|
[bench:run] β”‚ urllib3 - request               β”‚       1 β”‚ 54.23 req/sec β”‚  Β± 0.00 % β”‚              + 721.98 % β”‚
[bench:run] |─────────────────────────────────|─────────|───────────────|───────────|─────────────────────────|
[bench:run] β”‚ undici - dispatch               β”‚       1 β”‚ 54.85 req/sec β”‚  Β± 0.00 % β”‚              + 731.44 % β”‚
[bench:run] 
[bench:run] β”‚ Tests                           β”‚ Samples β”‚          Result β”‚ Tolerance β”‚ Difference with slowest β”‚
[bench:run] |─────────────────────────────────|─────────|─────────────────|───────────|─────────────────────────|
[bench:run] β”‚ urllib2 - request               β”‚       1 β”‚  753.05 req/sec β”‚  Β± 0.00 % β”‚                       - β”‚
[bench:run] |─────────────────────────────────|─────────|─────────────────|───────────|─────────────────────────|
[bench:run] β”‚ http - no keepalive             β”‚       1 β”‚  957.44 req/sec β”‚  Β± 0.00 % β”‚               + 27.14 % β”‚
[bench:run] |─────────────────────────────────|─────────|─────────────────|───────────|─────────────────────────|
[bench:run] β”‚ undici - fetch                  β”‚       1 β”‚ 1822.12 req/sec β”‚  Β± 0.00 % β”‚              + 141.97 % β”‚
[bench:run] |─────────────────────────────────|─────────|─────────────────|───────────|─────────────────────────|
[bench:run] β”‚ undici - pipeline               β”‚       1 β”‚ 2808.50 req/sec β”‚  Β± 0.00 % β”‚              + 272.95 % β”‚
[bench:run] |─────────────────────────────────|─────────|─────────────────|───────────|─────────────────────────|
[bench:run] β”‚ urllib3 - request               β”‚       1 β”‚ 2941.95 req/sec β”‚  Β± 0.00 % β”‚              + 290.67 % β”‚
[bench:run] |─────────────────────────────────|─────────|─────────────────|───────────|─────────────────────────|
[bench:run] β”‚ urllib3 enable timing - request β”‚       1 β”‚ 3945.72 req/sec β”‚  Β± 0.00 % β”‚              + 423.97 % β”‚
[bench:run] |─────────────────────────────────|─────────|─────────────────|───────────|─────────────────────────|
[bench:run] β”‚ http - keepalive                β”‚       1 β”‚ 4706.85 req/sec β”‚  Β± 0.00 % β”‚              + 525.04 % β”‚
[bench:run] |─────────────────────────────────|─────────|─────────────────|───────────|─────────────────────────|
[bench:run] β”‚ undici - request                β”‚       1 β”‚ 5736.53 req/sec β”‚  Β± 0.00 % β”‚              + 661.78 % β”‚
[bench:run] |─────────────────────────────────|─────────|─────────────────|───────────|─────────────────────────|
[bench:run] β”‚ undici - stream                 β”‚       1 β”‚ 6679.28 req/sec β”‚  Β± 0.00 % β”‚              + 786.97 % β”‚
[bench:run] |─────────────────────────────────|─────────|─────────────────|───────────|─────────────────────────|
[bench:run] β”‚ undici - dispatch               β”‚       1 β”‚ 7024.86 req/sec β”‚  Β± 0.00 % β”‚              + 832.86 % β”‚
fengmk2 commented 1 year ago

3.2.0