umeshwaghode
commented
3 years ago I have done trial and error and got this working for all http proxy, pac proxy as well as no proxy scenario.
Below two lines in agent.ts file is making the difference after these are commented:
req._header = null; req._implicitHeader();
These lines are already inside'if' condition that checks if req._header is not undefined, then why req._implicitHeader() is needed. This is causing some additional content getting appended during multipart/formdata upload.
@TooTallNate could you please share your thoughts if it is safe to comment these lines. If so, when can this fix be included in the next version?
Sample program is available at the above provided stackoverflow question.
TooTallNate
I am using superagent (6.1.0) module to invoke REST API. One of the API uploads a zip file. When I test by passing http proxy, it goes through fine. But when I use PAC proxy (proxy Auto config), superagent first uses pac-proxy-agent to resovle the proxy to be used, which happens to be the same http proxy and then uses http-proxy-agent to perform the request.. This time, the API fails with error 'Unexpected end of MIME multipart stream. MIME multipart message is not complete.'
Below are the logs with HTTP Proxy: D:\testnodejs>node pproxytest.js superagent creating
fs.ReadStreaminstance for file: D:\F\workspace\Java\tutorials-master\tutorials-master\algorithms\src\main.zip +0ms superagent-proxy Request#proxy('http://10.32.2.169:808') +0ms superagent-proxy patched superagent Request "url" property for changes +1ms proxy-agent creating new ProxyAgent instance: 'http://10.32.2.169:808' +0ms agent-base Converting legacy callback function to promise +0ms agent-base Resolving socket for 'http:' request: 'POST /ABC/projects/10058/sourceCode/attachments' +0ms http-proxy-agent Creating new HttpProxyAgent instance: Url { protocol: 'http:', slashes: true, auth: null, host: '10.32.2.169:808', port: '808', hostname: '10.32.2.169', hash: null, search: null, query: null, pathname: '/', path: '/', href: 'http://10.32.2.169:808/' } +0ms http-proxy-agent Creatingnet.Socket: { protocol: 'http:', slashes: true, auth: null, host: '10.32.2.169', port: 808, hostname: '10.32.2.169', hash: null, search: null, query: null, href: 'http://10.32.2.169:808/' } +1ms superagent POST http://api.example.com/ABC/projects/10058/sourceCode/attachments +12ms superagent setting FormData header: "content-type: multipart/form-data; boundary=--------------------------630107661057145674080211" +0ms superagent got FormData Content-Length: 24880 +4ms superagent POST http://api.example.com/ABC/projects/10058/sourceCode/attachments -> 204 +859ms <ref 2> Response { _events: [Object: null prototype] {}, _eventsCount: 0, _maxListeners: undefined, res: <ref 1> IncomingMessage { _readableState: ReadableState { objectMode: false, highWaterMark: 16384, buffer: BufferList { head: null, tail: null, length: 0 }, length: 0, pipes: [], flowing: true, ended: true, endEmitted: true, reading: false, sync: true, needReadable: false, emittedReadable: false, readableListening: false, resumeScheduled: false, errorEmitted: false, emitClose: true, autoDestroy: false, destroyed: false, errored: null, closed: false, closeEmitted: false, defaultEncoding: 'utf8', awaitDrainWriters: null, multiAwaitDrain: false, readingMore: true, decoder: [StringDecoder], encoding: 'utf8',}, request: Request { _events: [Object: null prototype] { abort: [Function (anonymous)] }, _eventsCount: 1, _maxListeners: undefined, _enableHttp2: false, _agent: ProxyAgent { proxy: [Url], proxyUri: 'http://10.32.2.169:808', proxyFn: [Function: httpOrHttpsProxy], promisifiedCallback: [Function (anonymous)] }, _formData: FormData { _overheadLength: 264, _valueLength: 5, _valuesToMeasure: [Array], writable: false, readable: true, dataSize: 0, maxDataSize: 2097152, pauseStreams: true, _released: true, _streams: [], _currentStream: null, _insideLoop: false, _pendingNext: false, _events: [Object: null prototype], _eventsCount: 1, _boundary: '--------------------------630107661057145674080211' }, method: 'POST', url: [Getter/Setter], _header: { accept: 'application/json', authorization: 'Bearer accesstoken' }, header: { Accept: 'application/json', Authorization: 'Bearer accesstoken' }, writable: true, _redirects: 0, _maxRedirects: 5, cookies: '', qs: {}, _query: [], qsRaw: [], _redirectList: [], _streamRequest: false, _url: 'http://api.example.com/ABC/projects/10058/sourceCode/attachments', _proxyUri: 'http://10.32.2.169:808', req: ClientRequest { _events: [Object: null prototype], _eventsCount: 2, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: false, _last: true, chunkedEncoding: false, shouldKeepAlive: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: true, sendDate: false, _removedConnection: false, _removedContLen: false, _removedTE: false, _contentLength: null, _hasBody: true, _trailer: '', finished: true, _headerSent: true, socket: [Socket], _header: 'POST http://api.example.com/ABC/projects/10058/sourceCode/attachments HTTP/1.1\r\n' + 'Host: api.example.com\r\n' + 'Accept-Encoding: gzip, deflate\r\n' + 'Accept: application/json\r\n' + 'Authorization: Bearer accesstoken\r\n' + 'content-type: multipart/form-data; boundary=--------------------------630107661057145674080211\r\n' + 'Content-Length: 24880\r\n' + 'Connection: close\r\n' + '\r\n', _keepAliveTimeout: 0, _onPendingData: [Function: noopPendingOutput], agent: [ProxyAgent], socketPath: undefined, method: 'POST', maxHeaderSize: undefined, insecureHTTPParser: undefined, path: 'http://api.example.com/ABC/projects/10058/sourceCode/attachments', _ended: true, res: [IncomingMessage], aborted: false, timeoutCb: null, upgradeOrConnect: false, parser: null, maxHeadersCount: null, reusedSocket: false, host: 'api.example.com', protocol: 'http:',
}, req: <ref 3> ClientRequest { _events: [Object: null prototype] { error: [Function (anonymous)], prefinish: [Function: requestOnPrefinish] }, _eventsCount: 2, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: false, _last: true, chunkedEncoding: false, shouldKeepAlive: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: true, sendDate: false, _removedConnection: false, _removedContLen: false, _removedTE: false, _contentLength: null, _hasBody: true, _trailer: '', finished: true, _headerSent: true, socket: Socket { connecting: false, _hadError: false, _parent: null, _host: null, _readableState: [ReadableState], _events: [Object: null prototype], _eventsCount: 5, _maxListeners: undefined, _writableState: [WritableState], allowHalfOpen: false, _sockname: null, _pendingData: null, _pendingEncoding: '', server: null, _server: null, parser: null, _httpMessage: [Circular 3],
}, text: '', body: {}, files: undefined, buffered: true, headers: { 'cache-control': 'no-cache', pragma: 'no-cache', expires: '-1', server: 'Microsoft-IIS/8.5', 'api-version': '1.0', 'x-aspnet-version': '4.0.30319', 'x-powered-by': 'ASP.NET', date: 'Thu, 11 Feb 2021 13:51:51 GMT', connection: 'close' }, header: { 'cache-control': 'no-cache', pragma: 'no-cache', expires: '-1', server: 'Microsoft-IIS/8.5', 'api-version': '1.0', 'x-aspnet-version': '4.0.30319', 'x-powered-by': 'ASP.NET', date: 'Thu, 11 Feb 2021 13:51:51 GMT', connection: 'close' }, statusCode: 204, status: 204, statusType: 2, info: false, ok: true, redirect: false, clientError: false, serverError: false, error: false, created: false, accepted: false, noContent: true, badRequest: false, unauthorized: false, notAcceptable: false, forbidden: false, notFound: false, unprocessableEntity: false, type: '', links: {}, setEncoding: [Function: bound ], redirects: [], pipe: [Function (anonymous)],
} Below are the logs with PAC Proxy:
D:\testnodejs>node pproxytest.js superagent creating (D:\testnodejs\node_modules\superagent\lib\node\parsers\json.js:22:7)
at IncomingMessage.emit (events.js:327:22)
at endReadableNT (internal/streams/readable.js:1327:12)
fs.ReadStreaminstance for file: D:\F\workspace\Java\tutorials-master\tutorials-master\algorithms\src\main.zip +0ms superagent-proxy Request#proxy('pac+http://localhost:6080/examples/proxy.pac') +0ms superagent-proxy patched superagent Request "url" property for changes +1ms proxy-agent creating new ProxyAgent instance: 'pac+http://localhost:6080/examples/proxy.pac' +0ms agent-base Converting legacy callback function to promise +0ms agent-base Resolving socket for 'http:' request: 'POST /ABC/projects/10058/sourceCode/attachments' +1ms pac-proxy-agent Creating PacProxyAgent with URI 'pac+http://localhost:6080/examples/proxy.pac' and options Url { protocol: 'pac+http:', slashes: true, auth: null, host: 'localhost:6080', port: '6080', hostname: 'localhost', hash: null, search: null, query: null, pathname: '/examples/proxy.pac', path: '/examples/proxy.pac', href: 'pac+http://localhost:6080/examples/proxy.pac' } +0ms pac-proxy-agent Loading PAC file: 'http://localhost:6080/examples/proxy.pac' +2ms get-uri getUri('http://localhost:6080/examples/proxy.pac') +0ms get-uri:http GET 'http://localhost:6080/examples/proxy.pac' +0ms get-uri:http allowing 5 max redirects +1ms get-uri:http usinghttpcore module +0ms superagent POST http://api.example.com/ABC/projects/10058/sourceCode/attachments +14ms superagent setting FormData header: "content-type: multipart/form-data; boundary=--------------------------148489407975531565009366" +0ms superagent got FormData Content-Length: 24880 +5ms get-uri:http got 200 response status code +14ms pac-proxy-agent GotReadableinstance for URI +16ms pac-proxy-agent Read 129 byte PAC file from URI +1ms pac-proxy-agent Creating new proxy resolver instance +1ms pac-proxy-agent url: 'http://api.example.com/ABC/projects/10058/sourceCode/attachments' +8ms pac-proxy-agent Attempting to use proxy: 'PROXY 10.32.2.169:808' +0ms http-proxy-agent Creating new HttpProxyAgent instance: { protocol: 'http:', slashes: true, auth: null, host: '10.32.2.169:808', port: '808', hostname: '10.32.2.169', hash: null, search: null, query: null, pathname: '/', path: '/', href: 'http://10.32.2.169:808/', filename: 'pac+http://localhost:6080/examples/proxy.pac' } +0ms http-proxy-agent Creatingnet.Socket: { protocol: 'http:', slashes: true, auth: null, host: '10.32.2.169', port: 808, hostname: '10.32.2.169', hash: null, search: null, query: null, href: 'http://10.32.2.169:808/', filename: 'pac+http://localhost:6080/examples/proxy.pac' } +0ms http-proxy-agent Regenerating stored HTTP header string for request +1ms http-proxy-agent Patching connection write() output buffer with updated header +1ms http-proxy-agent Output buffer: 'POST http://api.example.com/ABC/projects/10058/sourceCode/attachments HTTP/1.1\r\n' + 'Host: api.example.com\r\n' + 'Accept-Encoding: gzip, deflate\r\n' + 'Accept: application/json\r\n' + 'Authorization: Bearer accesstoken\r\n' + 'content-type: multipart/form-data; boundary=--------------------------148489407975531565009366\r\n' + 'Content-Length: 24880\r\n' + 'Connection: close\r\n' + '\r\n' +0ms superagent POST http://api.example.com/ABC/projects/10058/sourceCode/attachments -> 500 +878ms (node:22468) UnhandledPromiseRejectionWarning: Error: Internal Server Error at Request.callback (D:\testnodejs\node_modules\superagent\lib\node\index.js:883:15) at D:\testnodejs\node_modules\superagent\lib\node\index.js:1127:20 at IncomingMessage.