sailthru / sailthru-node-client

Sailthru Node.js client
http://docs.sailthru.com
MIT License
17 stars 15 forks source link

Fix the JavaScript heap out of memory error #48

Closed pempel closed 11 months ago

pempel commented 11 months ago

UI-615

This PR fixes the "JavaScript heap out of memory" error one of our customers has run into recently.

Let's call the processJob method or any other method that itself calls apiPostMultiPart:

const sailthru = require('sailthru-client').createSailthruClient('apiKey', 'apiSecret')
sailthru.enableLogging()

const data = {
  job: 'import',
  list: 'test-list',
  file: './test.txt'
}
sailthru.apiPost('job', data, ['file'], (error, response) => {
  console.log(error, response)
})

Here's the log message we can expect to see as a result of running the above code:

9 Nov 11:36:59 - sailthru-client - JSON Payload: {"api_key":"apiKey","format":"json","json":"{\"job\":\"import\",\"list\":\"test-list\"}","sig":"1a930db51af49a27d9cc1986eba4af6d","file":{"buffer":{"type":"Buffer","data":[123,10,32,32,34,97,34,58,32,49,44,10,32,32,34,98,34,58,32,50,44,10,32,32,34,99,34,58,32,51,10,125,10]},"filename":"test.txt","content_type":"application/octet-stream"}}

In case of a much bigger file (~ 200MB) Node.js gets us the following error depending on how much memory has been allocated to it:

FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory

This PR simply prevents the binary content of the file (or files) the user wants to send to the Sailthru API from appearing in the log output by replacing it with "[TRUNCATED]". Here's an example:

10 Nov 13:30:53 - sailthru-client - JSON Payload: {"api_key":"apiKey","format":"json","json":"{\"job\":\"import\",\"list\":\"test-list\"}","sig":"1a930db51af49a27d9cc1986eba4af6d","file":"[TRUNCATED]"}