sivcan / ResponseToFile-Postman

This project helps in writing responses (or any data) from postman to a file locally.
112 stars 104 forks source link

How to save a pdf file using this tool? #6

Open sinewave440hz opened 4 years ago

sinewave440hz commented 4 years ago

Hi, I have a get request that returns a pdf, how do I get this to save to a file? I've got as far as:

let opts = {
    requestName: request.name || request.url,
    fileExtension: 'pdf',
    mode: 'writeFile', // Change this to any function of the fs library of node to use it.
    uniqueIdentifier: pm.iterationData.get("ID"),
    responseData: pm.response.stream
};

I'm not quite sure what to do now, this just hangs and writes an incomplete pdf file. Any ideas?

BendicoE commented 3 years ago

I'm also facing this challenge. I have tried "everything" tweaking the server script.js and the collection Test script in Postman. I have managed to save PDF files, but they are not binary correct, Adobe Reader chokes when trying to open them. If I Send the request in Postman and then Save the response as a file in Postman, the PDF is fine. One suspicious thing is that the PDFs written by this server are approx twice the size of those saved from Postman. I'm using responseData: pm.response.text() and I suspect that either the .text() or the JSON.stringify(responseData) somehow "clobbers" the response body - which is Content-Type: application/pdf. I have tried different encodings for fs.writeFile in the server script, but to no avail. The file size seem to vary with different encodings, and sometimes the outcome is just a very small file (< 1K). It would be great if the author or some node/postman-wiz could shed some light on this, I really need to batch test my API which returns PDFs "en masse".

sivcan commented 3 years ago

@BendicoE @sinewave440hz Sorry for the delay, I haven't been active around this. I tried to take a jab at this problem, can you guys try out the following script changes once and let me know if it works for you guys?

Replace your script.js with the following:

const express = require('express'),
  app = express(),
  fs = require('fs'),
  shell = require('shelljs'),

   // Modify the folder path in which responses need to be stored
  folderPath = './Responses/',
  defaultFileExtension = 'pdf', // Change the default file extension
  bodyParser = require('body-parser'),
  DEFAULT_MODE = 'writeFile',
  path = require('path');

// Create the folder path in case it doesn't exist
shell.mkdir('-p', folderPath);

 // Change the limits according to your response size
app.use(bodyParser.raw({ limit: '1gb', type: 'application/pdf' }));
app.use(bodyParser.urlencoded({ limit: '1gb', extended: false })); 

app.get('/', (req, res) => res.send('Hello, I write data to file. Send them requests!'));

app.post('/write', (req, res) => {
  let extension = req.body.fileExtension || defaultFileExtension,
    fsMode = req.body.mode || DEFAULT_MODE,
    uniqueIdentifier = req.body.uniqueIdentifier ? typeof req.body.uniqueIdentifier === 'boolean' ? Date.now() : req.body.uniqueIdentifier : false,
    filename = `${req.body.requestName || req.query.filename}${uniqueIdentifier || ''}`,
    filePath = `${path.join(folderPath, filename)}.${extension}`,
    options = req.body.options || undefined;

  fs[fsMode](filePath, req.body, 'binary', (err) => {
    if (err) {
      console.log(err);
      res.send('Error');
    }
    else {
      console.log(`File successfully recorded: ${filename}.${defaultFileExtension}`);
      res.send('Success');
    }
  });
});

app.listen(3000, () => {
  console.log('ResponsesToFile App is listening now! Send them requests my way!');
  console.log(`Data is being stored at location: ${path.join(process.cwd(), folderPath)}`);
});

and update the Test script in the collection to be the following:

// The opts for the server, also includes the data to be written to file
// Not utilized for this demo at the moment.
let opts = {
    requestName: request.name || request.url,
    fileExtension: 'pdf',
    mode: 'writeFile', // Change this to any function of the fs library of node to use it.
    uniqueIdentifier: 'gibberish',
    responseData: pm.response.text(),
    options: {
        encoding: 'binary'
    }
};

pm.sendRequest({
    url: 'http://localhost:3000/write?filename=sample',
    method: 'POST',
    header: 'Content-Type:application/pdf',
    body: {
        mode: 'raw',
        raw: pm.response.text()
    }
}, function (err, res) {
    console.log(res);
});

Let me know if this works for you guys?

ameyasawant7 commented 3 years ago

Hi @sivcan,

I tried the above code and it works. The PDF are getting saved. However the PDFs are blank, I guess this is because we are not writing the responseData: pm.response.text() I tried to write the responseData by making following changes to script.js fs[fsMode](filePath, req.body.responseData, options, (err) => { but getting below error. It would be helpful if you could provide the solution for this because I need this to do regression testing.

Thanks

sivcan commented 3 years ago

@ameyasawant7 The req.body is the responseData itself, there's no nesting there. More ref here: https://learning.postman.com/docs/writing-scripts/script-references/postman-sandbox-api-reference/#sending-requests-from-scripts

Not sure why it's blank, could be because of multiple reasons. You might want to check by posting on stackoverflow, since I am no PDF expert. 😅

kaisajanzen commented 3 years ago

@BendicoE @sinewave440hz Sorry for the delay, I haven't been active around this. I tried to take a jab at this problem, can you guys try out the following script changes once and let me know if it works for you guys?

Replace your script.js with the following:

const express = require('express'),
  app = express(),
  fs = require('fs'),
  shell = require('shelljs'),

   // Modify the folder path in which responses need to be stored
  folderPath = './Responses/',
  defaultFileExtension = 'pdf', // Change the default file extension
  bodyParser = require('body-parser'),
  DEFAULT_MODE = 'writeFile',
  path = require('path');

// Create the folder path in case it doesn't exist
shell.mkdir('-p', folderPath);

 // Change the limits according to your response size
app.use(bodyParser.raw({ limit: '1gb', type: 'application/pdf' }));
app.use(bodyParser.urlencoded({ limit: '1gb', extended: false })); 

app.get('/', (req, res) => res.send('Hello, I write data to file. Send them requests!'));

app.post('/write', (req, res) => {
  let extension = req.body.fileExtension || defaultFileExtension,
    fsMode = req.body.mode || DEFAULT_MODE,
    uniqueIdentifier = req.body.uniqueIdentifier ? typeof req.body.uniqueIdentifier === 'boolean' ? Date.now() : req.body.uniqueIdentifier : false,
    filename = `${req.body.requestName || req.query.filename}${uniqueIdentifier || ''}`,
    filePath = `${path.join(folderPath, filename)}.${extension}`,
    options = req.body.options || undefined;

  fs[fsMode](filePath, req.body, 'binary', (err) => {
    if (err) {
      console.log(err);
      res.send('Error');
    }
    else {
      console.log(`File successfully recorded: ${filename}.${defaultFileExtension}`);
      res.send('Success');
    }
  });
});

app.listen(3000, () => {
  console.log('ResponsesToFile App is listening now! Send them requests my way!');
  console.log(`Data is being stored at location: ${path.join(process.cwd(), folderPath)}`);
});

and update the Test script in the collection to be the following:

// The opts for the server, also includes the data to be written to file
// Not utilized for this demo at the moment.
let opts = {
    requestName: request.name || request.url,
    fileExtension: 'pdf',
    mode: 'writeFile', // Change this to any function of the fs library of node to use it.
    uniqueIdentifier: 'gibberish',
    responseData: pm.response.text(),
    options: {
        encoding: 'binary'
    }
};

pm.sendRequest({
    url: 'http://localhost:3000/write?filename=sample',
    method: 'POST',
    header: 'Content-Type:application/pdf',
    body: {
        mode: 'raw',
        raw: pm.response.text()
    }
}, function (err, res) {
    console.log(res);
});

Let me know if this works for you guys?

This script consistently returns a "500 internal server error" regardless of many tweaks and attempts. The response body lists, "

TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined<br"

I genuinely appreciate this script and the potential awesome help it could be. Any ideas about why I continually get this error are appreciated. Thanks!

aitest321 commented 2 years ago

Update: today I also see blank pdf issue. My file is 215KB. After response download and saved it is 397KB. Strangely it doesn't have issue with the 3KB pdf from Dummy Request 1.

Update2: upon further inspection, pdf from Dummy Request 1 [http://www.africau.edu/images/default/sample.pdf] also increased by 8 bytes, but file can still be opened okay by pdf reader. image

hamiltonchua commented 1 year ago

Passing the response as a stream is what worked for me This is what the script in my Tests editor looks like

// The opts for the server, also includes the data to be written to file
let opts = {
    requestName: request.name || request.url,
    fileExtension: 'pdf',
    mode: 'writeFile', // Change this to any function of the fs library of node to use it.
    uniqueIdentifier: false,
    responseData: pm.response.stream
};

pm.sendRequest({
    url: 'http://localhost:3000/write',
    method: 'POST',
    header: 'Content-Type:application/json',
    body: {
        mode: 'raw',
        raw: opts
    }
}, function (err, res) {
    console.log(opts)
    console.log(res);
})

This is what the script.js looks like

const express = require('express'),
  app = express(),
  fs = require('fs'),
  shell = require('shelljs'),

   // Modify the folder path in which responses need to be stored
  folderPath = './Responses/',
  defaultFileExtension = 'json', // Change the default file extension
  bodyParser = require('body-parser'),
  DEFAULT_MODE = 'writeFile',
  path = require('path');

// Create the folder path in case it doesn't exist
shell.mkdir('-p', folderPath);

 // Change the limits according to your response size
app.use(bodyParser.json({limit: '50mb', extended: true}));
app.use(bodyParser.urlencoded({ limit: '50mb', extended: true })); 

app.get('/', (req, res) => res.send('Hello, I write data to file. Send them requests!'));

app.post('/write', (req, res) => {
  let extension = req.body.fileExtension || defaultFileExtension,
    fsMode = req.body.mode || DEFAULT_MODE,
    uniqueIdentifier = req.body.uniqueIdentifier ? typeof req.body.uniqueIdentifier === 'boolean' ? Date.now() : req.body.uniqueIdentifier : false,
    filename = `${req.body.requestName}${uniqueIdentifier || ''}`,
    filePath = `${path.join(folderPath, filename)}.${extension}`,
    options = req.body.options || undefined;

  fs[fsMode](filePath, Buffer.from(req.body.responseData), options, (err) => {
    if (err) {
      console.log(err);
      res.send('Error');
    }
    else {
      res.send('Success');
    }
  });
});

app.listen(3000, () => {
  console.log('ResponsesToFile App is listening now! Send them requests my way!');
  console.log(`Data is being stored at location: ${path.join(process.cwd(), folderPath)}`);
});

Note this part Buffer.from(req.body.responseData),