Closed richard-kurtosys closed 6 years ago
If I save the S3 buffer to a file on disk and then use:
var inStream = new hummus.PDFRStreamForFile("s3file.pdf"); // Only this changes
var outStream = new hummus.PDFStreamForResponse(res);
const pdfWriter = hummus.createWriterToModify(inStream, outStream,);
then everything works correctly and inStream
and outStream
both have a size of 268816
and the file downloads as 287kb (which is correct as I apply a watermark to it).
Hello Richard, maybe you can help me with a question I had.
My problem is, I receive from an interface a pdf as a base64 string and I need to append some base64 stringed images to it. So I think I first need to write the pdfbase64 string to a file but I donot know how to do that.
The document.file.Body you have is a base64 string? How did you save that buffer to a file on disk?
Thank You!
Hi @pwinkelm
The document.file.Body I have is an actual buffer - https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getObject-property
I used:
function writeFile(filename, buffer) {
return new Promise((resolve, reject) => {
fs.writeFile(filename, buffer, (error, data) => {
if (error) {
return reject(error);
}
return resolve(data);
});
});
}
await writeFile("filename.pdf", document.file.Body);
But you should be able to get away with something like: http://www.codeblocq.com/2016/04/Convert-a-base64-string-to-a-file-in-Node/
Back to the origional issue, would it make sense to pass in the buffer size to
EStatusCode status = traits.CopyToOutputStream(inModifiedSourceStream);
?
here https://github.com/galkahana/HummusJS/blob/master/src/deps/PDFWriter/PDFWriter.cpp#L710
inModifiedSourceStream should have a fixed size as it would be either a file or a buffer (not a stream). My C isn't good enough to be able to test and try this :(
I did some more research into this, trying different buffer sizes and running npm install --build-from-source
to generate a new /node_modules/hummus/binding/hummus.node
.
I came up with the following:
Input filesize 10618
Buffer size Average time in ms Output filesize
1 createWriterToModify 41,58715 29239
1024 createWriterToModify 5,50785 29885
65535 createWriterToModify 20,8952 84157
10485760 createWriterToModify 3168,794 10504384
Input filesize 18299457
Buffer size Average time in ms Output filesize
1 createWriterToModify Runs out of memory ---
1024 createWriterToModify 6395,38055 20276115
65535 createWriterToModify 6550,6411 20326291
10485760 createWriterToModify 7397,794 22947731
The first set is for the time to run createWriterToModify
on a 10k pdf.
The second set is for the time to run createWriterToModify
on a 18Mb pdf.
The very left has what I had the buffer size set to and the very right is the final output file size.
The quickest appears to be when the buffer size is set to 1024
. This also gives the second smallest filesize.
Based on this, would it be possible to change https://github.com/galkahana/HummusJS/blob/master/src/deps/PDFWriter/OutputStreamTraits.cpp#L36 to 1024
and not 10*1024*1024
?
use 1.0.85 version of PDFRStreamForBuffer. it's the cause of error, and so where the correction is to be made.
Awesome @galkahana - I'm looking forward to testing and trying this out!
:)
Hopefully I'll have some feedback in a day or 2 and I'll paste in some results.
Confirmation that this has been fixed in 1.0.86 :) Thank You Very Much @galkahana
For the incoming buffer be use to use new hummus.PDFRStreamForBuffer
, before I was using new PDFRStreamForBuffer
.
var inStream = new hummus.PDFRStreamForBuffer(document.file.Body);
Stats for 1.0.86 are:
Input filesize 10618 (1 page)
Buffer size Average time in ms Output filesize
(not changed) createWriterToModify 5.38735 29239
Input filesize 18299457 (1310 pages)
Buffer size Average time in ms Output filesize
(not changed) createWriterToModify 6727.76175 20275668
Hi,
I have a 269Kb pdf on S3 that I'm retrieving into a buffer using
s3.getObject
. I'm using the following code to create the inputs and outputsAfter the call to
hummus.createWriterToModify
,inStream
hasinStream.fileSize
=268816
andinStream.rposition
=253966
.The problem is
outStream.position
is10485760
.The pdf get's manipulated and displays correctly but is now a 10Mb file.
I went through the the source and found the following: https://github.com/galkahana/HummusJS/blob/master/src/deps/PDFWriter/PDFWriter.cpp#L710
EStatusCode status = traits.CopyToOutputStream(inModifiedSourceStream);
CopyToOutputStream
is here: https://github.com/galkahana/HummusJS/blob/master/src/deps/PDFWriter/OutputStreamTraits.cpp#L36 Is there a way to copy just the buffer size and not create a 10Mb object? Or is there a different solution to this?Thank you for the assistance!