Closed varunj90 closed 7 years ago
@raymondfeng @richardpringle
any thoughts?
@varunj90 It would be easier to debug this if I had an example of your remoteMethod, I am however taking a look. My initial guess would be that the content-type "application/pdf" is not supported by loopback
I will get back to you though
I have uploaded a snap of the remote method below. Looking forward to your response.
@varunj90 Have you tried returning a buffer object instead? I don't think strong-remoting recognizes the type 'file'. Also, what's the originalUrl
? could the cors.params.maxAge
parameter be cutting off the file transfer? Are you using loopback-component-storage?
I haven't played around with file transfers for a while, I will try to see if one of the other dev leads has a solid answer for you.
@richardpringle
originalUrl: /v1/documents/7424de78-1044-4ca4-b85f-c1de823b7235
We don't use the cors.params.maxAge paramter and loopback-component-storage.
Can you ask a developer who has worked on this to get back to us. We could try setting a webex to share screen and debug this to take this further.
Hey @varunj90, I appreciate your patience. I did a little bit of a deeper dive and talked to @bajtos as well. Here is the issue: remoteMethods have these rules for options (see the options section). The Type
of your return object is not a LoopBack Type, in other words type: 'file'
is not supported.
You could probably use loopback-component-storage to get the file transfer to work. You could also set something up on the client side to save raw buffer data if you return {type: 'buffer'}
.
I'm sorry, this is definitely not the response that you were hoping for, but you will have to make a 'feature request' to add that functionality to a regular LoopBack-model.
@richardpringle
thanks for your response. Do you have any examples in terms of setting this up using a buffer and also using loopback-component-storage?
@richardpringle @bajtos
I'm going to reiterate the problem: We are getting a pdf from a backend that we need to send to the client in a callback. Is there any way in loopback to perform this? We aren't sure if we are using the correct return type.
Looking at the code pasted above and the requirement : what are your recommendations?
Also setting up some time to view our code would be very beneficial to come to a conclusion/advice.
Thanks !
Here the module: LoopBack-component-storage Here is a tutorial (there is also a link to the example-repo within the doc).
As for setting up some time to view your code, we have a dedicated support channel for this purpose: support@strongloop.com. You can email them to set up a call with our support team.
Good luck!
@varunj90, Actually, is there any need to call the callback?
Can you do something like this?
model.send = function (res) {
res.send(pdf);
}
model.remoteMethod('send',
{
accepts: {arg: 'res', type: 'object', http: {source: 'res'}},
http: {verb: 'get'}
});
In the above case, I preloaded pdf
. You would have to modify my example to accept a docid
as well.
Let me know if that works for you.
Also, I would like to change the title of this issue to be more descriptive and label it as a feature request where you could use the callback: cb(null, pdf)
instead of directly calling res.send
.
I will wait for your response before I change the Title.
Zendesk ticket #1361 has been linked to this issue and is now the primary ticket.
@richardpringle - The reason why we need a callback is to call our middleware - which has our own error handler and adapter handler. If we simply do a res.send it might intercept it.
in terms of using res.send , we do something really similar there already (this works! just that we don't use callback here! ) :
Comment made from Zendesk by Jain, Varun on 2016-02-17 at 16:48:
Hi Joe,
Thanks for creating a new ticket and your quick response.
I have all the information as part of that thread. I have also attached screenshots of the remote method we are using right now and how it works well without a callback.
Question to you is: how to use the callback in a scenario like this? I see @richardpringle has questioned if we need a callback and I've responded why it would be nice to have a callback. Wondering if this is a limitation on loopback that needs a feature update.
Thanks,
Varun Jain I IT Solutions Developer | API Management and Build Out
79 Wellington Street West, 7th Floor, Toronto, ON, M5K 1A2
T: 416-307-3385 M: 647-987-1813
If you want to deal with file download as a remote method, use this as an example:
Comment made from Zendesk by Joe Simonsen on 2016-02-17 at 17:09:
Hello,
I asked another one of our Developers to look a this as well and he just commented on your issue. He pointed out a few code samples that could help deal with file downloads as a remote method, which are located here:
https://github.com/strongloop/loopback-component-storage/blob/master/lib/storage-service.js#L320 (remoting metadata)
https://github.com/strongloop/loopback-component-storage/blob/master/lib/storage-handler.js#L193 (implementation)
Joe Simonsen
Let me summarise the discussion above as I understand it:
1) @varunj90 is able to write a remote method returning a PDF file to download by using res.send
, using a similar approach as in our loopback-component-storage
2) However, when calling res.send
directly, cb()
is not called and thus "afterRemote" and "afterRemoteError" hooks are not triggered, therefore it's not possible to "call our own error handler and adapter handler"
In my opinion, the matter is clear: @varunj90 needs a new feature - support for Buffer
return arguments.
Here is an example code using the new feature as I am envisioning it:
Document.getDocumentById = function(id, cb) {
// load the document data as a Buffer, for example
// note that "content" will be a Buffer because we did not specify any encoding
fs.readFile(path.join(store, id), function(err, content) {
if (err) return cb(err);
cb(null, content);
});
};
Document.remoteMethod('getDocumentById', {
isStatic: true,
accepts: { arg: 'id', type: 'string', required: true },
returns: { arg: 'content', type: 'buffer', root: true, encoding: 'raw' }
http: { verb: 'get', path: '/:id' },
});
It is important to specify root:true
in returns
arg metadata, otherwise the result is wrapped in an object, e.g. { content: <value> }
(the key is the arg name).
In order to preserve backwards compatibility, I am proposing to add a new flag encoding: 'raw'
that would indicate that the Buffer value should be sent as-is. At the moment, we always serialise values to JSON, which is something we should preserve for backwards compatibility.
Here is a short overview of different responses depending on returns
flags based on my knowledge of the code. I did not run an example to verify my assumptions, so I may be incorrect in some places:
returns: { arg: 'content', type: 'Buffer' }
-> { "content": { "$type": "base64", "$data": "(base64-encoded content)" } }
returns: { arg: 'content', type: 'buffer', root: true }
-> { "$type": "base64", "$data": "(base64-encoded content)" }
returns: { arg: 'content', type: 'buffer', root: true, encoding: 'raw' }
-> (the exact buffer value)
@bajtos
Seems like you got it spot on. That is pretty much what we need and whats missing as a feature in loopback.
@bajtos I believe that we already support stream
as a return type. Buffer
is not necessarily a good idea as it has to keep all bytes in memory before writing to the response stream.
@ritch Can you confirm the stream
support?
@raymondfeng
Buffer is not necessarily a good idea as it has to keep all bytes in memory before writing to the response stream.
+1, I fully agree that passing around the full response in Buffer
is suboptimal. OTOH, I think it does not hurt to support both Stream
and Buffer
as callback "returns" argument, as the overhead in the terms of extra code lines is minimal.
I believe that we already support stream as a return type.
AFAIK, a remote method can return a stream, but only on the same tick, which makes it a bit more difficult to return a http response body stream created asynchronously.
myMethod = function() {
// easy
return fs.open('...');
};
myHttpMethod = function() {
request('http://example.com/file', function(err, res) {
// cannot simply return res :(
});
};
@bajtos
Going forward what is the solution to my problem then. I still am unable to to get the pdf file in the callback i.e. asynchronously.
Is this a new feature you have to develop? OR Are you prescribing a solution in the answers above that I didn't catch?
Is this a new feature you have to develop? OR Are you prescribing a solution in the answers above that I didn't catch?
It's a new feature we need to develop. I sent a pull request for peer review, see https://github.com/strongloop/strong-remoting/pull/284
@varunj90 I have released strong-remoting@2.26.0
which adds support for "file" arguments.
Here is an example method using the new feature:
Document.getDocumentById = function(id, cb) {
new IronMountain().getDocument(id, function(err, result) {
if (err) return cb(err);
var contentType = 'application/pdf';
var contentDisposition = 'attachment; filename=' + docid + '.pdf';
cb(null, result.body, contentType, contentDisposition);
});
}
Document.remoteMethod('getDocumentById', {
isStatic: true,
accepts: { arg: 'id', type: 'string', required: true },
returns: [
{ arg: 'body', type: 'file', root: true },
{ arg: 'Content-Type', type: 'string', http: { target: 'header' } },
{ arg: 'Content-Disposition', type: 'string', http: { target: 'header' } },
],
http: { path: '/:id', verb: 'get' }
});
@richardpringle I don't know if you need the "triage" label for book-keeping purposes, could you please eventually remove it?
@bajtos We don't bookkeep anymore for the triage process. He just forgot to remove it when escalating to feature. I will remove it now.
Thanks
@bajtos
this feature is part of which release/version of loopback?
the latest version of loopback :2.27.0 is using "strong-remoting": "^3.0.0-alpha.1", https://github.com/strongloop/loopback/blob/master/package.json
and i don't see the feature part of the release changes: https://github.com/strongloop/strong-remoting/blob/master/CHANGES.md
I'm asking you this question because strong remoting is used by loopback and therefore I need to know which is the right version to use.
@varunj90 see my earlier comment, the feature was implemented strong-remoting, which is a dependency of loopback.
I have released strong-remoting@2.26.0 which adds support for "file" arguments.
Unless you are locking down your dependencies (e.g. with npm-shrinkwrap), you can get the new version by running rm -rf node_modules/loopback && npm install
(for example).
hi @bajtos
the document that I am receiving is a Buffer.
response.body : 2016-02-29T21:43:36.464Z-Response Body - {"type":"Buffer","data":[37,80,68,70,45,49,46,52,10,37,199,236,143,162,10,53,32,48,32,111,98,106,10,60,60,47,76,101,110,103,116,104,32,54,32,48,32,82,47,70,105,108,116,101,114,32,47,70,108,97,116,101,68,101,99,111,100,101,62,62,10,115,116,114,101,97,109,10,120,156,77,141,203,10,194,48,16,69,209,106,213,81,252,134,89,166,139,142,201,36,109,147,173,32,130,59,75,118,214,85,197,130,80,161,245,255,193,62,20,157,187,57,92,184,103,26,148,164,24,101,159,47,148,53,236,242,12,171,23,12,53,230,199,15,180,21,52,96,73,247,55,20,255,92,214,184,247,221,208,161,35,151,162,191,131,36,231,172,204,70,169,66,182,140,153,98,98,244,53,136,201,52,152,5,243,112,17,249,7,196,204,150,140,193,120,92,222,224,34,150,81,172,72,106,35,173,8,63,104,140,88,253,48,136,82,201,148,104,119,245,39,96,109,200,36,140,113,218,137,164,237,21,162,120,174,55,97,209,110,251,7,7,15,231,46,111,17,2,45,42,101,110,100,115,116,114,101,97,109,10,101,110,100,111,98,106,10,54,32,48,32,111,98,106,10,49,55,50,10,101,110,100,111,98,106,10,52,32,48,32,111,98,106,10,60,60,47,84,121,112,101,47,80,97,103,101,47,77,101,100,105,97,66,111,120,32,91,48,32,48,32,54,49,50,32,55,57,50,93,10,47,82,111,116,97,116,101,32,48,47,80,97,114,101,110,116,32,51,32,48,32,82,10,47,82,101,115,111,117,114,99,101,115,60,60,47,80,114,111,99,83,101,116,91,47,80,68,70,32,47,84,101,120,116,93,10,47,69,120,116,71,83,116,97,116,101,32,49,48,32,48,32,82,10,47,70,111,110,116,32,49,49,32,48,32,82,10,62,62,10,47,67,111,110,116,101,110,116,115,32,53,32,48,32,82,10,62,62,10,101,110,100,111,98,106,10,51,32,48,32,111,98,106,10,60,60,32,47,84,121,112,101,32,47,80,97,103,101,115,32,47,75,105,100,115,32,91,10,52,32,48,32,82,10,93,32,47,67,111,117,110,116,32,49,10,47,82,111,116,97,116,101,32,48,62,62,10,101,110,100,111,98,106,10,49,32,48,32,111,98,106,10,60,60,47,84,121,112,101,32,47,67,97,116,97,108,111,103,32,47,80,97,103,101,115,32,51,32,48,32,82,10,62,62,10,101,110,100,111,98,106,10,55,32,48,32,111,98,106,10,60,60,47,84,121,112,101,47,69,120,116,71,83,116,97,116,101,10,47,79,80,77,32,49,62,62,101,110,100,111,98,106,10,49,48,32,48,32,111,98,106,10,60,60,47,82,55,10,55,32,48,32,82,62,62,10,101,110,100,111,98,106,10,49,49,32,48,32,111,98,106,10,60,60,47,82,57,10,57,32,48,32,82,62,62,10,101,110,100,111,98,106,10,49,50,32,48,32,111,98,106,10,60,60,47,76,101,110,103,116,104,49,32,52,55,49,54,47,70,105,108,116,101,114,47,70,108,97,116,101,68,101,99,111,100,101,47,76,101,110,103,116,104,32,49,51,32,48,32,82,62,62,115,116,114,101,97,109,10,120,156,237,87,125,112,84,213,21,63,247,190,119,95,246,51,239,190,151,221,205,110,72,216,221,132,208,216,128,9,217,108,62,32,146,5,18,226,132,175,240,21,136,178,66,136,32,169,32,81,32,13,132,17,130,162,105,24,52,150,5,5,167,18,109,171,66,68,192,58,66,100,166,3,186,84,74,163,82,72,24,167,164,124,76,67,117,104,51,19,58,213,12,31,251,210,243,118,3,140,83,219,50,227,31,254,81,238,217,123,223,61,231,190,189,247,156,223,57,247,220,119,129,0,128,21,54,130,0,21,51,102,103,229,64,180,40,6,108,42,107,86,84,215,197,120,57,4,64,42,107,234,87,123,38,117,77,107,65,65,15,214,127,46,173,123,108,197,253,51,79,10,56,118,10,249,148,199,150,175,93,26,123,95,200,5,136,203,88,182,164,250,209,207,154,218,175,3,168,200,67,222,50,20,152,31,97,207,225,251,253,200,143,88,182,98,117,195,208,122,139,176,121,111,249,202,154,234,
Error: Cannot create a file response from \"object\""
It's reaching line 644 in strong-remoting/lib/http-context.js and creating this error
Does this work with buffers? Any thoughts on what I'm doing wrong?
@varunj90 Do you have root:true
flag set for your type:'file'
callback ("returns") argument? What is your Node.js version? How do you create the buffer object?
Could you please write a small loopback app reproducing the problem, so that I can run it on my machine? See https://github.com/strongloop/strong-remoting/blob/a6a75fc86a429458294168d2751df9a013b85da3/test/rest.test.js#L2112-L2176 for inspiration.
@bajtos
Yes I do have that flag setup:
Node version is 0.10.39. Does the node version matter? Could you check for that.
Loopback Version: "name": "loopback", "version": "2.22.1"
The buffer object is created part of the body of the response of a server on the internet which send a pdf file. We are trying to use the result.body and then using the header application/pdf.
I won't paste the entire console.log - but here's a snippet related to the result.body
Thanks for your help @bajtos
@varunj90 thank you for the reply. To be honest, I don't know what's exactly happening in your application, I was asking for the Node version just as a speculative long shot.
It seems to me that your buffer gets converted to a JSON-like object, but that's even more confusing, because Buffer.prototype.toJSON()
returns simply an array on Node v0.10, while it returns a structure similar to what you see in your response in Node v4.x.
$ node -v
v0.10.42
$ node
> new Buffer([1,2,3]).toJSON()
[ 1, 2, 3 ]
## switch versions ##
$ node -v
v4.3.0
$ node
> new Buffer([1,2,3]).toJSON()
{ type: 'Buffer', data: [ 1, 2, 3 ] }
To add even more to the confusion, strong-remoting uses yet another encoding, see https://github.com/strongloop/loopback/issues/1907 for details.
Could you please modify the strong-remoting instance in your project's node_modules and add some logging code into SharedMethod.toResult
(lib/shared-method.js#L522-L578) to check if the buffer gets converted before it enters this method, or in this method, or sometime later?
@bajtos So I put some console logs in the code you mentioned:
Hopefully this helps in some way .....
/**
* Returns a reformatted Object valid for consumption as JSON from an Array of
* results from a remoting function, based on `returns`.
*/
SharedMethod.toResult = function(returns, raw, ctx) {
var result = {};
if (!returns.length) {
return;
}
returns = returns.filter(function(item, index) {
if (index >= raw.length) {
console.log('**********HERE 1 ***************');
return false;
}
if (ctx && ctx.setReturnArgByName(item.name || item.arg, raw[index])) {
console.log('**********HERE 2 ***************');
return false;
}
_ if (item.root) {
console.log('**********HERE 3 ***************');
var isFile = convertToBasicRemotingType(item.type) === 'file';
result = isFile ? raw[index] : convert(raw[index]);
console.log('**********HERE 3 - RESULT ***************', result);
return false;
}
_
return true;
});
returns.forEach(function(item, index) {
var name = item.name || item.arg;
if (convertToBasicRemotingType(item.type) === 'file') {
console.warn('%s: discarded non-root return argument %s of type "file"',
this.stringName,
name);
return;
}
var value = convert(raw[index]);
result[name] = value;
});
console.log('**********HERE 4 RESULT ***************', result);
return result;
function convert(val) {
switch (SharedMethod.getType(val)) {
case 'date':
return {
$type: 'date',
$data: val.toString()
};
case 'buffer':
return {
$type: 'base64',
$data: val.toString('base64')
};
}
return val;
}
};
It comes to the 'HERE 3' block :
Continues all the way to reach the error
Could you please log what's the returns
value when it enters SharedMethod.toResult
? Also for ITEM 3
, move it few lines down and include more data:
if (item.root) {
var isFile = convertToBasicRemotingType(item.type) === 'file';
console.log('**********HERE 3 ITEM %j IS FILE? %s ***************', item, is file);
result = isFile ? raw[index] : convert(raw[index]);
console.log('**********HERE 3 - RESULT ***************', result);
return false;
}
@varunj90 Would you mind creating a small app reproducing the problem, based on https://github.com/strongloop/loopback-sandbox? It would speed things up as I'll be able to debug this myself locally...
_returns: _
console.log('*****_HERE 3 ITEM %j IS FILE? %s ***', item, isFile) : *_
Comment made from Zendesk by Jain, Varun on 2016-03-02 at 16:27:
Hi there,
I have currently been trying to resolve this issue on git hub: https://github.com/strongloop/loopback/issues/2063
@bajtos (Miroslav Bajtoš) has been very responsive and I appreciate the help. But in order to get this resolved, I would like to instantiate a webex meeting with him so that we can do some live coding and get this issue resolved.
Please let me know of the next steps.
Thanks,
Varun Jain
IT Solutions Developer | API Management and Build Out | EETS
79 Wellington Street West, 7th Floor, Toronto, ON, M5K 1A2
T: 416-307-3385 M: 647-987-1813
Sorry, I forgot to ask about one more thing - what's raw[index]
?
@bajtos
if (item.root) {
//console.log('**********HERE 3 ***************');
var isFile = convertToBasicRemotingType(item.type) === 'file';
//console.log('**********HERE 3 ITEM %j IS FILE? %s ***************', item, isFile);
console.log('---------RAW INDEX------------',raw[index] )
result = isFile ? raw[index] : convert(raw[index]);
---------RAW INDEX------------ { type: 'Buffer', data: [ 37, 80, 68, 70, 45, 49, 46, 52, 10, 37, 199, 236, 143, 162, 10, 53, 32, 48, 32, 111, 98, 106, 10, 60, 60, 47, 76, 101, 110, 103, 116, 104, 32, 54, 32, 48, 32, 82, 47, 70, .........................
Comment made from Zendesk by Joe Simonsen on 2016-03-03 at 17:34:
Hello,
I spoke with Miroslav and he can be available for a webex Monday March 7th at 9am EST. Will this time work for you?
Joe Simonsen
@Joesimonsen @bajtos
I never got a response for the webex meeting. Can we schedule it for sometime this week at 10am EST. Please provide the details.
@varunj90 I tried to confirm the time with you via your support ticket, but got no reply there.
@Joesimonsen
please check the support ticket - i replied thursday friday last week and today:
@Joesimonsen @bajtos
I never got a response for the webex meeting. Can we schedule it for sometime this week at 10am EST. Please provide the details.
Varun Jain IT Solutions Developer | API Management and Build Out | EETS 79 Wellington Street West, 7th Floor, Toronto, ON, M5K 1A2 T: 416-307-3385 M: 647-987-1813
From: Jain, Varun _Sent: Friday, March 04, 2016 1:00 PM _To: 'strongloop/loopback'; 'strongloop/loopback' Subject: RE: [loopback] Issue downloading pdf; file is corrupted (#2063)
Can you confirm if we are going ahead with Monday at 10am EST?
Also can you provide the webex details and phone number to dial.
Thanks,
Varun Jain IT Solutions Developer | API Management and Build Out | EETS 79 Wellington Street West, 7th Floor, Toronto, ON, M5K 1A2 T: 416-307-3385 M: 647-987-1813
From: Jain, Varun _Sent: Thursday, March 03, 2016 12:46 PM _To: 'strongloop/loopback'; strongloop/loopback Subject: RE: [loopback] Issue downloading pdf; file is corrupted (#2063)
Thanks for reaching out Joe.
Can we do 10am EST?
Thanks
Varun Jain IT Solutions Developer | API Management and Build Out | EETS 79 Wellington Street West, 7th Floor, Toronto, ON, M5K 1A2 T: 416-307-3385 M: 647-987-1813
Comment made from Zendesk by Joe Simonsen on 2016-03-07 at 20:05:
Hello,
I do not see your last update here. The last message I see is my message trying to confirm the webex time. Can you confirm you see this message?
Joe Simonsen
Comment made from Zendesk by Jain, Varun on 2016-03-07 at 20:06:
Yes I do see this message.
Can we book the slot and you can send me the details?
Thanks,
Varun Jain
IT Solutions Developer | API Management and Build Out | EETS
79 Wellington Street West, 7th Floor, Toronto, ON, M5K 1A2
T: 416-307-3385 M: 647-987-1813
Comment made from Zendesk by Joe Simonsen on 2016-03-07 at 20:12:
Let me confirm with our developer that the time will work and I will book it.
Joe Simonsen
@Joesimonsen - were you able to find a time slot?
Comment made from Zendesk by Joe Simonsen on 2016-03-08 at 21:48:
Hello,
Just to update you we are still looking into this and I am trying to find a time that works for our developers for a webex.
Joe Simonsen
@Joesimonsen - any updates? It's been a week.
Comment made from Zendesk by Jain, Varun on 2016-03-16 at 17:49:
Hi again,
When can you set up some time?
Varun Jain
IT Solutions Developer | API Management and Build Out | EETS
79 Wellington Street West, 7th Floor, Toronto, ON, M5K 1A2
T: 416-307-3385 M: 647-987-1813
Yes I do see this message.
Can we book the slot and you can send me the details?
Thanks,
Varun Jain
IT Solutions Developer | API Management and Build Out | EETS
79 Wellington Street West, 7th Floor, Toronto, ON, M5K 1A2
T: 416-307-3385 M: 647-987-1813
From: StrongSupport [mailto:support@strongloop.zendesk.com]
Sent: Monday, March 07, 2016 3:05 PM
To: Jain, Varun
Subject: [StrongSupport] Re: Issue downloading pdf; file is corrupted
Comment made from Zendesk by Joe Simonsen on 2016-03-16 at 18:07:
Hello,
Apologies for the delay here. Our main developer in this area is currently out sick and should return on the 18th. I will speak with him again when he is back to set something up as soon as possible.
Apologies again for the delay here.
Joe Simonsen
Hi,
I am receiving a pdf file from the server on the internet. This is in binary format and as soon as I try adding it in a callback, the file gets corrupted/damaged. Whereas if I simply return the file without a cb, it gets downloaded.
_var options = { url: config.baseURL + config.getDocument.url + docId, encoding: null, method: 'GET', qs: {'a': true, 'alf_ticket': ticket} };
// send out request request(options, function(error, response, body) {onGetDocumentResponse(error, response, body, callback);}); } _
In my remote method when I do the following
_ params.res.setHeader('Content-Type', 'application/pdf'); params.res.setHeader('Content-Disposition', 'attachment; filename=' + params.docid + '.pdf');
It downloads the file perfectly, but as soon I introduce callback
cb(null,result.body); The pdf received is damaged (size is 195 bytes vs 4kb of original file)
I've read on multiple threads of having issues in downloading pdfs in node. Is there any solution that you could recommend which would allow us to use a callback instead of a return statement (since that would intercept all middleware)
@raymondfeng @richardpringle