search

bhushankummar / amazon-mws

Amazon MWS NodeJS Wrapper
MIT License
197 stars 78 forks source link

Huge Reports Issue #104

Closed beastmike1 closed 5 years ago

beastmike1 commented 5 years ago

Hi, we are having troubles processing some reports from sellers with a huge catalogue, like 600k products for each marketplace.

For example, an open listing report from one of that sellers after been parsed for your library and saved to a txt file, it weighs more than 200 mb.

I think that a way to pipe the report result directly to a file it could be better than caching all the response in a variable in that cases, would that be possible?

Thanks.

marcospampi commented 5 years ago

The http stream isn't accessible by the current interface, but is in lib/AmazonMwsResource.js As for now you can use the pseudo-argument __RAW__ that will return the blob of it. You could expose the stream by adding another pseudo-argument, as I did back then for the blob one.

beastmike1 commented 5 years ago

To do it in a streaming fashion as said in java examples in AmazonDocumentation

in Reports -> GetReportSample.java

// Note that depending on the type of report being downloaded, a report can reach // sizes greater than 1GB. For this reason we recommend that you always program to // MWS in a streaming fashion. Otherwise, as your business grows you may silently reach // the in-memory size limit and have to re-work your solution.

jesusvalle commented 5 years ago

@bhushankumarl, @marcospampi, this will be so interesting for my project too.