Can't load dataset items in total size > 512MB

apify / apify-client-js

Apify API client for JavaScript / Node.js.

https://docs.apify.com/api/client/js

Apache License 2.0

67 stars 27 forks source link

Can't load dataset items in total size > 512MB #247

Open valekjo opened 2 years ago

valekjo commented 2 years ago

Node has 512MB limit on string size, if the response from API is larger, the client fails to parse it.

I guess the issue might also manifest in key-value-stores? Other api endpoints probably never produce this large responses.

Reported on slack: https://apifier.slack.com/archives/C0L33UM7Z/p1652976377122799

mnmkng commented 2 years ago

I don't think this is a bug really. It's quite obvious that strings of unlimited sizes cannot be loaded or parsed. That's why the list functions of client have offsets and limits and the KVS functions have stream options.

I would vote for closing this.

fnesveda commented 2 years ago

@valekjo Where did this bug manifest? Could you link the Slack thread or another discussion where this was reported?

fnesveda commented 2 years ago

It would also be cool if we supported some client.paginate() function, which automatically iterates over the pages in a list returned by the API, like Octokit does it, but that's not really related to this issue specifically.