search

apify / apify-cli

Apify command-line interface helps you create, develop, build and run Apify actors, and manage the Apify cloud platform.
https://docs.apify.com/cli
122 stars 18 forks source link

Add command to download dataset #390

Closed drobnikj closed 1 month ago

drobnikj commented 1 year ago

Users are struggling with downloading large datasets.

We need to introduce new command, that handle downloading dataset for them. see slack https://apifier.slack.com/archives/C010Q0FBYG3/p1696943625090939

jancurn commented 1 year ago

Pls let's discuss the syntax before we implement it, to make sure it's future proof and consistent with other commands. For example:

apify download-dataset [ID_OR_NAME] --format csv --limit 100 --offset 50 --output /tmp/file.csv ...

We can start with fewer options and add more later

fnesveda commented 1 year ago

Let's check if we have some analytics which formats people use the most, but I think for starters it would be enough to support CSV, JSON and JSONL.

Also we could take the format transformation code that we have on the platform, put it in apify-shared and use it in CLI from there.

jancurn commented 7 months ago

Once again we stumbled upon lack of this command, see https://apify.slack.com/archives/C06819MJN9Z/p1710923614723909?thread_ts=1710831604.067589&cid=C06819MJN9Z