search

ardriveapp / ardrive-cli

The ArDrive Command Line Interface (CLI) is a Node.js application for terminal-based ArDrive workflows. It also offers utility operations for securely interacting with Arweave wallets and inspecting various Arweave blockchain conditions.
GNU Affero General Public License v3.0
78 stars 14 forks source link

readme

ardrive-cli

The ArDrive Command Line Interface (CLI) is a Node.js application for terminal-based [ArDrive] workflows. It also offers utility operations for securely interacting with Arweave wallets and inspecting various [Arweave] blockchain conditions.

Create your first drive and permanently store your first file on the permaweb with a series of simple CLI commands like so:

ardrive create-drive --wallet-file /path/to/my/wallet.json --drive-name "Teenage Love Poetry"
{
    "created": [
        {
            "type": "drive",
            "metadataTxId": "giv2R8Xj0bbe6l5taBTQJk_38zwIrMH_g1-knSCisjU",
            "entityId": "898687ea-b678-4f86-b4e7-49560b190356",
            "bundledIn": "Vj2x4IBEAezBvhj5RgtA247W_q3S10suI6l0E30GPoE",
            "entityName": "Teenage Love Poetry"
        },
        {
            "type": "folder",
            "metadataTxId": "VljnttwUxRStnVuPYakF9e2whjhYJVWB0nSxD5dVyJ8",
            "entityId": "f0c58c11-430c-4383-8e54-4d864cc7e927",
            "bundledIn": "Vj2x4IBEAezBvhj5RgtA247W_q3S10suI6l0E30GPoE",
            "entityName": "Teenage Love Poetry"
        },
        {
            "type": "bundle",
            "bundleTxId": "Vj2x4IBEAezBvhj5RgtA247W_q3S10suI6l0E30GPoE"
        }
    ],
    "tips": [],
    "fees": {
        "Vj2x4IBEAezBvhj5RgtA247W_q3S10suI6l0E30GPoE": "44579472"
    }
}

ardrive upload-file --wallet-file /path/to/my/wallet.json --parent-folder-id "f0c58c11-430c-4383-8e54-4d864cc7e927" --local-path ./helloworld.txt --dest-file-name "ode_to_ardrive.txt"
{
    "created": [
        {
            "type": "file",
            "entityName": "ode_to_ardrive.txt",
            "entityId": "bd2ce978-6ede-4b0d-8f79-2d7bc235a0e0",
            "dataTxId": "tSMcfvAQu_tKLUkdvRRbqdX93oAf3h6c9eJsSj8mXL4",
            "metadataTxId": "EvE06MmE9IKeUzFMnxSgY1M5tJX4uHU64-n8Pf_lZfU",
            "bundledIn": "qjdHiQoWlSjCvhj5RgtA247W_q3S10suI6l0E30GPoE",
            "sourceUri": "file://Users/BestArDriver/Uploads/helloworld.txt"
        },
        {
            "type": "bundle",
            "bundleTxId": "qjdHiQoWlSjCvhj5RgtA247W_q3S10suI6l0E30GPoE"
        }
    ],
    "tips": [
        {
            "txId": "qjdHiQoWlSjCvhj5RgtA247W_q3S10suI6l0E30GPoE",
            "recipient": {
                "address": "i325n3L2UvgcavEM8UnFfY0OWBiyf2RrbNsLStPI73o"
            },
            "winston": "10000000"
        }
    ],
    "fees": {
        "qjdHiQoWlSjCvhj5RgtA247W_q3S10suI6l0E30GPoE": 44579472
    }
}

This project is in a state of active development. Use at your own risk!

Table of Contents

  1. ArDrive
    1. ArFS
    2. Data Portability
    3. Intended Audience
  2. Getting Started
    1. Prerequisites
      1. Git
      2. NVM (Optional - Recommended)
    2. Quick Start
      1. Installing and Starting the CLI From NPM Package
      2. Upgrading to the Latest CLI Version
    3. Build and Run from Source
      1. Install Yarn 3
      2. Husky (Developers Only)
      3. Using a custom ArDrive-Core-JS (Optional)
      4. Installing and Starting the CLI From Source
      5. Recommended Visual Studio Code extensions (Developers Only)
    4. Limitations
  3. Using the CLI
    1. CLI Help
    2. CLI Version
    3. Wallet Operations
    4. Working With Entities
      1. Dry Run
      2. Uploading to Turbo (BETA)
    5. Working With Drives
      1. Understanding Drive Hierarchies
        1. Fetching Drive Info
      2. Understanding Drive and File Keys
        1. Derive a Drive Key
        2. Derive a File Key
      3. Managing Drive Passwords
        1. Supplying Your Password: Environment Variable
        2. Supplying Your Password: STDIN
        3. Supplying Your Password: Prompt
      4. Creating Drives
      5. Listing Drives for an Address
      6. Listing Every Entity in a Drive
      7. List Drive Pipeline Examples
        1. Get Share Links for Files in the Drive
        2. Get Total Size of Files in the Drive
        3. Get Total Count of Files in the Drive
    6. Working With Folders
      1. Creating Folders
      2. Moving Folders
      3. Renaming Folders
      4. Viewing Folder Metadata
      5. Listing Contents of a Folder
    7. Working With Files
      1. Uploading a Single File
      2. Download a Single File (BETA)
      3. Rename a Single File
      4. Uploading a Folder with Files
      5. Progress Logging of Transaction Uploads
      6. Downloading a Folder with Files
      7. Downloading a Drive
      8. Uploading Multiple Files
      9. Name Conflict Resolution on Upload
      10. Understanding Bundled Transactions
      11. Uploading a Non-Bundled Transaction
      12. Fetching the Metadata of a File Entity
      13. Retrying a Failed File Data Transaction (Public Unbundled Files Only)
      14. Moving Files
      15. Uploading Manifests
      16. Hosting a Webpage with Manifest
      17. Uploading With a Custom Content Type
      18. Uploading From a Remote URL
      19. Uploading a Custom Manifest
      20. Uploading Files with Custom MetaData
      21. Applying Unique Custom MetaData During Bulk Workflows
    8. Other Utility Operations
      1. Monitoring Transactions
      2. Dealing With Network Congestion
      3. Check for network congestion before uploading
      4. Front-run Congestion By Boosting Miner Rewards
      5. Send AR Transactions From a Cold Wallet
      6. Using a Custom Arweave Gateway
      7. Persistent Caching of ArFS Entity Metadata
  4. All ArDrive CLI Commands
  5. Getting Help

ArDrive

[ArDrive] is a permanent storage platform whose [applications and core libraries][ardrive-github] offer hierarchical organization, privacy via complete end-to-end encryption, flexibility, extensibility, and access control over your most valuable data, all made possible by its innovative core technology, the [Arweave File System (ArFS) Protocol][arfs].

ArFS

[ArFS] is a data modeling, storage, and retrieval protocol designed to emulate common file system operations and to provide aspects of mutability to your data hierarchy on [Arweave]'s otherwise permanent, immutable data storage blockweave.

Data Portability

Data uploaded via the ArDrive CLI, once indexed by Arweave's Gateways and sufficiently seeded across enough nodes on the network, can be accessed via all other ArDrive applications including the [ArDrive Web application][ardrive-web-app] at https://app.ardrive.io.

All transactions successfully executed by ArDrive can always be inspected in the [Viewblock blockchain explorer].

Intended Audience

This tool is intended for use by:

For deeper integrations with the [ArDrive] platform, consider using the [ArDrive Core][ardrive-core] (Node) library's configurable and intuitive class interfaces directly within your application.

Getting Started

To simply install the latest version of the CLI to your local system and get started, follow the Quick Start instructions. To build and/or develop the CLI from source, follow the Build and Run from Source instructions. In either case, be sure to satisfy the requirements in the Prerequisites section.

Prerequisites

The following tools must be installed whether performing a Quick Start or Building from Source:

Git

Some of ArDrive's dependencies are transitively installed via Git. Install it, if necessary, and ensure that it's available within your terminal environment:

Download Git

NVM (Optional - Recommended)

This project uses the Node Version Manager (NVM) and an .nvmrc file to lock the recommended Node version used by the latest version of ardrive-core-js.

Note for Windows: We recommend using WSL for setting up NVM on Windows using the [instructions described here][wsl-install]

Follow these steps to get NVM up and running on your system:

  1. Install NVM using [these installation instructions][nvm-install].
  2. Navigate to this project's root directory
  3. Ensure that the correct version of Node is installed by performing: nvm install
  4. Use the correct version of Node, by performing: nvm use

IT IS STRONGLY RECOMMENDED THAT YOU AVOID GENERATING WALLETS VIA SEED PHRASE WITH THE CLI USING ANY NODE VERSION OTHER THAN THE ONE SPECIFIED IN .nvmrc.

Quick Start

Once you've satisfied any necessary prerequisites, the fastest way to get up and running is to globally install the latest version of the ArDrive CLI to your local system via NPM:

Installing and Starting the CLI From NPM Package

npm install -g ardrive-cli

# then invoke the CLI from anywhere on your system:
ardrive

Upgrading to the Latest Version

If you globally installed the CLI via NPM, then upgrade to the latest version by simply performing:

npm update -g ardrive-cli

Build and Run from Source

Install Yarn 3

Both the ArDrive CLI and ArDrive Core JS use Yarn 3 to manage dependencies and initiate workflows, so follow the [yarn installation instructions][yarn-install] in order to get the latest version. In most cases:

# Brew (OSX):
brew install yarn

# Or with NPM (all supported platforms):
npm install -g yarn

Husky (Developers Only)

We use husky 6.x to manage the git commit hooks that help to improve the quality of our commits. Please run:

yarn husky install

to enable git hooks for your local checkout. Without doing so, you risk committing non-compliant code to the repository.

Using a custom ArDrive-Core-JS (Optional)

To test a with a custom version of the ardrive-core-js library on your local system, change the "ardrive-core-js" line in package.json to the root of your local ardrive-core-js repo:

- "ardrive-core-js": "1.0.0"
+ "ardrive-core-js": "../ardrive-core-js/"

Installing and Starting the CLI From Source

Now that your runtime and/or development environment is set up, to install the package simply run:

yarn && yarn build

And then start the CLI (always from the root of this repository):

yarn ardrive

For convenience in the non-developer case, you can install the CLI globally on your system by performing the following step:

yarn pack

# then using the path generated by yarn from the step above:
npm install i -g /path/to/package.tgz

# then invoke the CLI from anywhere on your system:
ardrive

Recommended Visual Studio Code extensions (Developers Only)

To ensure your environment is compatible, we also recommend the following VSCode extensions:

Limitations

Number of files in a bulk upload: Theoretically unlimited
Max individual file size: 2GB (Node.js limitation)
Max file name length: 255 bytes
Max ANS-104 bundled transaction size: 500 MiB per bundle. App will handle creating multiple bundles.
Max ANS-104 data item counts per bundled transaction: 250 Files per bundle (500 Data Items).

Using the CLI

CLI Help

Learn to use any command:

ardrive --help

CLI Version

You can print out the version by running any of:

ardrive --version
ardrive -V

Wallet Operations

Browsing of ArDrive public data is possible without the need for an [Arweave wallet][kb-wallets]. However, for all write operations, or read operations without encryption/decryption keys, you'll need a wallet.

As you utilize the CLI, you can use either your wallet file or your seed phrase interchangeably. Consider the security implications of each approach for your particular use case carefully. If at any time you'd like to generate a new wallet altogether, start by generating a new seed phase. And if you'd like to use that seed phrase in the form of a wallet file, or if you'd like to recover an existing wallet via its seed phrase, use either or both of the following commands:

# Generate seed-phrase
ardrive generate-seedphrase
"this is an example twelve word seed phrase that you could use"

# Generate/recover wallet file (with example output file path)
ardrive generate-wallet -s "this is an example twelve word seed phrase that you could use" > /path/to/wallet/file.json

Public attributes of Arweave wallets can be retrieved via their 43-character Arweave wallet address. You can retrieve the wallet address associated with [your wallet file or 12-word seed phrase][kb-wallets] (e.g. wallets generated by [ArConnect][arconnect]) like so:

# Wallet file
ardrive get-address -w /path/to/wallet/file.json

# Seed Phrase (with sample output)
ardrive get-address -s "this is an example twelve word seed phrase that you could use"
HTTn8F92tR32N8wuo-NIDkjmqPknrbl10JWo5MZ9x2k

You'll need AR in your wallet for any write operations you perform in ArDrive. You can always check your wallet balance (in both AR and Winston units) by performing:

# Getting the balance for your own wallet
ardrive get-balance -w /path/to/wallet/file.json

# Getting the balance for ANY wallet (with sample output)
ardrive get-balance -a "HTTn8F92tR32N8wuo-NIDkjmqPknrbl10JWo5MZ9x2k"
1500000000000 Winston
1.5 AR

If, at any time, you need to send AR out of your wallet to another wallet address, you may perform:

# Using our previously generated wallet as the destination...
ardrive send-ar -w /path/to/wallet/file.json --dest-address "HTTn8F92tR32N8wuo-NIDkjmqPknrbl10JWo5MZ9x2k" --ar-amount 2.12345

Working With Entities

[ArDrive]'s [ArFS] integration provides for hierarchical organization of your file and folder data on Arweave.

The fundamental entity types specified by ArFS are:

Each instance of these entities have a Version 4 UUID entity ID that is commonly referred to by its entity type, i.e. drive ID, folder ID, and file ID.

When you execute write functions with the CLI, the JSON output will contain information about the Arweave Transaction IDs that were registered when writing your entities to the blockweave, any miner rewards or ArDrive Community tips that were disbursed from your wallet, and any new entity IDs and, when applicable, encryption keys that were generated in the process of creating the entities. Typically, you'll want to keep track of those and get proficient with retrieving them in order to build your drive hierarchy to your liking. See Understanding Drive and File Keys for more info.

Dry Run

An important feature of the ArDrive CLI is the --dry-run flag. On each command that would write an ArFS entity, there is the option to run it as a "dry run". This will run all of the steps and print the outputs of a regular ArFS write, but will skip sending the actual transaction:

ardrive <my-command> <other-options> --dry-run

This can be very useful for gathering price estimations or to confirm that you've copy-pasted your entity IDs correctly before committing to an upload.

Uploading to Turbo (BETA)

Users can optionally choose to send each ArFS entities created to [ArDrive Turbo][ardrive-turbo] using the --turbo flag. Instead of using AR from an Arweave wallet, you can use Turbo Credits or take advantage of free/discounted upload promotions.

ardrive <my-command> <other-options> --turbo

This flag will skip any balance check on the CLI side. Turbo will check a user's balance and accept/reject a data item at the time of upload. The --turbo flag by default will send your files to upload.ardrive.io to be bundled. To change the Turbo destination, users can use the --turbo-url flag.

Working With Drives

Understanding Drive Hierarchies

At the root of every data tree is a "Drive" entity. When a drive is created, a Root Folder is also created for it. The entity IDs for both are generated and returned when you create a new drive:

# Use `tee` to keep a receipt of the full set of transactions info and `jq` to focus on the data of interest
ardrive create-drive --wallet-file /path/to/my/wallet.json --drive-name "Teenage Love Poetry" |
tee created_drive.json |
jq '[.created[] | del(.metadataTxId, .entityName, .bundledIn)]'
[
    {
        "type": "drive",
        "entityId": "6939b9e0-cc98-42cb-bae0-5888eca78885"
    }
    {
        "type": "folder",
        "entityId": "d1535126-fded-4990-809f-83a06f2a1118"
    }
]

The relationship between the drive and its root folder is clearly visible when retrieving the drive's info:

ardrive drive-info -d "6939b9e0-cc98-42cb-bae0-5888eca78885"
| jq '{driveId, rootFolderId}'
{
    "driveId": "6939b9e0-cc98-42cb-bae0-5888eca78885",
    "rootFolderId": "d1535126-fded-4990-809f-83a06f2a1118"
}

All file and folder entities in the drive will be anchored to it by a "Drive-ID" GQL Tag. And they'll each be anchored to a parent folder ID, tracked via the "Parent-Folder-ID" GQL tag, forming a tree structure whose base terminates at the Root Folder.

Understanding Drive and File Keys

Private Drives achieve privacy via end-to-end encryption facilitated by hash-derived "Keys". Drive Keys encrypt/decrypt Drive and Folder data, and File Keys encrypt/decrypt File Data.

The relationships among your data and their keys is as follows:

When you create private entities, the returned JSON data from the ArDrive CLI will contain the keys needed to decrypt the encrypted representation of your entity that is now securely and permanently stored on the blockweave.

To derive the drive key again for a drive, perform the following:

# Will throw an error if the wallet or password specified can't be used to decrypt the on-chain drive
ardrive get-drive-key -w /path/to/my/wallet.json -d "6939b9e0-cc98-42cb-bae0-5888eca78885" -P

To derive the file key again for a file, perform the following:

# Will throw an error if the drive key or drive-key-derivation data specified can't be used to decrypt the on-chain file
ardrive get-file-key --file-id "bd2ce978-6ede-4b0d-8f79-2d7bc235a0e0" --drive-id "6939b9e0-cc98-42cb-bae0-5888eca78885" --drive-key "yHdCjpCK3EcuhQcKNx2d/NN5ReEjoKfZVqKunlCnPEo"

Managing Drive Passwords

The ArDrive CLI's private drive and folder functions all require either a drive password OR a drive key. Private file functions require either the drive password or the file key. Keys and passwords are sensitive data, so manage the entry, display, storage, and transmission of them very carefully.

Drive passwords are the most portable, and fundamental, encryption facet, so a few options are available during private drive operations for supplying them:

Supplying Your Password: Environment Variable

# Securely type your password into a read prompt, store it to TMP_ARDRIVE_PW, and export it for the shell session
read -rs TMP_ARDRIVE_PW
export ARDRIVE_DRIVE_PW=$(TMP_ARDRIVE_PW)
ardrive <some private command> -w /path/to/wallet.json -P

Supplying Your Password: STDIN

# Pipe your drive password to the ArDrive CLI
cat /path/to/my/drive/password.txt | ardrive <some private command> -w /path/to/wallet.json -P

# Redirect your drive password to the ArDrive CLI
ardrive <some private command> -w /path/to/wallet.json -P < /path/to/my/drive/password.txt

Supplying Your Password: Prompt

# When all other options fail, the CLI will prompt for your password (NOT COMPATIBLE WITH PIPES AND REDIRECTS!)
ardrive <some private command> -w /path/to/wallet.json -P
? Enter drive password: › ********

Creating Drives

# Public drive
ardrive create-drive --wallet-file /path/to/my/wallet.json --drive-name "My Public Archive"

# Private drive
ardrive create-drive --wallet-file /path/to/my/wallet.json --drive-name "Teenage Love Poetry" -P

Listing Drives for an Address

You can list all the drives associated with any Arweave wallet address, though the details of private drives will be obfuscated from you unless you provide the necessary decryption data.

# List all your own drives
ardrive list-all-drives -w /path/to/my/wallet.json -P

# List any address's drives
ardrive list-all-drives --address "HTTn8F92tR32N8wuo-NIDkjmqPknrbl10JWo5MZ9x2k"

Listing Every Entity in a Drive

Useful notes on listing the contents of drives: