scosman / zipstreamer

Zip File Streaming Microservice - stream zip files on the fly
MIT License
119 stars 18 forks source link
golang microservice on-the-fly serverless streaming streaming-server zip zip-file

Zipstreamer Logo

Build and Test Format and Vet Docker Generation Go Report Card Go Reference

ZipStreamer is a golang microservice for streaming zip files from a series of web links, on the fly. For example, if you have 200 files on S3, and you want to download a zip file of them, you can do so in 1 request to this server.

Highlights include:

Content

JSON Zip File Descriptor

Each HTTP endpoint requires a JSON description of the desired zip file. It includes a root object with the following structure:

Example JSON description with 2 files:

{
  "suggestedFilename": "tps_reports.zip",
  "files": [
    {
      "url":"https://server.com/image1.jpg",
      "zipPath":"image1.jpg"
    },
    {
      "url":"https://server.com/image2.jpg",
      "zipPath":"in-a-sub-folder/image2.jpg"
    }
  ]
}

HTTP Endpoints

POST /download

This endpoint takes a http POST body containing the JSON zip file descriptor, and returns a zip file.

Example usage with curl #### Example curl usage of `POST /download` endpoint ``` # download a sample json descriptor curl https://gist.githubusercontent.com/scosman/f57a3561fed98caab2d0ae285a0d7251/raw/4a9630951373e50f467f41d8c7b9d440c13a14d2/zipJsonDescriptor.json > zipJsonDescriptor.json # call POST /download endpoint, passing json descriptor in body curl --data-binary "@./zipJsonDescriptor.json" http://localhost:4008/download > archive.zip ```

GET /download

This endpoint fetches a JSON zip file descriptor hosted on another server, and returns a zip file. This is useful over the POST /download endpoint for a few use cases:

This endpoint requires one of two query parameters describing where to find the JSON zip file descriptor:

Example usage with curl #### Example curl usage of `GET /download` endpoint with zsurl parameter ``` curl -X GET "http://localhost:4008/download?zsurl=https://gist.githubusercontent.com/scosman/f57a3561fed98caab2d0ae285a0d7251/raw/4a9630951373e50f467f41d8c7b9d440c13a14d2/zipJsonDescriptor.json" > archive.zip ``` #### Example curl usage of `GET /download` endpoint with zsid parameter ``` # start server with ZS_LISTFILE_URL_PREFIX ZS_LISTFILE_URL_PREFIX="https://gist.githubusercontent.com/scosman/" ./zipstreamer ``` ``` # call `GET /download` endpoint with zsid curl -X GET "http://localhost:4008/download?zsid=f57a3561fed98caab2d0ae285a0d7251/raw/4a9630951373e50f467f41d8c7b9d440c13a14d2/zipJsonDescriptor.json" > archive.zip ```

POST /create_download_link

This endpoint takes a http POST body containing the JSON zip file descriptor, stores it in a local cache, and returns a link ID which allows the caller to fetch the zip file via an additional call to GET /download_link/{link_id}.

This is useful for if you want to trigger a browser "Save File" UI, which isn't shown for POST requests. See GET /download for a server side alternative to achieve this.

Important:

Here is an example response body containing the link ID. See docs for GET /download_link/{link_id} below for how to fetch this zip file:

{
  "status":"ok",
  "link_id":"b4ecfdb7-e0fa-4aca-ad87-cb2e4245c8dd"
}

Example usage: see GET /download_link/{link_id} documentation below.

GET /download_link/{link_id}

Call this endpoint with a link_id generated with /create_download_link to download that zip file.

Example usage with curl #### Example curl usage of `POST /create_download_link` and `GET /download_link/{link_id}` endpoints working together ``` # download a sample json descriptor curl https://gist.githubusercontent.com/scosman/f57a3561fed98caab2d0ae285a0d7251/raw/4a9630951373e50f467f41d8c7b9d440c13a14d2/zipJsonDescriptor.json > zipJsonDescriptor.json # call POST endpoint to create link curl --data-binary "@./zipJsonDescriptor.json" http://localhost:4008/create_download_link # Call GET endpoint to download zip. Note: must copy UUID from output of above POST command into this URL curl -X GET "http://localhost:4008/download_link/UUID_FROM_ABOVE" > archive.zip ```

Deploy

Heroku - One Click Deploy

Deploy

Be sure to enable session affinity if you're using multiple servers and using /create_download_link.

Google Cloud Run - One Click Deploy, Serverless

Run on Google Cloud

Cloud Run is ideal serverless environment for ZipStreamer, as it routes many requests to a single container instance. ZipStreamer is designed to handle many concurrent requests, and will be cheaper to run on this serverless architecture than a instance-per-request architecture like AWS Lamba or Google Cloud Functions.

Important

Docker

This repo contains an dockerfile, and an image is published on Github Packages.

Build Your Own Image

To build your own image, clone the repo and run:

docker build --tag docker-zipstreamer .
# Start on port 8080
docker run --env PORT=8080 -p 8080:8080 docker-zipstreamer

Run Official Package from Github Packages

Official packages are published on Github packages. To pull latest stable release:

docker pull ghcr.io/scosman/packages/zipstreamer:stable
# Start on port 8080
docker run --env PORT=8080 -p 8080:8080 ghcr.io/scosman/packages/zipstreamer:stable

Note: stable pulls the latest github release. Use ghcr.io/scosman/packages/zipstreamer:latest for top of tree.

Configuration Options

These environment variables can be used to configure the server:

Why

I was mentoring at a "Teens Learning Code" class, but we had too many mentors, so I had some downtime.

Logo

Zipper portion of logo by Kokota from Noun Project (Creative Commons CCBY)