Cloud storage abstraction package for Go.
Version: 0.1.0
Project status: Stable. Approaching v1 release
It is recommended that you vendor this package as changes are possible before v1
Contributors: Mat Ryer, David Hernandez, Ernesto Jiménez, Corey Prak, Piotr Rojek, Jason Hancock
Artwork by Mel Jensen inspired by Renee French and and is licensed under the Creative Commons Attribution 3.0 License
Stow provides implementations for storage services, blob stores, cloud storage etc. Read the blog post announcing the project.
The concepts of Stow are modeled around the most popular object storage services, and are made up of three main objects:
Location
- a place where many Container
objects are storedContainer
- a named group of Item
objectsItem
- an individual filelocation1 (e.g. Azure)
├── container1
├───── item1.1
├───── item1.2
├───── item1.3
├── container2
├───── item2.1
├───── item2.2
location2 (e.g. local storage)
├── container1
├───── item1.1
├───── item1.2
├───── item1.3
├── container2
├───── item2.1
├───── item2.2
Import Stow plus any of the implementation packages that you wish to provide. For example, to support Google Cloud Storage and Amazon S3 you would write:
import (
"github.com/graymeta/stow"
_ "github.com/graymeta/stow/google"
_ "github.com/graymeta/stow/s3"
)
The underscore indicates that you do not intend to use the package in your code. Importing it is enough, as the implementation packages register themselves with Stow during initialization.
To connect to a location, you need to know the kind
string (available by accessing the Kind
constant in the implementation package) and a stow.Config
object that contains any required configuration information (such as account names, API keys, credentials, etc). Configuration is implementation specific, so you should consult each implementation to see what fields are required.
kind := "s3"
config := stow.ConfigMap{
s3.ConfigAccessKeyID: "246810",
s3.ConfigSecretKey: "abc123",
s3.ConfigRegion: "eu-west-1",
}
location, err := stow.Dial(kind, config)
if err != nil {
return err
}
defer location.Close()
// TODO: use location
You can walk every Container using the stow.WalkContainers
function:
func WalkContainers(location Location, prefix string, pageSize int, fn WalkContainersFunc) error
For example:
err = stow.WalkContainers(location, stow.NoPrefix, 100, func(c stow.Container, err error) error {
if err != nil {
return err
}
switch c.Name() {
case c1.Name(), c2.Name(), c3.Name():
found++
}
return nil
})
if err != nil {
return err
}
Once you have a Container
, you can walk every Item inside it using the stow.Walk
function:
func Walk(container Container, prefix string, pageSize int, fn WalkFunc) error
For example:
err = stow.Walk(containers[0], stow.NoPrefix, 100, func(item stow.Item, err error) error {
if err != nil {
return err
}
log.Println(item.Name())
return nil
})
if err != nil {
return err
}
Once you have found a stow.Item
that you are interested in, you can stream its contents by first calling the Open
method and reading from the returned io.ReadCloser
(remembering to close the reader):
r, err := item.Open()
if err != nil {
return err
}
defer r.Close()
// TODO: stream the contents by reading from r
If you want to write a new item into a Container, you can do so using the container.Put
method passing in an io.Reader
for the contents along with the size:
contents := "This is a new file stored in the cloud"
r := strings.NewReader(contents)
size := int64(len(contents))
item, err := container.Put(name, r, size, nil)
if err != nil {
return err
}
// item represents the newly created/updated item
An Item
can return a URL via the URL()
method. While a valid URL, they are useful only within the context of Stow. Within a Location, you can get items using these URLs via the Location.ItemByURL
method.
Item
by URLIf you have a Stow URL, you can use it to lookup the kind of location:
kind, err := stow.KindByURL(url)
kind
will be a string describing the kind of storage. You can then pass kind
along with a Config
to stow.New
to create a new Location
where the item for the URL is:
location, err := stow.Dial(kind, config)
You can then get the Item
for the specified URL from the location:
item, err := location.ItemByURL(url)
Cursors are strings that provide a pointer to items in sets allowing for paging over the entire set.
Call such methods first passing in stow.CursorStart
as the cursor, which indicates the first item/page. The method will, as one of its return arguments, provide a new cursor which you can pass into subsequent calls to the same method.
When stow.IsCursorEnd(cursor)
returns true
, you have reached the end of the set.