StorageAPI for Images and Files

[grabbou]

Draft

To store Images, we have to use CloudinaryImage if we are on Heroku and so on or if we like ability to resize or manipulate our image. Not everybody is happy with paying monthly-fee for having only those features. Although Cloudinary is great - not everyone is using all of its features. That's why we need to kick our Image and Files to next level and make it like in other CMS systems.

Storage API

REPOSITORY: https://github.com/grabbou/StorageAPI/

Will be released as stand-alone NPM package and after that, integrated into Keystone.

Deliver Keystone.StorageAPI for developers to easily use storage (when available) in their projects. Storage API can be used either in fields or in our own views and modules. It's public. Define once, use multiple times.

Directory structure

providers/
providers/storage/index.js // this is place for our 'class' **Storage**
providers/storage/....
providers/storage/amazon.js //name 'amazon' now is available as keystone.set('storage')
providers/storage/azure.js

// once we finish, it's time to write mailApi
providers/mail/index.js // mail resolver, works like storage one as well (obtain() method)
providers/mail/postmark.js
providers/mail/mandrill.js
providers/mail/smtp.js

Storage

• obtain(options) returns current storage provider based on keystone.set('storage') specified. We can optionally pass argument provider here in case our custom field allows to set different provider for it (for example - files in S3, images in Cloudinary). Automatically checks whether given provider is available (file with it exists in our providers folder).

  StorageClient

• @constructor - registers new client based on keystone.set('storage config'). This should be an object, specific for different providers. Can accept either single object (then we assume it's config for keystone.set('storage') or object array (in case we use multiple storage providers inside our app):

keystone.set('storage', {
   'azure': {},
   'amazon': {}
});

• upload(options), delete(options), exists(options) - easy manipulation
• _ensureContainer() - ensures that container exists - based on keystone.set('storage container'). We do not have any default value here as container name should be unique across cloud platform

Please note, that async methods are wrapped within promises (Q library) to avoid callback chain and provider better functionality

List of supported providers to start with

• Everything provided by pkgcloud - Amazon, Azure, HP, Rackspace, OpenShift
• Filesystem
• FTP / SFTP

  Pros:

• Use core, consistent storage layer to access file systems. Developers can now easily write their own modifications and fields using our API without thinking whether config is supported or not. Easier in maintaining. If API changes - only one file to edit, not 12 plugins that are interacting with them.
• No more hard-coded supported providers. We simply add new file under providers/storage and users can start using it automatically. Our Storage checks whether keystone.set('storage') provider is available, if not - throws new Error('Unsupported provider').
• Users can easily create their own PR with new providers (or we can even include them as dependencies or think of plug & play architecture for the future).

  Fields

Removing multiple fields that are doing the same job and replacing them with following ones. We are going to delete ui-azure.js, ui-s3file.js and many many more (with CSS3 stylesheets as well! Going simple now!)

File

Simple field for handling files. Accepts any storage (uses default one specified in the project). Returns false if uploaded file doesn't match allowedTypes.

Schema:

• url {String} - full_res url
• file_name {String}
• provider {String}
• container {String} - might be path in case we used FTP/SFTP.

Accepted fields:

• required - if file is required. If yes and no file provided, returns an error.
• provider - overwrites provider for that field. Remember to specify configuration. If array specified, multiple actions are performed.
• size - maximum size - if wrong, returns an error
• allowedTypes

Available hooks:

• pre: upload
• post: upload

  Image extends Field

Generic ImageField for uploading & handling images. Deletes CloudinaryImage as it's now built-in within this field. The same with other image-related fields.

Schema:

• inherited

Accepted fields:

• inherited

Underscored methods

• Full Cloudinary API support (if Cloudinary is not default provider or provider for that field, use our wrapper for gm). Key feature is to mimic Cloudinary functionality (the same method calls, almost the same effects). Our Node API simply checks whether image matching options specified is present on server, if not - we generate it and return a link. Please note that we are keeping ful-res image in the cloud storage just to have the ability to resize them in the future.

Available hooks:

• inherited

  Wysiwyg

Removes hard-coded CloudinarySupport. Allows to upload file to default storage provider when available. Allows overwriting that by specifying keystone.set('wysiwyg storage'). To enable image upload - call keystone.set('wysiwyg image upload'). It means you can now upload your wysiwyg images easily wherever you want. Either to FTP or Amazon S3. Want to resize your images while uploading? Easy! Just set keystone.set('wysiwyg image options') and we will adjust your input using ImageField undescored methods. No more resizing before uploading high-res photos.

Breaking changes

• Removed all file fields and image fields. Replaced with those new ones.
• Removed lots of unnecessary keystone.set configs, like cloudinary and so on.

  Updates

  Update 1

• After writing new field - we should move post/pre and other methods to basic Field class and implement them where possible. In most cases, they are all the same. In others, just make them abstract by throwing new Error('Method not implemented') so one can easily overwrite them and implement. Useful when you are writing everything from scratch and want to make sure that you declared everything needed to work.

  Update 2

• updateHandler needs to be rewritten. For now, we can add new fields to that case, but for the future, we should have in our field method returning value. Based on that value, we can decide what action we should take in ActionQueue. Something like {Field}.getUploadHandlerType. No need to modify core files after that.

[wmertens]

See also https://github.com/keystonejs/keystone/issues/3209#issuecomment-236323406 for my take on how Storage Adapters are responsible for handling files, the rest of Keystone should just work with File objects (the bags of data that the Storage Adapter uses to represent the file).

[peterjohansson92]

Maby im in the wrong discussion for this question, but what is the status about using s3 instead of cloudinary in the wysiwyg editor for image uploading? Currently we use "wysiwyg cloudinary images" in the init, but is it possible to use s3 instead? If so, i would be very pleased if someone could tell me how.

[gautamsi]

Keystone 4 is going in maintenance mode. No major changes expected. see #4913 for details.

I see a PR was done for S3 image upload for wysiwyg in v4, see #1614