Using File Collection S3 storage adapter, where to put bucket

[anthonymanzo]

Hi, I have this code in the kitchen json: "collections" : [ { "name":"employee_pictures", "type":"file_collection", "storage_adapters":["s3"] }, And I've set my S3 access keys in my settings.json inside of the env property. However, I have no place to put the s3 bucket name and hence meteor throws an error when I compile:

W20170124-10:26:11.768(-8)? (STDERR) Error: FS.Store.S3 you must specify the "bucket" option W20170124-10:26:11.768(-8)? (STDERR) at new FS.Store.S3 (packages/cfs_s3.js:114:11)

Where should I put this info? Also, I haven't been able to find examples of how to use a fs collection in the examples, yet the s3 and fs storage adapters are mentioned in the API. Is this just a stub, or is it supported?

Thanks again. Tony

[perak]

@anthonymanzo

Hi Tony,

To be honest I never used S3 storage adapter with file collection, but let's modify kitchen to allow this. Can you please let me know what/where to add/change to make S3 works properly? Maybe you can provide minimal .json file and modify generated code to allow S3 works - that would be perfect.

[perak]

(example-upload, generated code modified manually to allow S3 storage is perfect - maybe I can modify and deploy fixed kitchen today/tomorrow)

[anthonymanzo]

HI Perak, I think we’ll need to add a few things:

1. An area to define bucket, ‘store’ and any transform options information in your collection def. (disregard the accesskey and secret acesskey, those I define on server env safely) — see example from https://github.com/CollectionFS/Meteor-CollectionFS/tree/master/packages/s3:

var imageStore = new FS.Store.S3("images", { region: "my-s3-region", //optional in most cases accessKeyId: "account or IAM key", //required if environment variables are not set secretAccessKey: "account or IAM secret", //required if environment variables are not set bucket: "mybucket", //required ACL: "myValue", //optional, default is 'private', but you can allow public or secure access routed through your app URL folder: "folder/in/bucket", //optional, which folder (key prefix) in the bucket to use fileKey: function(fileObj) { return new Date().getTime() + "-" + fileObj.name(); }

// The rest are generic store options supported by all storage adapters transformWrite: myTransformWriteFunction, //optional transformRead: myTransformReadFunction, //optional maxTries: 1 //optional, default 5 });

Images = new FS.Collection("images", { stores: [imageStore] });

2. A way to use things like tags in the kitchen field definitions — I’ve tried to use custom fields and display helpers but they don’t display properly in the dataview.

After publishing on server and subscribing on client it would look something like this:

{{#each images}} {{> image}} {{/each}}

3. And probably a way in the publication to define queries that search on key or filename?

BTW, here is an example of my settings.json where the env information that AWS S3 needs:

"env":{ "AWS_ACCESS_KEY_ID”:"xxxx", "AWS_SECRET_ACCESS_KEY”:"xxxx" }

Thanks for your help!

Tony

On Jan 24, 2017, at 11:56 AM, Petar Korponaić notifications@github.com<mailto:notifications@github.com> wrote:

@anthonymanzohttps://github.com/anthonymanzo

Hi Tony,

To be honest I never used S3 storage adapter with file collection, but let's modify kitchen to allow this. Can you please let me know what/where to add/change to make S3 works properly? Maybe you can provide minimal .json file and modify generated code to allow S3 works - that would be perfect.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/perak/kitchen-examples/issues/50#issuecomment-274919931, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AH95W3dfp1ZV6U8hD_g_C6Wi2fYQGLtRks5rVlddgaJpZM4LsoC5.

[perak]

@anthonymanzo OK, in next version 0.9.78 I added this:

Storage adapters now can be defined in storage_adapter_options instead storage_adapters (both works for backward compatibility) like this:

{
    "name": "files",
    "type": "file_collection",
    "storage_adapter_options": { 
        "s3": {
            "bucket": "mybucket",
                        ...
        }
    }
}

And resulting code is:

this.Files = new FS.Collection("files", {
    stores: [
        new FS.Store.S3("files", {
            bucket: "mybucket",
                    ...
        })
    ]
});

storage_adapter_options is json object with "s3", "dropbox", "filesystem" or "gridfs" in the root, and these objects can contain anything. JSON is converted to javascript and inserted into code.

I hope this helps.

Unfortunately, I'll not able to deploy latest version tonight, I hope I'll deploy tomorrow ~24 hours after now.

[anthonymanzo]

Awesome, I’ll give a try when it’s deployed. Thanks.

On Jan 24, 2017, at 2:45 PM, Petar Korponaić notifications@github.com<mailto:notifications@github.com> wrote:

@anthonymanzohttps://github.com/anthonymanzo OK, in next version 0.9.78 I added this:

Storage adapters now can be defined in storage_adapter_options instead storage_adapters (both works for backward compatibility) like this:

{ "name": "files", "type": "file_collection", "storage_adapter_options": { "s3": { "bucket": "mybucket", ... } } }

And resulting code is:

this.Files = new FS.Collection("files", { stores: [ new FS.Store.S3("files", { bucket: "mybucket", ... }) ] });

storage_adapter_options is json object with "s3", "dropbox", "filesystem" or "gridfs" in the root, and these objects can contain anything. JSON is converted to javascript and inserted into code.

I hope this helps.

Unfortunately, I'll not able to deploy latest version tonight, I hope I'll deploy tomorrow ~24 hours after now.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/perak/kitchen-examples/issues/50#issuecomment-274963788, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AH95W2QtwwHd3EMHB0a79ZgN5eP1YN_Uks5rVn7xgaJpZM4LsoC5.

[perak]

@anthonymanzo version 0.9.78 is deployed few minutes ago.

[perak]

BTW, storage_adapter_options can only be edited directly in json ("source" tab in a GUI) - I didn't implemented "fancy" editor - not yet.

[anthonymanzo]

Storage adapter code is now showing up in the subscriptions - thanks. Minor problem, I can't render the images with the data-view component - I need a way to return tags -- would use {{{ employeePictureToImgTag employee_picture }}} but the generated code for th data view is only using the standard handlebars double {{ }} which escapes html.

I am a noob at handlebars/blaze so please forgive my ignorance if this is not an issue.

[perak]

@anthonymanzo triple braces: already reported here https://github.com/perak/kitchen-site/issues/363 will be fixed in next release.

[anthonymanzo]

Hi, Just wanted to give an update that I haven't been able to test this as I'm having trouble with the latest version of cfs:s3 not working for me, I had the old version on lockdown. I'm going to close this issue for you until then!

[anthonymanzo]

Hi, another comment on this. Got it working (more or less) and want to know how to define multiple stores for a collection (ie a largthumb, smallthumb set). Any ideas?

[anthonymanzo]

Here's my code. Kitchen generator is only picking up the first store: "collections" : [

        {
            "name": "employee_thumbs",
            "type": "file_collection",
            "storage_adapter_options": { 
                "s3": {
                    "bucket": "salk-saw-assets",
                    "folder":"thumbs",
                    "transformWrite":"function(fileObj, readStream, writeStream) {

// Transform the image into a 30x thumbnail gm(readStream, fileObj.name()).resize('77', '77').stream().pipe(writeStream); }", }, "s3": { "bucket": "salk-saw-assets", "folder":"large_thumbs", "transformWrite":"function(fileObj, readStream, writeStream) { // Make the image a predictable 225px thumbnail gm(readStream, fileObj.name()).resize('225', '225').stream().pipe(writeStream); }", } }

        },

[perak]

@anthonymanzo yes, bug is: kitchen enumerates "distinct" storage adapters, instead each one. Will fix that for next release, but I'll not be able to deploy it next ~7 days. I suggest you to use copy_files and write code manually.