fromURl serverside upload - how to with new API?

[ghost]

What's the proper approach for using fromUrl for a server side upload using the new APIs, I've created a simple example, and I'm experiencing the following problem: has no method 'fromUrl'

Is there any obvious mistake in the approach below, or is it because the devel version is still in flux?

APPROACH:

creating a test meteor application

• with collectionFS devel on blaze-rc0
• for testing server upload .fromUrl
  • reason: I'm experiencing an error

mrt --release blaze-rc0 create test-fromUrl

..edit smart.json ---result--- { "packages": { "iron-router": { "git": "https://github.com/EventedMind/iron-router.git", "branch": "blaze-integration" }, "blaze-layout": { "git": "https://github.com/EventedMind/blaze-layout.git", "branch": "devel" }, "collectionFS": { "git": "https://github.com/CollectionFS/Meteor-CollectionFS.git", "branch": "devel" }, "cfs-gridfs": { "git": "https://github.com/CollectionFS/Meteor-cfs-gridfs.git", "branch": "master" }, "cfs-filesystem": { "git": "https://github.com/CollectionFS/Meteor-cfs-filesystem.git", "branch": "master" }, "cfs-handlebars": { "git": "https://github.com/CollectionFS/Meteor-cfs-handlebars.git", "branch": "shark" }, "cfs-graphicsmagick": { "git": "https://github.com/CollectionFS/Meteor-cfs-graphicsmagick.git", "branch": "master" } }

}

meteor remove autopublish insecure mrt add iron-router mrt add blaze-layout mrt add collectionFS mrt add cfs-gridfs mrt add cfs-filesystem mrt add cfs-handlebars mrt add cfs-graphicsmagick mrt update

..edit test-fromUrl.js ---result--- // common.js var Images = new FS.Collection("images", { stores: [new FS.Store.FileSystem("images", {path: "~/uploads"})] });

if (Meteor.isClient) { Template.hello.greeting = function () { return "Welcome to test-fromUrl."; };

Template.hello.events({ 'click input': function () { // template data, if any, is available in 'this' if (typeof console !== 'undefined') console.log("You pressed the button"); } }); }

if (Meteor.isServer) { Meteor.startup(function () { // code to run on server at startup var fileObj = FS.File.fromUrl('http://www.fourwinds10.net/resources/uploads/images/meteor-shower-nightsky-jpl-nasa-gov.jpg', 'image.jpg'); Images.insert(fileObj); });

}

meteor ---output--- user@localhost:~/devel/test-fromUrl# meteor [[[[[ ~/devel/test-fromUrl ]]]]]

=> Started proxy. => Meteor 0.7.1.2 is available. Update this project with 'meteor update'. => Started MongoDB. W20140317-15:21:25.468(1)? (STDERR) Failed to load c++ bson extension, using pure JS version I20140317-15:21:25.689(1)? TempStore: /tmp/cfs W20140317-15:21:25.794(1)? (STDERR) W20140317-15:21:25.795(1)? (STDERR) /users/user/.meteor/tools/6c38ffe2fb/lib/nodemodules/fibers/future.js:173 W20140317-15:21:25.796(1)? (STDERR) throw(ex); W20140317-15:21:25.798(1)? (STDERR) ^ W20140317-15:21:25.802(1)? (STDERR) TypeError: Object function (ref, createdByTransform) { // 9 W20140317-15:21:25.802(1)? (STDERR) var self = this; // 10 W20140317-15:21:25.803(1)? (STDERR) // 11 W20140317-15:21:25.803(1)? (STDERR) self.createdByTransform = !!createdByTransform; // 12 W20140317-15:21:25.804(1)? (STDERR) // 13 W20140317-15:21:25.804(1)? (STDERR) if ((typeof File !== "undefined" && ref instanceof File) || // 14 W20140317-15:21:25.805(1)? (STDERR) (typeof Blob !== "undefined" && ref instanceof Blob)){ // 15 W20140317-15:21:25.805(1)? (STDERR) self.attachData(ref); // 16 W20140317-15:21:25.805(1)? (STDERR) } else { // 17 W20140317-15:21:25.806(1)? (STDERR) if (typeof ref !== 'object') { // 18 W20140317-15:21:25.806(1)? (STDERR) ref = {}; // 19 W20140317-15:21:25.806(1)? (STDERR) } // 20 W20140317-15:21:25.807(1)? (STDERR) // 21 W20140317-15:21:25.807(1)? (STDERR) // Extend self with filerecord related data // 22 W20140317-15:21:25.807(1)? (STDERR) .extend(self, FS.Utility.cloneFileRecord(ref)); // 23 W20140317-15:21:25.808(1)? (STDERR) } // 24 W20140317-15:21:25.808(1)? (STDERR) } has no method 'fromUrl' W20140317-15:21:25.809(1)? (STDERR) at app/test-fromUrl.js:24:30 W20140317-15:21:25.809(1)? (STDERR) at /users/user/devel/test-fromUrl/.meteor/local/build/programs/server/boot.js:159:61 W20140317-15:21:25.810(1)? (STDERR) at Array.forEach (native) W20140317-15:21:25.810(1)? (STDERR) at Function..each..forEach (/users/user/.meteor/tools/6c38ffe2fb/lib/node_modules/underscore/underscore.js:79:11) W20140317-15:21:25.811(1)? (STDERR) at /users/user/devel/test-fromUrl/.meteor/local/build/programs/server/boot.js:159:5 => Exited with code: 8

^C

[raix]

I'm not 100% its not safe api area at the moment since we are changing alot on internals to get 100% streaming architecture.

  var file = new FS.File({ name: 'sweet-meteor.jpg' }); // This could be grapped from the url
  file.attachData('http://www.fourwinds10.net/resources/uploads/images/meteor-shower-nightsky-jpl-nasa-gov.jpg');
  images.insert(file);

This could be changing in the near future if we find a better pattern. The one you tried makes sense too var file = FS.File.fromUrl() but in most cases we will have to rig the file object with a filename for user readabillity - the system dont require a filename except for forcing a download (the user would typically expect a filename)

.. attachData makes sense

@aldeed Ideas:

• allow insert of files without data, upload when attached? (like resume)
• direct inserts:

  // Not supported at the moment
  images.insert(fileObj); // Dont have to have any data attached since we have createWriteStream?
  images.insert(File); // On the client this would be fine
  images.insert(urlString);
  images.insert('filename', buffer/blob/readStream);

[aldeed]

@casperanderseneu, as @raix said, fromUrl is gone and you can use attachData, or you can pass directly to insert first arg, but either method requires that you provide a callback, too:

var file = new FS.File({ name: 'sweet-meteor.jpg' });
file.attachData('http://www.fourwinds10.net/resources/uploads/images/meteor-shower-nightsky-jpl-nasa-gov.jpg', function (error) {
  !error && images.insert(file);
});

// OR

images.insert('http://www.fourwinds10.net/resources/uploads/images/meteor-shower-nightsky-jpl-nasa-gov.jpg', function (error, result) {
});

The callback is necessary because we need to HEAD the URL before inserting in order to get the type and size, which are required.

@raix, direct insert already works. Insert and then upload when attached could be tricky because we filter inserts based on type, size, etc. and we won't know those until data is attached?

I agree that we can try to determine filename automatically from the URL if it has an extension. Also, we can likely get rid of the need for those insert and attachData callbacks on the server by adding wrapAsync support, I just didn't get around to doing that yet.

[raix]

cool :) both buffer and blob got a size - remote url should get content-length in request response. but we have a problem with size using readStream - we could have the writeStream emit error if filter denies more data.

[aldeed]

insert and attachData no longer need a callback when attaching a URL on the server.

[aldeed]

Just to clarify, also, there is no actual buffered data on the client or the server when you "attach" a URL or a filepath. In those cases, we're basically just making note of the URL or path and then we later stream it directly into the TempStore on the server.

[ghost]

@aldeed Images.insert("http://domain.name/image.png"); works great here, file is created and easily retrievable.

However I'm experiencing a problem with Images.insert("/path/to/image.png"); The file is actually created on the file system, and in db.cfs.images.filerecord, but the size is 0 in both the document and the file system.

• same result when using Images.insert("/path/to/image.png", function (error) {])); and:

var file = new FS.File({ name: 'sweet-meteor.jpg' }); file.attachData('/path/to/image.png', function (error) { !error && Images.insert(file); }); // OR Images.insert('/path/to/image.png', function (error, result) {});

Is the approach for using filepath different? or still to be implemented?

[aldeed]

@casperanderseneu, I think I've fixed this now. Still working on adding tests for the cfs-file pkg and that should eliminate these pesky bugs.

[aldeed]

Also, I think it should now be automatically extracting the filename from URL or filepath if there is one. So you should not have to worry about setting name anymore, unless you want it to be different (or unless it's a URL without an extension).

[ghost]

@aldeed Automatic filename extract works for both url and filepath, I can confirm.

However Images.insert("/path/to/image.png"); still inserts a 0 byte file in both db.cfs.images.filerecord and the file system.

Maybe there is a problem in how .attachData is handling filepath, since it does insert files with data properly when using a url instead of a filepath.

[Rebolon]

Actually the problem is still here : On client, using

Images.insert('http://myHome/myImage.jpg', function(err, res) {console.log(arguments);})

Will produce

Error: FS.File.attachData requires a callback when attaching a URL on the client

Whereas

var file = new FS.File(); // This could be grapped from the url
file.attachData('http://myHome/myImage.jpg', function() {console.log(arguments);});
Images.insert(file, function(){console.log(arguments);});

Will succeed.

[vuhpham]

@Rebolon Same to me

[nikhildaga]

I am experiencing the same.