Showing video from gridfs link

[ignl]

Hi,

I got stuck a little bit. Don't have reproducible example at the moment but basically everything works for me except when I try to load video (in base64) from a link (as in the example) it doesn't work. Then I right click and do save as and paste file content (data:video/webm;base64,GkXfo59ChoEBQ....) directly into src of video tag and it plays fine. I think I am missing something simple. Any ideas?

Some maybe helpful code excerpts: HTML

<video webkit-playsinline width="300" height="240" controls preload="metadata">
     <source src="{{link md5}}?cache=172800" type="video/webm">
                    Your browser does not support the video tag.
</video>

Client

"link": function (md5) {
        return posts.baseURL + "/md5/" + md5
}

Server

posts.allow({
    read: function (userId, file) {
        return (true);
    }
});

Initialisation

posts = new FileCollection('posts', 
    { resumable: true, 
        resumableIndexName: 'rsmIdx',
        baseURL: '/gridfs/fs',
        maxUploadSize: 16*1024*1024 - 1024,
        http: [
            { method: 'get',
                path: '/md5/:md5',
                lookup: function (params, query) { 
                    return { md5: params.md5 };
                }
            }
        ]
    }
);

[vsivsi]

If I understand, it seems that you have a mismatch in the type of the file you are storing in gridFS.

The base64 data URI works in the source tag because it resolves (when automatically decoded as part of reading the URI) into a video/webm type file. If you store a video/webm type file (the binary, not a base64 encoded URI) in gridFS, then everything should work. The URI in the src tag must resolve to a video file, not to a text file containing a base64 encoded URI. There is no mechanism to "recursively resolve" a URI that returns another URI as text.

Hope that helps.

[ignl]

Thanks for an answer. Yes it is saved as base64 to mongo. I record video with webcam, get base64 data and save it to the meteor-file-collection. So as I understand, I have 2 options: encode somehow? base64 to webm before saving to gridfs or load base64 with http get in my template and set text as src directly. Am I right? With first option it will probably work as a stream and will be more performant or I am mistaken?

[vsivsi]

Right, if you want to be able to efficiently store the video and seek using the player without downloading the whole thing, then the video needs to be stored in gridFS in the original binary video format, not as a base64 encoding of the binary format.

How are you getting base64 in the first place? Surely the webcam is not producing a base64 data URI as its output?!

[ignl]

I am using this package: https://github.com/lukemadera/meteor-video-capture/ It seems pretty good, but a bit lacking in documentation, so right now I am looking if its possible to get original binary data after recording.

[vsivsi]

I see. There are 3 alternatives that I can think of:

1. Fork that package so it returns an HTML 5 file (composed of binary blobs) instead of a base64 encoded data URI
2. Somehow convert the base64 data URI back into an HTML5 file of binary blobs on the client
3. Upload as a base64 data URI and then perform the conversion to binary on the server and write it to a different file.

Any of those will probably work, but 2) and 3) are wasteful because they are working to undo unnecessary work that was done by the video capture package in the first place.

For a little background on HTML files and creating them to upload to a file-collection, see: https://github.com/vsivsi/meteor-file-collection/issues/29

[ignl]

Thanks Vaughn, I went through some tickets and saw #29 too. Gotta say your support here is outstanding. Like really, some expensive products not always have that kind of support! One thing that I see that could be helpful is some kind of diagram at the start of documentation which visually explains architecture and different options. Thank you for this great package!

[vsivsi]

You can always repay me in documentation PRs!

[ignl]

Ok, maybe I will try later when have a bit more expierence.

[ignl]

I have another question. Its not exactly clear for me how do I upload files if I want to use only Meteor.methods approach? I currently have something like this:

posts.allow({
    read: function (userId, file) {
        return (true);
    }
});

Meteor.methods({
    // add post and uploaded file for a post if it exists
    'addPost': function (author, base64FileContent) {
        var writer = posts.upsertStream({
                _id: new Meteor.Collection.ObjectID(),
                contentType: 'video/webm',
                metadata: { author: author, postDate: new Date() }
            }
        );
        writer.end(base64FileContent);
    }
});

First of all I had to set allow rule for reading otherwise links weren't working. Is thats how it should be? I don't use allow/deny rules at all in my application, but here it seems unavoidable. Also I don't really understand how to insert file content. This upstream worked with base64, but now when I changed it to binary data I get this warning: Warning: gridfs-locking-stream Write Lock Expired for file e0e33ba2b8818004c2af9401 and no insert. Example in documentation is only for insert from client with resumable. How to do same thing through Meteor.methods? Will resumable work? Or I simply should make an exception for this and insert from client exactly like in documentation?

[vsivsi]

I really don't recommend sending file data as a parameter to a Meteor method call. That means that the underlying DDP protocol will be carrying the data, and it is known to be very unhappy when message sizes grow beyond the tens to hundreds of kilobytes. There are too many potential problems with that approach for me to even get into. Just don't. I simply won't provide any support for that way of using this package.

For large files you should use the built-in resumable.js upload support. If you really don't want to do that, then you should define a HTTP POST or PUT route and do a vanilla HTTP request with the data in the body. But in any case, all file data should use HTTP, not DDP. You can use jQuery or the Meteor HTTP API to make such calls. But resumable.js will be the best and easiest way.

For reading/writing file data using HTTP, yes you need to define allow/deny rules, because: Security. It's not that hard, and then you know exactly what choices you are making.

[ignl]

Ok thank you for clearing this up. Will use resumable and allow/deny.

[ignl]

Ok I hope these will be my last questions :) Currently I am doing like this:

function insertPost(title, .....) {
        posts.resumable.on('fileAdded', function (file) {
            posts.insert({
                    _id: new Meteor.Collection.ObjectID(),
                    contentType: 'video/webm',
                    filename: file.fileName,
                    metadata: {author: Meteor.user(), title: title, .....}
                },
                function (err, _id) {  // Callback to .insert
                    if (err) {
                        FlashMessages.sendError(err);
                    }
                    posts.resumable.upload();
                }
            );
        });
       posts.resumable.addFile(new File([recordedVideo], "test.webm"));
}

This allows me to set various metadata for the document because I record video, fill various fields and then press save button which calls this method. So I set metadata because for each save call I can change an event handler (dirty solution I know). However If I would add some real file uploads too that obviously wouldn't work. Its not clear to me what is the best practice to create file document with various metadata fields. Any suggestions? Also right now I am getting errors like these:

~0.1MB - 1 (100 kb) chunks
GET data: net::ERR_INVALID_URL
~0.4MB - 4 (100 kb) chunks
~0.4MB - 4 (100 kb) chunks
~0.4MB - 4 (100 kb) chunks
HEAD http://localhost:3000/gridfs/fs/_resumable?resumableChunkNumber=1&resumabl…leFilename=test.webm&resumableRelativePath=test.webm&resumableTotalChunks=1 404 (Not Found)
  ResumableChunk.$.test 
  ResumableChunk.$.send 
  (anonymous function)  
  $h.each   
  (anonymous function)  
  $h.each   
  $.uploadNextChunk 
  $.upload  
  (anonymous function)  
  (anonymous function)  
  (anonymous function)  
  _.extend._maybeInvokeCallback 
  _.extend.dataVisible  
  (anonymous function)  
  _.each._.forEach  
  _.extend._runAfterUpdateCallbacks 
  _.extend._livedata_data   
  onMessage 
  (anonymous function)  
  _.each._.forEach  
  self.socket.onmessage 
  REventTarget.dispatchEvent    
  SockJS._dispatchMessage   
  SockJS._didMessage    
  that.ws.onmessage 
HEAD http://localhost:3000/gridfs/fs/_resumable?resumableChunkNumber=1&resumabl…leFilename=test.webm&resumableRelativePath=test.webm&resumableTotalChunks=1 404 (Not Found)
  ResumableChunk.$.test 
  ResumableChunk.$.send 
  (anonymous function)  
  $h.each   
  (anonymous function)  
  $h.each   
  $.uploadNextChunk 
  $.upload  
  (anonymous function)  
  (anonymous function)  
  (anonymous function)  
  _.extend._maybeInvokeCallback 
  _.extend.dataVisible  
  (anonymous function)  
  _.each._.forEach  
  _.extend._runAfterUpdateCallbacks 
  _.extend._livedata_data   
  onMessage 
  (anonymous function)  
  _.each._.forEach  
  self.socket.onmessage 
  REventTarget.dispatchEvent    
  SockJS._dispatchMessage   
  SockJS._didMessage    
  that.ws.onmessage 
GET http://localhost:3000/gridfs/fs/md5/d41d8cd98f00b204e9800998ecf8427e?cache=172800 416 (Requested Range Not Satisfiable)
POST http://localhost:3000/gridfs/fs/_resumable 404 (Not Found)
  ResumableChunk.$.send 
  testHandler   

[vsivsi]

The problems you are seeing result from the fact that you aren't using the _id value generated by resumable when the file is added. That is how resumable knows which file on the server it is uploading to.

So you basically have things backwards. You should create a handler for the resumable fileAdded event, and inside that handler, do the insert, using the _id value for the file that you get from resumable. You can attach other metadata, etc to the file object that you "add" to resumable, and then use it when you insert the file into fileCollection. But the file.uniqueIdentifier <==> _id connection is critical, because it is what connects the resumable upload session to the correct gridFS file on the server.

That is just what is shown in the example code: https://github.com/vsivsi/meteor-file-collection

myFiles.resumable.on('fileAdded', function (file) {
      // Create a new file in the file collection to upload
      myFiles.insert({
        _id: file.uniqueIdentifier,  // This is the ID resumable will use
        filename: file.fileName,
        contentType: file.file.type
        },
        function (err, _id) {  // Callback to .insert
          if (err) { return console.error("File creation failed!", err); }
          // Once the file exists on the server, start uploading
          myFiles.resumable.upload();
        }
      );

// Sometime later....
myFiles.resumable.addFile(new File([recordedVideo], "test.webm"));

[ignl]

Ok I managed to get everything almost working, big thanks to you. There is still one issue though. Once I upload a file I immediately insert it into collections which is then reactively shown on the same page, so upload result is seen right away. And it works, just unfortunately the video is not shown (probably because insert is faster than file upload and gridfs simply cant return data on that url yet). After I do full page refresh it loads video and works fine. I tried to use resumable 'fileSuccess' event which is reloading video tag with jquery like that: $('#videoId').load(location.href + ' #videoId'); But it didn't help. Also it maybe not very clean solution. What do you consider as a best practice for this situation? Maybe I missed it in documentation?

[vsivsi]

Hi, I don't really know how you are building your client UI (blaze?) but the solution is to not add the video player element to the DOM until the upload is complete. You can easily wait for that by reactively depending on the md5 or length attributes of the file. The length will be zero until the upload is complete, and the md5 value will likewise change from the default (all zero length files have the same md5...) to the calculated value. This is just a basic use of Meteor client UI reactivity, no file-collection special sauce required.

A similar mechanism is used in the sample app to prevent the analogous problem from occurring for images. See: https://github.com/vsivsi/meteor-file-sample-app/blob/master/sample.coffee#L143 https://github.com/vsivsi/meteor-file-sample-app/blob/master/sample.jade#L60

[ignl]

Yeah I did like in example, now its ok, thanks!