[FR] Possibility to store asset data in database?

[arifje]

Hi,

During the development of our new website we noticed a huge slowdown/lag in case of multiple videos on a page. Not only during initial encoding, but also everytime getFileInfo() is fired, which we use for grabbing dimensions.

Two thoughts;

1) can we push the encoding part to the background, so the page will still be loaded instead or timing out eventually? 2) is it possible to store file info (dimensions, filesize etc.) in the database, so we don't have to read the files/assets from our CDN.

Thanks in advance!

[khalwat]

The encoding happens in the background; what you can do is if it returns null put a placeholder up (because it means the video is encoding).

As for grabbing the dimensions, hopefully you can just output the thumbnails at a fixed size, so you don't need to read in the size on each request?

[arifje]

Thanks for your reply.

About the placeholder; that is already our approach here. But for some reason, if 10+ encoding instances are fired, the page times out. Not sure how to fix this yet. With one or two videos it works fine.

About the dimensions: we are displaying different videos with different ratio's; sometimes they are landscape, but vertical/portrait videos are created more and more often these days. And we have to pass these parameters to our player so it will be displayed correctly. This is why it would be much better for performance if these attributes are stored in the database. For example, when the encoding process is fired. But I don't know of craft allows to store extra data from an asset by default. I know we can create custom fields, but then we have to define them I guess, so these values can be stored into the right place.

[khalwat]

So something like a Transcoder field type where you could set the parameters, etc. and access things like the URL, height, width, duration, etc.?

[arifje]

Yes, a field that we can add to the asset in which all data that getFileInfo() returns is stored. Would improve the performance (in our case) very much.

Would be nice if we can grab this data directly from the video/assest object: video.fileInfo.duration and video.fileInfo.width etc.

[khalwat]

I'm trying to figure out a smart way to do this, because currently the transcoded video/audio is treated like a cache... so they can be cleared, and you can overwrite them, etc.

But perhaps a way to optionally add the result to an AssetVolume is a good idea.

[arifje]

But storing it to a assetVolume means that it will be stored in a local file right? Personally I think it's better to store it into a database. Also because in our situation we're using a load balanced setup with multiple webservers, which means that this data related to a video object is stored on all webservers.

Is it possible to store the data in a table that belongs to the plugin and access the values from a template?

I think it's best to store it into a custom field (custom type: transcoderData), which you can attach to an entry. The only thing is that it needs to be linked to the plugin via config or something, or passed as an parameter when encoding a video, like this:

{% set encodedVideo = craft.transcoder.getVideoUrl (video, {
  dataField: myCustomField
} ) %}

Then we can grab the data when like this:

{{ myCustomField.height }}
{{ myCustomField.width }}

And, if there's no customField passed, it uses the current method and threats it like cache.

[arifje]

In addition: currently I am working on a load balanced structure with multiple backends. If I want to access the video data we're talking about I'll have to install ffmpeg/ffprobe on all webservers, which is far from ideal. So maybe this is an extra reason to move this data to a database level?

[arifje]

Another issue/conclusion that I bumped into; all our webservers aren't setup/suited for encoding videos. Sure, I can install ffmpeg and ffprobe will work, but these servers are meant for serving web pages and most of them are running on a VPS with Lighttpd only.

For now I solved that by proxying all of our team member IP's to another webserver that can handle the encoding (dedicated server). So if we preview our entries on the front end the encoding process is triggered, files are written to a NFS share.

But this brings a huge risk: if we forget to preview an entry, nothing is encoded, which lead to non-working entries.

Solution: trigger this process in the control panel, after a video has uploaded.

Is this possible or something we can work on? Not sure if I can fix this myself, but we have some budget for this as it is very important for us. Then we can also implement the video data issue.

Cheers!

[khalwat]

So what you're hoping to have happen is a way to remotely trigger the encoding of the video on another server? And then copy the resulting file back to the webserver?

[arifje]

Triggering the process on another server could also be a solution I guess, but that's not what I meant (and also sounds more difficult task to complete, but maybe I am wrong).

Let me try to explain our current situation;

We have a load balancer, with 10+ webservers behind it. 8 out of 10 webservers are VPS servers, 2 of them are dedicated. Because the transcoder plugin is triggered from a template I have no control over which server is going to be doing this, because when someone is visiting the website, it randomly gets pointed to one of these servers. So we have to make sure the template is called already and triggered the encoding process before a visitor views the page.

I "solved" this by proxying our team members to one of the dedicated servers, which means that for certain IP's our loadbalancer always requests one of the dedicated webservers. The process is triggered there and so the encoding starts...

But as I mentioned in my other comment this brings a risk; if admins forget to call/view the entry/template the encoding process won't get fired. Visitors that access the website via the other webservers (that won't run ffmpeg or just don't have the capacity to encode), are serving the encoded files, but also can't encode thumbnails.

This is why I was thinking of a way to move/duplicate this part from the front end to the back end. If we can trigger the encoding part (getVideoUrl,getVideoThumbnailUrl, getGifFilename) - right after we upload the file/asset in the CP - this would solve all of these problems. In this case, we aren't dependent on the front end template. Plus, we know on which server we are working on.