Transloadit Adaptive and Versions

[rmaspero]

I am trying to use transloadits /video/adaptive robot and there are a couple issues I am encountering.

First problem is that when using the /video/adaptive robot it produces an array of results not just a single result in a hash. This array is in no particular order so sometimes Shrine stores random files and not the final playlist which is what I really want. (I guess in the future some people might want to store the other playlists or segments as well?)

The next issue is that the path when saving to S3 for this robot is kind of important as the playlists build themselves from the relative location of the segments. Meaning I need to maintain the folder structure produced by the default path "${file.meta.relative_path}/${file.name}"

And finally because I want to use versions because I am also generating a thumbnail from the video. I am unable to pass through steps without making them files. I seem able to pass the import step, but no other steps?

[rmaspero]

I've managed to solve the second and third issue by using a template and passing in the versions fields in the assemble command. However I still encounter the problem that for the /video/adaptive robot the returned JSON is an array of files, in different orders every time.

A possible fix would be if a result key is an array with more than one value to create versions for each file using the base name as their version key?

[rmaspero]

I imagine it would look something like this inside transloadit_save?

if response["results"].fetch(key).count > 1
  response["results"].fetch(key).each do |result|
    uploaded_file = store.transloadit_uploaded_file(result)
    compiled_name = "#{name}_#{result['meta']['relative_path'] or ''}_#{result.fetch('basename') or ''}"
    hash.update(compiled_name => uploaded_file)
  end
else
  result = response["results"].fetch(key)[0]
  uploaded_file = store.transloadit_uploaded_file(result)
  hash.update(name => uploaded_file)
end
hash

That is for the versions bit but it would also need to do similar if not using versions.

[janko]

First problem is that when using the /video/adaptive robot it produces an array of results not just a single result in a hash. This array is in no particular order so sometimes Shrine stores random files and not the final playlist which is what I really want. (I guess in the future some people might want to store the other playlists or segments as well?)

I'm having a hard time visualizing what you mean here, mostly because I don't know exactly what is the difference between the results of the /video/adaptive robot and most other robots. I get that the result of that robot are multiple files, but I didn't understand the what you mean by Shrine storing "random files and not the final playlist". Could you explain on some sample response from Transloadit?

Yes, Shrine cannot save arrays of files at the moment, but you can always convert an array into a hash and then they're treated like versions.

{part_1: file1, part_2: file2, ...}

The next issue is that the path when saving to S3 for this robot is kind of important as the playlists build themselves from the relative location of the segments. Meaning I need to maintain the folder structure produced by the default path "${file.meta.relative_path}/${file.name}"

That doesn't sound like it's going to be a problem, we can create shrine uploaded file data using any location.

And finally because I want to use versions because I am also generating a thumbnail from the video. I am unable to pass through steps without making them files. I seem able to pass the import step, but no other steps?

Hmm, it should be possible to pass in any steps (e.g. you can pass your own export step instead of the one that shrine-transloadit generates automatically), because you can always drop down to a lower layer. Have you seen the "Import & Export" and "Transloadit gem" sections of the README?

[rmaspero]

@janko-m For my example where I am using versions to run both the thumbnail and the adaptive robots the returned JSON from transloadit looks a little like this:

"results":{ "thumbnailed": [ { id:.....}], "adaptive": [{id:.....},{id:.......},etc...] }

So versions deals well with the fact there is thumbnail and adaptive but since adaptive is an array of hashes and not just one file the transloadit_save function just picks one of the hashes and stores that. The problem here is the order of those hashes changes so I get a different file saved each time. It would be good if each hash could be saved as a separate version.

The full returned JSON is quite large but have linked it here: https://www.dropbox.com/s/334qyskln85rjuu/c33d6750c4b711e7ab3f3bb95132b45e.json?dl=0

[janko]

@rmaspero Thank you for these details!

I'm working on a patch now, it should be finished later today or tomorrow, and then it would be great if you could test it.

The idea is that given the following #transloadit_process definition:

class MyUploader < TransloaditUploader
  def transloadit_process(io, context)
    original  = transloadit_file(io)
    thumbnail = original.add_step("thumbnailed", "/image/resize", width: 500)
    playlist  = original.add_step("adaptive", "/video/adaptive", ...)

    { thumbnail: thumbnail, playlist: playlist }
  end
end

Shrine-transloadit would save the processed files to your Shrine attachment column in the following way:

{
  "thumbnail": {
    "id": "...",
    "storage": "store",
    "metadata": {...}
  },
  "playlist_0": {
    "id": "...",
    "storage": "store",
    "metadata": {...}
  },
  "playlist_1": {
    "id": "...",
    "storage": "store",
    "metadata": {...}
  },
  ...
  "playlist_n": {
    "id": "...",
    "storage": "store",
    "metadata": {...}
  }
}

Shrine would then retrieve them as a hash:

{
  thumbnail:  Shrine::UploadedFile.new(...),
  playlist_0: Shrine::UploadedFile.new(...),
  playlist_1: Shrine::UploadedFile.new(...),
  ...
  playlist_n: Shrine::UploadedFile.new(...)
}

The versions would be written in the order in which Transloadit returned them. Would that work for you?

[rmaspero]

@janko-m That would work marvellously! I am more than happy to test it 😄

[janko]

@rmaspero Here is the related documentation, let me know how it works for you, and if all is good I will release a new version.

[rmaspero]

@janko-m It wasn't until I tired it I realised we have made one little mistake. Since the files get given back to you in a random order there is no way to know which file is which without looping through them all. e.g. if you get playlist_0, playlist_1 and playlist_2 for different uploads the file you are looking for maybe playlist_0 in one upload but playlist_2 in another.

For robots like the adaptive one they suggest you use the relative path in the file export location maybe we could use that to avoid multiple files of same name?

Would it better to name them with something like?

{name}_#{result['meta']['relativepath']}#{result.fetch('basename')

[janko]

@rmaspero Ok, it seems that this is specific to the /video/adaptive robot.

Since the files get given back to you in a random order there is no way to know which file is which without looping through them all. e.g. if you get playlist_0, playlist_1 and playlist_2 for different uploads the file you are looking for maybe playlist_0 in one upload but playlist_2 in another.

Is there some information that Transloadit gives which we can use to sort these files in the correct order? Do these files even need to be sorted in some specific order?

For robots like the adaptive one they suggest you use the relative path in the file export location maybe we could use that to avoid multiple files of same name?

Doesn't Transloadit already do this automatically? As I understood you cannot control the result filenames, they are always all exported in the same "directory", no?

Would it better to name them with something like?

{name}#{result['meta']['relative_path']}#{result.fetch('basename')

You mean to have this?

{
  "playlist480x270_217792_30480x270_217792_30": Shrine::UploadedFile.new(...),
  "playlist640x360_268584_30640x360_268584_30": Shrine::UploadedFile.new(...),
  # ...
  "playlist960x540_764232_30seg_1": Shrine::UploadedFile.new(...)
}

I'm not sure how that would help, it looks pretty terrible if you ask me 😛

[janko]

Note that once the record has been updated with the processed files, you can still access all these information via the transloadit metadata key, Shrine copies the meta hash into it:

video.file[:playlist_0].metadata["transloadit"] #=>
# {
#   "duration": 15.71,
#   "width": 500,
#   "height": 282,
#   "framerate": 30,
#   "video_bitrate": 764232,
#   "overall_bitrate": 898219,
#   "aspect_ratio": "1.773",
#   "video_codec": "ffh264",
#   "audio_bitrate": 128216,
#   "audio_samplerate": 44100,
#   "audio_channels": 2,
#   "audio_codec": "ffaac",
#   "seekable": true,
#   "date_recorded": null,
#   "date_file_created": "2017/11/08 12:15:34",
#   "date_file_modified": "2017/11/08 19:05:26 GMT",
#   "device_vendor": "Apple",
#   "device_name": null,
#   "device_software": null,
#   "latitude": null,
#   "longitude": null,
#   "rotation": 0,
#   "album": null,
#   "comment": null,
#   "year": null,
#   "encoding_profile": "main",
#   "encoding_level": "3.1",
#   "beats_per_minute": null,
#   "genre": null,
#   "artist": null,
#   "performer": null,
#   "lyrics": null,
#   "title": "My%20Movie%201",
#   "band": null,
#   "disc": null,
#   "track": null,
#   "thumb_index": 0,
#   "thumb_offset": 7.855,
#   "description": null,
#   "location": null,
#   "city": null,
#   "state": null,
#   "country": null,
#   "country_code": null,
#   "keywords": null,
#   "aperture": null,
#   "exposure_compensation": null,
#   "exposure_mode": null,
#   "exposure_time": null,
#   "flash": null,
#   "focal_length": null,
#   "f_number": null,
#   "iso": null,
#   "light_value": null,
#   "metering_mode": null,
#   "shutter_speed": null,
#   "white_balance": null,
#   "orientation": null,
#   "has_clipping_path": false,
#   "creator": null,
#   "author": null,
#   "copyright": null,
#   "copyright_notice": null,
#   "frame_count": 1,
#   "colorspace": "RGB",
#   "average_color": "#7c756d",
#   "xp_title": null,
#   "xp_comment": null,
#   "xp_keywords": null,
#   "xp_subject": null
# }

This means that you should be able to create a helper method that dynamically sorts the files in the way that you want.

[rmaspero]

@janko-m Agreed my suggestion does not make for pretty names at all. I had thought about doing a helper is just didn't seem great way to loop over a lot of returned files each time I want one.

In the adaptive robot you can define a name for the master playlist and that is what most people will want, maybe there is a way of differentiating that file so it is always the same?

Otherwise what I might do is after the result has been saved in shrine, loop over them once and look at the transloadit data to rename my master playlist in the hash.

[janko]

I had thought about doing a helper is just didn't seem great way to loop over a lot of returned files each time I want one.

Yeah, I definitely agree. I think I'm understanding the /video/adaptive output more and more, and I realize now the reasons for your naming suggestion.

It seems that we have 3 different sets of files here:

• The main playlist file (in your example that's ipad_playlist.m3u8)
• Segment files (named seg_<n>.ts)
• Other playlist files (named also with a *.m3u8 extension)

It seems that there is one playlist and set of segment files for each width-height pair, so we should probably somehow include the dimensions in the version name, as you said. We also have segment_index which I presume we can use for sorting inside a single width-height group.

I understand the main playlist file and the segment files, but I'm confused about the role of those other .m3u8 files? Will there be just one per dimension? Why does the main ipad_playlist.m3u8 file have specific dimensions (width/height), isn't it supposed to cover everything?

In the adaptive robot you can define a name for the master playlist and that is what most people will want, maybe there is a way of differentiating that file so it is always the same?

Yeah, I think it's possible, because the transloadit response holds the step parameters, so I think it should be possible to extract the name of the playlist that was passed in (in this example it's ipad_playlist.m3u8).

Maybe we could then have something like this:

class MyUploader < Shrine
  def transloadit_process(io, context)
    hls = transloadit_file(io)
      .add_step("adaptive", "/video/adaptive", ...)

    { hls: hls }
  end
end

{
  hls_master_playlist:   Shrine::UploadedFile.new(...), # points to ipad_playlist.m3u8
  hls_480_270_playlist:  Shrine::UploadedFile.new(...), # points to 480x270_217792_30/480x270_217792_30.m3u8
  hls_640_360_playlist:  Shrine::UploadedFile.new(...), # points to 640x360_268584_30/640x360_268584_30.m3u8
  hls_960_540_playlist:  Shrine::UploadedFile.new(...), # points to 960x540_764232_30/960x540_764232_30.m3u8
  hls_640_360_segment_0: Shrine::UploadedFile.new(...), # points to 640x360_268584_30/seg_0.ts
  hls_640_360_segment_1: Shrine::UploadedFile.new(...), # points to 640x360_268584_30/seg_1.ts
  hls_960_540_segment_0: Shrine::UploadedFile.new(...), # points to 960x540_764232_30/seg_0.ts
  hls_960_540_segment_1: Shrine::UploadedFile.new(...), # points to 960x540_764232_30/seg_1.ts
}

What do you think? It would probably be nice if Shrine supported nested versions, then we could do something like this:

{
  hls: {
    master_playlist: Shrine::UploadedFile.new(...), # points to ipad_playlist.m3u8
    480_270: {
      playlist:      Shrine::UploadedFile.new(...), # points to 480x270_217792_30/480x270_217792_30.m3u8
    },
    640_360: {
      playlist:      Shrine::UploadedFile.new(...), # points to 640x360_268584_30/640x360_268584_30.m3u8,
      segment_0:     Shrine::UploadedFile.new(...), # points to 640x360_268584_30/seg_0.ts
      segment_1:     Shrine::UploadedFile.new(...), # points to 640x360_268584_30/seg_1.ts
    },
    960_540: {
      playlist:      Shrine::UploadedFile.new(...), # points to 960x540_764232_30/960x540_764232_30.m3u8
      segment_0:     Shrine::UploadedFile.new(...), # points to 960x540_764232_30/seg_0.ts
      segment_1:     Shrine::UploadedFile.new(...), # points to 960x540_764232_30/seg_1.ts
    }
  }
}

But maybe that wouldn't be that useful, because as you said we only need to retrieve the master playlist?

[janko]

@rmaspero I posted to early, I updated the comment now.

[rmaspero]

@janko-m So as I understand it basically the purpose of the robot is to take one file. Process it into different quality versions and cut those versions up into segments (segments are typically 10 seconds). Each of those versions get a playlist (the .m3u8 files) which tells the player how to stitch the segments together in the correct order.

Then a master playlist that references these other playlists and defines the quality of each of them so when the video is streamed the player can automatically adjust the stream based upon bandwidth. The segments act as a way for the player to only load the bit of the video it needs in that quality.

I'm not sure why the master playlist has a width etc.. but it seems to match the lowest quality stream. My experience is that it starts streaming the lowest quality first then ups the stream if it can, so maybe that is why it returns that?

Your suggested naming would be ideal. Nesting would be incredible but probably over kill atm. I feel like I have opened a can of worms with this one, sorry.

[janko]

I'm not sure why the master playlist has a width etc.. but it seems to match the lowest quality stream. My experience is that it starts streaming the lowest quality first then ups the stream if it can, so maybe that is why it returns that?

Yes, that's a great explanation. Thanks, the roles of each file are much clearer to me now.

Your suggested naming would be ideal. Nesting would be incredible but probably over kill atm. I feel like I have opened a can of worms with this one, sorry.

No problem, I think it would be great if shrine-transloadit handled /video/adaptive in a way that makes it easier to use the results. About the flat structure, I don't find it ideal that it would be merged with other results (e.g. video thumbnails).

I will see if I can quickly add support for nesting and arrays to the versions plugin, I think this structure would be the ideal to work with:

{
  hls: {
    master_playlist: Shrine::UploadedFile.new(...), # points to ipad_playlist.m3u8
    480_270: {
      playlist: Shrine::UploadedFile.new(...), # points to 480x270_217792_30/480x270_217792_30.m3u8
    },
    640_360: {
      playlist: Shrine::UploadedFile.new(...), # points to 640x360_268584_30/640x360_268584_30.m3u8,
      segments: [
        Shrine::UploadedFile.new(...), # points to 640x360_268584_30/seg_0.ts
        Shrine::UploadedFile.new(...), # points to 640x360_268584_30/seg_1.ts
      ]
    },
    960_540: {
      playlist: Shrine::UploadedFile.new(...), # points to 960x540_764232_30/960x540_764232_30.m3u8
      segments: [
        Shrine::UploadedFile.new(...), # points to 960x540_764232_30/seg_0.ts
        Shrine::UploadedFile.new(...), # points to 960x540_764232_30/seg_1.ts
      ]
    }
  }
}

[janko]

Adding nested versions to Shrine will require more planning than I initially thought. For now I would suggest to use the multiple files feature (which is still unreleased) and use a helper method to dynamically rearrange the structure of read uploaded files based on the metadata (shrine-transloadit automatically saves Transloadit metadata for each file).

[janko]

The transloadit plugin has been completely rewritten for Shrine 3.

It now has a much more explicit API, where you generate steps directly, so you have full control where to export the files. In combination with the new derivatives plugin coming in Shrine 3, you can save Transloadit results in any format you want.

Here is an example of how HLS + thumbnails might look like:

Shrine.plugin :transloadit,
  auth: { key: "...", secret: "..." },
  credentials: { cache: :s3_store, store: :s3_store } # point to "s3_store" Transloadit credentials

Shrine.plugin :derivatives

class VideoUploader < Shrine
  Attacher.transloadit_process do
    import   = file.transloadit_import_step
    adaptive = transloadit_step "adaptive", "/video/adaptive", use: import,
      technique: "hls"
    thumbs   = transloadit_step "thumbs", "/video/thumbs", use: import
    export   = store.transloadit_export_step use: [adaptive, thumbs],
      path: "${file.meta.relative_path}/${file.name}"

    assembly = transloadit.assembly(steps: [import, adaptive, thumbs, export])
    assembly.create!
  end

  Attacher.transloadit_save do |results|
    adaptive = store.transloadit_files(results["adaptive"])
    thumbs   = store.transloadit_files(results["thumbs"])

    merge_derivatives(adaptive: adaptive, thumbs: thumbs)
  end
end

class Video < Sequel::Model
  include VideoUploader::Attachment(:file)
end

video    = Video.create(file: file)
response = video.file_attacher.transloadit_process

# we'll poll for purposes of this example
response.reload_until_finished!

video.file_attacher.transloadit_save(response["results"])
video.save

video.file_derivatives #=> 
# {
#   adaptive: [
#     #<Shrine::UploadedFile @id="480x270_542112_30/480x270_542112_30.m3u8" ...>,
#     #<Shrine::UploadedFile @id="960x540_1570824_30/seg__0.ts" ...>,
#     #<Shrine::UploadedFile @id="my_playlist.m3u8" ...>,
#     ...
#   ],
#   thumbs: [
#     #<Shrine::UploadedFile @id="thumb1.jpg">,
#     #<Shrine::UploadedFile @id="thumb2.jpg">,
#     #<Shrine::UploadedFile @id="thumb3.jpg">,
#     ...
#   ]
# }

I believe this resolves the issue. Shrine 3.0 is expected to be released by the end of October, but in the meantime I released 1.0.0.beta of shrine-transloadit gem with the rewritten API, which relies on 3.0.0.beta2 of shrine gem.