Use video frame as cover artwork

[xinyazhang]

I have several m4a files tagged by iTunes, but I found iTunes embedded cover artworks as the video track. For example, ffprobe outputs the following info for one of these m4a files.

  Duration: 00:02:06.90, start: 0.000000, bitrate: 149 kb/s
    Stream #0:0(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 142 kb/s (default)
    Metadata:
      creation_time   : 2036-09-18 22:48:48
    Stream #0:1: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown), 490x500 [SAR 1:1 DAR 49:50], 90k tbr, 90k tbn, 90k tbc

Can we parse the video track and use its image as the artwork?

P.S. I'll handle this if nobody else are working on it.

[Oleksiy-Yakovenko]

I'm fairly sure that it is possible to get the content of this "video" track, and it will be a simple JPEG file. But it's hard to say for sure without having the file.

Anyway, no, deadbeef can't read album art from mp4 files (yet). But should be possible.

[xinyazhang]

After taking a look of the plugins there are two approaches to implement this.

The first one is to call ffmpeg to address this issue Pros: handles all formats, e.g. an MKV file with XviD video stream. (for real video stream I'm planning to sample an arbitrary frame) Cons: replying on ffmpeg to work, if user disabled or doesn't have ffmpeg then it doesn't work.

The other one is to relay the request to aac plugin, which parses the m4a file. Pros: Works fine for users who wants aac artwork support but doesn't have ffmpeg. Cons: only works for m4a files, only works for limited image formats.

I prefer the first one but OK with both. What do you think of it?

[Oleksiy-Yakovenko]

How about solution 3: call libmp4ff directly to get to the album art?

[Oleksiy-Yakovenko]

Do you have a real example of an audio file with album art encoded as a video stream?

[xinyazhang]

Yes I have, but all of them are copyrighted, but I think I can craft one with iTunes.

[xinyazhang]

Here it is: 01 Short1.m4a.zip

Crafted from MediaCoder Files Test 5, with a doge image from Google images.

Btw, solution 3 has the same pros and cons as solution 2.

[Oleksiy-Yakovenko]

Ah, so you're interested in supporting all the various containers, which in deadbeef are only supported by ffmpeg, right?

Then I don't see many other options, than exposing some sort of album art API from ffmpeg plugin.

[Oleksiy-Yakovenko]

I looked in your file -- the cover art image embedded in it is a regular PNG file, which could be loaded by album art plugin, if the offset was known (this can be relatively easily obtained via libmp4ff)

[xinyazhang]

My motivation is to support artworks tagged by iTunes. However, the tricky part is, it seems iTunes prefers to store album artwork as a video track.

I tried to add cover arts to several mp3 files and ended up with getting an mp3 file with a video track.

I think I'll take the ffmpeg approach since it covers more cases.

[Oleksiy-Yakovenko]

How can you add a video track to a MP3 file?

[xinyazhang]

1. Drag & drop an Mp3 file to iTunes
2. Get info
3. Copy&paste an image to the "album art" tab
4. Check the flashy mp3 file with an video track (through "Show in Finder")

[deleted the file]

[Oleksiy-Yakovenko]

There's a standard embedded APIC frame with PNG file in the ID3v2 tag of your MP3 file. I can't find any indication that this file contains a video track. BTW, deadbeef can read that.

[xinyazhang]

Sorry, I should verify it before claiming.

It seems we only need to solve the m4a problem for now.

On Apr 15, 2016, at 12:49 AM, Alexey Yakovenko notifications@github.com wrote:

There's a standard embedded APIC frame with PNG file in the ID3v2 tag of your MP3 file. I can't find any indication that this file contains a video track.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub

[Oleksiy-Yakovenko]

Yes. And I think the best way to solve this is to use libmp4ff, and not ffmpeg.

[xinyazhang]

I found the code is already there in libmp4ff, but failed to work due to a bug. I'll fix it later.

[Oleksiy-Yakovenko]

That's, unfortunately, the case with most of the functionality in libmp4ff.. Do you need any help with fixing the problem? I'm quite familiar with the code.

[xinyazhang]

Thanks. I think I can do it.

[Oleksiy-Yakovenko]

Since we figured out that we're talking about only supporting album art in mp4 files -- this issue is a duplicate of #1038