[question] Re-use batch-pattern fields

Hello, first of all thanks for maintaining such a great tool! 😄

I'm trying to convert a bunch of files stored in a directory structure "input/%g/%a/%s/%p - %n/". Each of these directories also contains additional files with narrator, comment, copyright info, etc. I would also like to copy the artist tag into the albumartist tag as well, because the audioplayer I use maps the latter to the book writer.

I would like to be able to use something like this:

m4b-tool merge -v "input/" --output-file="output/" \
--batch-pattern="input/%g/%a/%s/%p - %n/" --albumartist="%a" \
--writer="input/%g/%a/%s/%p - %n/narrator.txt" \
--year="input/%g/%a/%s/%p - %n/releasedate.txt" \
--comment="input/%g/%a/%s/%p - %n/comment.txt" \
--copyright="input/%g/%a/%s/%p - %n/copyright.txt"

But this command doesn't work as I intend; the files are converted correctly and contain the right genre, artist, author etc fields, i.e. the ones matched by the input pattern, but fields like comment or copyright are just filled with the strings I provide on the command line, without actually expanding the input pattern fields. The albumartist field, for example, just contains the "%a" string and not the actual artist name. I know I could replace "%a" by "%t", and indeed in this way the albumartist gets filled correctly, but I want to keep both artist and albumartist at the same time.

Another thing I would like to be able to do (not shown above) is to put each output .m4b file in a directory with the same name, i.e. having output files output/%g/%a/%s/%p - %n/%p - %n.m4b instead of the current behavior that generates output files output/%g/%a/%s/%p - %n.m4b.

Of course I can "bypass" the issue by writing a shell script that, for each input directory, passes options to m4b-tool merge as e.g. --comment="$(cat comment.txt)" (which I tested and works correctly, btw), but then it loses generality, requires extra effort and defies the whole point of the --batch-pattern option.

Is there a way, that I'm missing, to accomplish what I intend to do? Thanks in advance! 😃

Wow, this is an interesting approach :-) Well, there is a way that might be a lot easier and which does work full auto (at least in the latest pre-release).

Just put a file ffmetadata.txt into the directory of your audiobook, e.g.: input/Fantasy/J.K. Rowling/Harry Potter/1 - Harry Potter and the Philosophers Stone/ffmetadata.txt

The contents of this file may be (be sure to use UTF-8 as file encoding):

;FFMETADATA1
writer=My Narrator
date=2011/10/01
comment=this is a comment \
with "multiple lines"
copyright=My Copyright

So you have only one file for all the information. See https://ffmpeg.org/ffmpeg-formats.html#Metadata-1 for more details or the following snippet for all supported tags:

"album"
"sort_album" OR "album-sort"
"sort_name" OR "title-sort"
"sort_artist" OR "artist-sort"
"writer" OR "composer"
"genre"
"copyright"
"encoded_by"
"title"
"language"
"artist"
"album_artist"
"performer"
"disc"
"publisher"
"track"
"encoder"
"lyrics"
"date" (e.g. 2011 or 2011/12/31)
"description"
"longdesc" OR "synopsis"

Does this solve your problem?

Hi @sandreas, thank you for helping out so quickly! I've made a script that builds a ffmetadata.txt based on the .txt files I already have for each book, leaving out the tags filled automatically with description.txt and the ones filled automatically by the --batch-pattern option.

It looks like this:

#!/usr/bin/env bash

# Put a '\' before each newline
description=$(awk 1 ORS='\\\n' description.txt)
releasedate=$(awk 1 ORS='\\\n' releasedate.txt)
copyright=$(awk 1 ORS='\\\n' copyright.txt)
narrator=$(awk 1 ORS='\\\n' narrator.txt)

# Remove the last '\'
description=${description%?}
releasedate=${releasedate%?}
copyright=${copyright%?}
narrator=${narrator%?}

# Create a FFMETADATA1 heredoc and spit it out
cat << EOF
;FFMETADATA1
album_artist=XXX
comment=$description
date=$releasedate
copyright=$copyright
composer=$narrator
EOF

By using the latest pre-release version I've got almost everything to work, except that now:

--encoded-by=XXX seems to be ignored, both when passed from the command line and when set through encoder=XXX in ffmetadata.txt.
The comment field is ignored when set through ffmetadata.txt, but it works when set via the command line (btw the comment field is used by my audiobook player to show a synopsis/description, that's why I use it).

I know really know whether this is a bug in m4b-tool or in the version of ffmpeg I'm using:

ffmpeg version 4.2.2 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple LLVM version 10.0.1 (clang-1001.0.46.4)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.2.2-with-options_1 --enable-shared --cc=clang --host-cflags= --host-ldflags= --enable-gpl --enable-libaom --enable-libmp3lame --enable-libopus --enable-libsnappy --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --disable-libjack --disable-indev=jack --enable-opencl --enable-videotoolbox --disable-htmlpages --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfdk-aac --enable-libgsm --enable-libmodplug --enable-libopenh264 --enable-libopenjpeg --enable-librsvg --enable-librtmp --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtesseract --enable-libtwolame --enable-libvidstab --enable-libvmaf --enable-libwavpack --enable-libwebp --enable-libxml2 --enable-libxvid --enable-libzimg --enable-openssl --enable-nonfree --enable-version3
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100

Update: as for the output folder structure, I've reshuffled the files in such a way that now --batch-pattern="input/%g/%a/%s/%p - %n/%p - %n/" should describe by folder structure.

I was hoping, in this way, to get an output folder structure like output/%g/%a/%s/%p - %n/%p - %n.m4b; however, it doesn't work no matter what I try.

I've tried for example to make an input structure input/%g/%a/%s/%p - %n/data/, but also this one doesn't work and I keep getting 0 matches for pattern XXX.

Ok, I think i have to take a closer look at this - although this is a very special use case. Unfortunately I'm pretty busy at the moment, so it may take a while to be fixed (I'm not gonna make predictions here :-))

If you have shell scripting skills, maybe you have PHP skills and try to fix it yourself...? So here are some hints:

See https://github.com/sandreas/m4b-tool#building-from-source for build instructions (might be a bit outdated, but you should get it working)
Now take a look at the files https://github.com/sandreas/m4b-tool/blob/master/src/library/M4bTool/Audio/Tag/InputOptions.php and https://github.com/sandreas/m4b-tool/blob/master/src/library/M4bTool/Command/MergeCommand.php#L696
As you see, the InputOptions is a TagImproverInterface and just takes the existing $tag and ->mergeOverwrites all tags that are present as input option (e.g. --author)
Now you have two options
- Copy the InputOptions.php to MyCustomOptions.php and append it via $tagChanger->add(new MyCustomOptions($this->input, $flags)); (more elegant)
- Just hack your changes to InputOptions itself (to get the current folder, use $currentFolder = new SplFileInfo($this->input->getArgument(static::ARGUMENT_INPUT)); (pretty dirty but much easier)

This may enable you to read the txt files of your structure using file_get_contents... I'm sorry i can't help you further at the moment.

Hi @sandreas, thank again. 😃

I used to have some PHP skills ~10yrs ago, but now they're definitely a bit rusty...😅

I think that so far, for my use case, the fastest thing is a dirty Bash for loop that does the job. But I also like the FFMETADATA1 solution, so if I have some time I'll try to look at why the comment and encoder fields are not getting properly parsed.

Having the possibility to re-use the batch-pattern fields like I originally wanted to do would be a nice boon, too; but I realize this a bit of a low-priority thing and we all have tons of other stuff to do, so no worries...😉

I'll get back here if I have news!

so if I have some time I'll try to look at why the comment and encoder fields are not getting properly parsed

It would be great if you could provide a ffmetadata.txt that does not work for testing purposes. Perhaps this points me into the right direction.

One thing i suspect to be the problem is the part:

comment=$description

Empty lines and lines starting with ‘;’ or ‘#’ are ignored.

Metadata keys or values containing special characters (‘=’, ‘;’, ‘#’, ‘\’ and a newline) must be escaped with a backslash ‘\’.

Ffmetadata1 is tricky to generate and even more tricky to parse. So be sure to also escape =;#\ correctly and also beware of " and ' because it might change the behaviour of the shell script. Perhaps you try with a fixed comment "MyComment" and see if this works... but as i said, I'll take a look if you have the time to wait a little and provide a ffmetadata.txt :-)

Hello again @sandreas, sorry for the late reply.

I've set up a MWE in a folder called MWE that contains the following 3 files:

cover.jpg (take for example the LibriVox cover)
ff-16b-2c-44100hz.mp4 (sample audio file taken from https://docs.espressif.com/projects/esp-adf/en/latest/design-guide/audio-samples.html)

ffmetadata.txt with the following content (since we all like Harry Potter's examples 😄):

;FFMETADATA1
album=Harry Potter and the Philosopher's Stone
album_artist=J.K. Rowling
title=Harry Potter and the Philosopher's Stone
artist=J.K. Rowling
description=Harry Potter has never even heard of Hogwarts...
longdesc=Harry Potter has never even heard of Hogwarts when the letters ...
comment=Harry Potter has never even heard of Hogwarts when the letters start dropping ...
date=2015/12/20
copyright=©1997 J.K. Rowling (P)2015 J.K. Rowling
composer=Stephen Fry
encoder=AlbusDumbledore

Just a MediaInfo dump of the original ff-16b-2c-44100hz.mp4 file:

General
Complete name                            : ####/MWE/ff-16b-2c-44100hz.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/mp41)
File size                                : 2.94 MiB
Duration                                 : 3 min 7 s
Overall bit rate mode                    : Constant
Overall bit rate                         : 132 kb/s
Recorded date                            : 2016-04-12
Writing application                      : Lavf57.27.100

Audio
ID                                       : 1
Format                                   : AAC
Format/Info                              : Advanced Audio Codec
Format profile                           : LC
Codec ID                                 : mp4a-40-2
Duration                                 : 3 min 7 s
Duration_LastFrame                       : -20 ms
Bit rate mode                            : Constant
Bit rate                                 : 132 kb/s
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 44.1 kHz
Frame rate                               : 43.066 FPS (1024 SPF)
Compression mode                         : Lossy
Stream size                              : 2.90 MiB (99%)
Default                                  : Yes
Alternate group                          : 1

Then I've performed the following two conversions:

./m4b-tool.phar --version
m4b-tool latest-19-gdb42c19
mp4tags --version
mp4tags - MP4v2 2.0.0

[Using ffmetadata.txt]

./m4b-tool.phar merge -vvv --force --no-conversion --jobs=1 "MWE" --output-file=MWE.m4b --genre="Fantasy" --series="Harry Potter" --series-part="1"
'ffmpeg' '-hide_banner' '-version'
== load input files ==
searching for cover in MWE
using cover MWE/cover.jpg
skip cover extraction, a custom cover has been specified: MWE/cover.jpg
using cover MWE/cover.jpg
only 1 file in merge list, copying file
'ffmpeg' '-hide_banner' '-i' 'MWE/ff-16b-2c-44100hz.mp4' '-f' 'ffmetadata' '-'
'mp4info' 'MWE/ff-16b-2c-44100hz.mp4'
'ffmpeg' '-hide_banner' '-i' 'MWE/ff-16b-2c-44100hz.mp4' '-f' 'ffmetadata' '-'
tagFile - filename: MWE-tmpfiles/tmp_MWE.m4b
full tag: {"encoder":"m4b-tool","title":"Harry Potter and the Philosopher's Stone","sortTitle":"Harry Potter 1 - Harry Potter and the Philosopher's Stone","artist":"J.K. Rowling","sortArtist":null,"genre":"Fantasy","writer":"Stephen Fry","album":"Harry Potter and the Philosopher's Stone","sortAlbum":"Harry Potter 1 - Harry Potter and the Philosopher's Stone","disk":null,"disks":null,"albumArtist":"J.K. Rowling","year":{"date":"2015-12-20 00:00:00.000000","timezone_type":3,"timezone":"UTC"},"track":null,"tracks":null,"cover":{},"description":"Harry Potter has never even heard of Hogwarts...","longDescription":"Harry Potter has never even heard of Hogwarts when the letters ...","comment":null,"copyright":"\u00a91997 J.K. Rowling (P)2015 J.K. Rowling","encodedBy":null,"type":2,"performer":null,"language":null,"publisher":null,"lyrics":null,"chapters":[{"name":"1","start":0,"length":187109}],"series":"Harry Potter","seriesPart":"1","extraProperties":[],"removeProperties":[]}
'mp4tags' '-help'
'mp4tags' '-A' 'Harry Potter and the Philosopher'\''s Stone' '-a' 'J.K. Rowling' '-s' 'Harry Potter and the Philosopher'\''s Stone' '-g' 'Fantasy' '-w' 'Stephen Fry' '-m' 'Harry Potter has never even heard of Hogwarts...' '-l' 'Harry Potter has never even heard of Hogwarts when the letters ...' '-R' 'J.K. Rowling' '-y' '2015/12/20' '-C' '©1997 J.K. Rowling (P)2015 J.K. Rowling' '-E' 'm4b-tool' '-i' '2' '-sortname' 'Harry Potter 1 - Harry Potter and the Philosopher'\''s Stone' '-sortalbum' 'Harry Potter 1 - Harry Potter and the Philosopher'\''s Stone' 'MWE-tmpfiles/tmp_MWE.m4b'
'mp4chaps' '-i' 'MWE-tmpfiles/tmp_MWE.m4b'
'mp4art' '--remove' '--art-any' 'MWE-tmpfiles/tmp_MWE.m4b'
'mp4art' '--add' 'MWE/cover.jpg' 'MWE-tmpfiles/tmp_MWE.m4b'
tagged file tmp_MWE.m4b (artist: J.K. Rowling, name: Harry Potter and the Philosopher's Stone, chapters: 1)
moved temporary tmp_MWE.m4b to MWE.m4b
successfully merged 1 files to MWE.m4b

[Command-line comment and encoder]

./m4b-tool.phar merge -vvv --force --no-conversion --jobs=1 "MWE" --output-file=MWE.m4b --genre="Fantasy" --series="Harry Potter" --series-part="1" --comment="Harry Potter has never even heard of Hogwarts when the letters start dropping ..." --encoder="AlbusDumbledore"
'ffmpeg' '-hide_banner' '-version'
== load input files ==
searching for cover in MWE
using cover MWE/cover.jpg
skip cover extraction, a custom cover has been specified: MWE/cover.jpg
using cover MWE/cover.jpg
only 1 file in merge list, copying file
'ffmpeg' '-hide_banner' '-i' 'MWE/ff-16b-2c-44100hz.mp4' '-f' 'ffmetadata' '-'
'mp4info' 'MWE/ff-16b-2c-44100hz.mp4'
'ffmpeg' '-hide_banner' '-i' 'MWE/ff-16b-2c-44100hz.mp4' '-f' 'ffmetadata' '-'
tagFile - filename: MWE-tmpfiles/tmp_MWE.m4b
full tag: {"encoder":"AlbusDumbledore","title":"Harry Potter and the Philosopher's Stone","sortTitle":"Harry Potter 1 - Harry Potter and the Philosopher's Stone","artist":"J.K. Rowling","sortArtist":null,"genre":"Fantasy","writer":"Stephen Fry","album":"Harry Potter and the Philosopher's Stone","sortAlbum":"Harry Potter 1 - Harry Potter and the Philosopher's Stone","disk":null,"disks":null,"albumArtist":"J.K. Rowling","year":{"date":"2015-12-20 00:00:00.000000","timezone_type":3,"timezone":"UTC"},"track":null,"tracks":null,"cover":{},"description":"Harry Potter has never even heard of Hogwarts...","longDescription":"Harry Potter has never even heard of Hogwarts when the letters ...","comment":"Harry Potter has never even heard of Hogwarts when the letters start dropping ...","copyright":"\u00a91997 J.K. Rowling (P)2015 J.K. Rowling","encodedBy":null,"type":2,"performer":null,"language":null,"publisher":null,"lyrics":null,"chapters":[{"name":"1","start":0,"length":187109}],"series":"Harry Potter","seriesPart":"1","extraProperties":[],"removeProperties":[]}
'mp4tags' '-help'
'mp4tags' '-A' 'Harry Potter and the Philosopher'\''s Stone' '-a' 'J.K. Rowling' '-s' 'Harry Potter and the Philosopher'\''s Stone' '-g' 'Fantasy' '-w' 'Stephen Fry' '-m' 'Harry Potter has never even heard of Hogwarts...' '-l' 'Harry Potter has never even heard of Hogwarts when the letters ...' '-R' 'J.K. Rowling' '-y' '2015/12/20' '-c' 'Harry Potter has never even heard of Hogwarts when the letters start dropping ...' '-C' '©1997 J.K. Rowling (P)2015 J.K. Rowling' '-E' 'AlbusDumbledore' '-i' '2' '-sortname' 'Harry Potter 1 - Harry Potter and the Philosopher'\''s Stone' '-sortalbum' 'Harry Potter 1 - Harry Potter and the Philosopher'\''s Stone' 'MWE-tmpfiles/tmp_MWE.m4b'
'mp4chaps' '-i' 'MWE-tmpfiles/tmp_MWE.m4b'
'mp4art' '--remove' '--art-any' 'MWE-tmpfiles/tmp_MWE.m4b'
'mp4art' '--add' 'MWE/cover.jpg' 'MWE-tmpfiles/tmp_MWE.m4b'
tagged file tmp_MWE.m4b (artist: J.K. Rowling, name: Harry Potter and the Philosopher's Stone, chapters: 1)
moved temporary tmp_MWE.m4b to MWE.m4b
successfully merged 1 files to MWE.m4b

As you can already see from the verbose output of the two commands, the first one results into

full tag: {
    "encoder":"m4b-tool", [...] 
    "comment":null, [...] 
}

while the second one results into

full tag: {
    "encoder":"AlbusDumbledore", [...]
    "comment":"Harry Potter has never even heard of Hogwarts when the letters start dropping ...", [...] 
}

If you also take a look at the MediaInfo dump of the two files (the first has been renamed to MWE_wrong.m4b, while the second to MWE_correct.m4b):

General
Complete name                            : ####/MWE_wrong.m4b
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/mp41)
File size                                : 2.94 MiB
Duration                                 : 3 min 7 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 132 kb/s
Album                                    : Harry Potter and the Philosopher's Stone
Album/Sorted by                          : Harry Potter 1 - Harry Potter and the Philosopher's Stone
Album/Performer                          : J.K. Rowling
Track name                               : Harry Potter and the Philosopher's Stone
Performer                                : J.K. Rowling
Composer                                 : Stephen Fry
Genre                                    : Fantasy
ContentType                              : Audiobook
Description                              : Harry Potter has never even heard of Hogwarts...
Recorded date                            : 2015/12/20
Tagged date                              : UTC 2020-03-05 16:03:10
Writing application                      : m4b-tool
Copyright                                : ©1997 J.K. Rowling (P)2015 J.K. Rowling
Cover                                    : Yes
Title/Sort                               : Harry Potter 1 - Harry Potter and the Philosopher's Stone
LongDescription                          : Harry Potter has never even heard of Hogwarts when the letters ...

Audio
ID                                       : 1
Format                                   : AAC
Format/Info                              : Advanced Audio Codec
Format profile                           : LC
Codec ID                                 : mp4a-40-2
Duration                                 : 3 min 7 s
Duration_LastFrame                       : -20 ms
Bit rate mode                            : Variable
Bit rate                                 : 132 kb/s
Maximum bit rate                         : 149 kb/s
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 44.1 kHz
Frame rate                               : 43.066 FPS (1024 SPF)
Compression mode                         : Lossy
Stream size                              : 2.90 MiB (99%)
Default                                  : Yes
Alternate group                          : 1
Menus                                    : 2

Menu #1
00:00:00.000                             : 1

Menu #2
ID                                       : 2
Codec ID                                 : text
Duration                                 : 3 min 7 s
Encoded date                             : UTC 2020-03-05 16:03:10
Tagged date                              : UTC 2020-03-05 16:03:10
Menu For                                 : 1
00:00:00.000                             : 1

General
Complete name                            : ####/MWE_correct.m4b
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/mp41)
File size                                : 2.94 MiB
Duration                                 : 3 min 7 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 132 kb/s
Album                                    : Harry Potter and the Philosopher's Stone
Album/Sorted by                          : Harry Potter 1 - Harry Potter and the Philosopher's Stone
Album/Performer                          : J.K. Rowling
Track name                               : Harry Potter and the Philosopher's Stone
Performer                                : J.K. Rowling
Composer                                 : Stephen Fry
Genre                                    : Fantasy
ContentType                              : Audiobook
Description                              : Harry Potter has never even heard of Hogwarts...
Recorded date                            : 2015/12/20
Tagged date                              : UTC 2020-03-05 16:34:08
Writing application                      : AlbusDumbledore
Copyright                                : ©1997 J.K. Rowling (P)2015 J.K. Rowling
Cover                                    : Yes
Comment                                  : Harry Potter has never even heard of Hogwarts when the letters start dropping ...
Title/Sort                               : Harry Potter 1 - Harry Potter and the Philosopher's Stone
LongDescription                          : Harry Potter has never even heard of Hogwarts when the letters ...

Audio
ID                                       : 1
Format                                   : AAC
Format/Info                              : Advanced Audio Codec
Format profile                           : LC
Codec ID                                 : mp4a-40-2
Duration                                 : 3 min 7 s
Duration_LastFrame                       : -20 ms
Bit rate mode                            : Variable
Bit rate                                 : 132 kb/s
Maximum bit rate                         : 149 kb/s
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 44.1 kHz
Frame rate                               : 43.066 FPS (1024 SPF)
Compression mode                         : Lossy
Stream size                              : 2.90 MiB (99%)
Default                                  : Yes
Alternate group                          : 1
Menus                                    : 2

Menu #1
00:00:00.000                             : 1

Menu #2
ID                                       : 2
Codec ID                                 : text
Duration                                 : 3 min 7 s
Encoded date                             : UTC 2020-03-05 16:34:08
Tagged date                              : UTC 2020-03-05 16:34:08
Menu For                                 : 1
00:00:00.000                             : 1

So far I've encountered issues just with these two fields, I don't know if there are also other fields which are not properly parsed.

PS: It seems that fortunately, even if there is a ' in of the fields of ffmetadata.txt, it doesn't create issues. I've tried both with and without it, and the result doesn't change.

Thank you for your huge effort to provide all this information. I try to look into this in the next weeks - unfortunately I'm busy atm.

Perhaps another useful hint: In the latest pre-release you could use the --equate parameter (multiple usage of --equate is supported) to achieve some improvements. --equate takes the first tag of a comma separated list and copies it to all fields that are provided after the first one, e.g --equate=a,b,c,d would copy tag a to tag b, c and d.

So --equate=artist,album_artist --equate=description,comment should copy artist to album_artist and description to comment, if it works correctly. Since there is definitely something wrong with the tagging, i would keep this issue open until i find the time to investigate further, even if this solves your problem.

Hi @sandreas, thanks for the hint on the --equate option, it seems to fit my needs! I'll check it out asap 😃

Ok, the only thing that really did not work, was the comment in ffmetadata. Some further hints:

Although it might sound a bit weird, to keep the encoder it is required to use --encoder= which is an empty parameter value and results in not overwriting the value by m4b-tool(reason is, that most of the time the encoder value coming out of the tags is trash and should be overwritten with m4b-tool)
To re-use --batch-pattern fields you could use the --equate option described above, this also will solve your issue with the missing --comment, if you use --equate=comment,longdesc
The ffmetadata-reader was indeed missing comment, this will be fixed in the next pre-release, which will be uploaded after some other small fixes

That's great, thanks a lot! 😄

Using the latest version:

./m4b-tool.phar --version
m4b-tool latest-27-g34dd3b4
mp4tags --version
mp4tags - MP4v2 2.0.0

It doesn't seem to accept the ffmetadata.txt file. I create a file ffmetadata.txt with the content:

;FFMETADATA1
album=Thrawn: Alliances
album-sort=2 - Thrawn: Alliances
album_artist=Timothy Zahn
title=Thrawn: Alliances
artist=Timothy Zahn
description=Grand Admiral Thrawn and Darth Vader team up against a threat to the Empire in this thrilling novel from best-selling author Timothy Zahn.
longdesc=“I have sensed a disturbance in the Force.”
date=2018/07/24
copyright=Random House Audio
composer=Marc Thompson

My folder structure is:

inputtest
  author
    name 
    series
      name

Inside the name folder (either in a series or just under the author), i have a m4b/mp3/multiple-mp3s, a cover.jpg/png and ffmetadata.txt.

I use the command:

./m4b-tool.phar merge -vvv --force --jobs=16 --output-file="Plex/" --batch-pattern="inputtest/%a/%s/%p - %n/"  --batch-pattern="inputtest/%a/%n/" "inputtest/"

The output using ffprobe shows the regular tags from the batch-pattern (eg. author, name). If the file already contains tags in eg. name, then it isn't even overridden using the new values.

Some more information when running: ./m4b-tool.phar merge -vvv --force --jobs=16 --output-file="Plex/" --batch-pattern="inputtest/%a/%s/%p - %n/" --batch-pattern="inputtest/%a/%n/" "inputtest/"

'ffmpeg' '-hide_banner' '-version'
1 matches for pattern inputtest/%a/%s/%p - %n/
================================
merge inputtest/Timothy Zahn/Thrawn/2 - Thrawn Alliances/
  =>  Plex/Timothy Zahn/Thrawn/2 - Thrawn Alliances.m4b
- name: Thrawn Alliances
- artist: Timothy Zahn
- series: Thrawn
- series-part: 2

================================

================================
0 matches for pattern inputtest/%a/%n/

================================
processing inputtest/Timothy Zahn/Thrawn/2 - Thrawn Alliances/
== load input files ==
searching for cover in inputtest/Timothy Zahn/Thrawn/2 - Thrawn Alliances
using cover inputtest/Timothy Zahn/Thrawn/2 - Thrawn Alliances/Cover.jpg
skip cover extraction, a custom cover has been specified: inputtest/Timothy Zahn/Thrawn/2 - Thrawn Alliances/Cover.jpg
preparing conversion with 16 simultaneous jobs, please wait...
'ffmpeg' '-nostats' '-loglevel' 'panic' '-hide_banner' '-i' 'inputtest/Timothy Zahn/Thrawn/2 - Thrawn Alliances/Thrawn Alliance.mp3' '-map_metadata' '0' '-max_muxing_queue_size' '9999' '-strict' 'experimental' '-movflags' '+faststart' '-vn' '-y' '-ab' '64k' '-ar' '22050' '-ac' '0' '-acodec' 'aac' '-f' 'mp4' 'Plex/Timothy Zahn/Thrawn/2 - Thrawn Alliances-tmpfiles/1-Thrawn Alliance-converting.m4b'
using cover inputtest/Timothy Zahn/Thrawn/2 - Thrawn Alliances/Cover.jpg
only 1 file in merge list, copying file
'ffmpeg' '-hide_banner' '-i' 'Plex/Timothy Zahn/Thrawn/2 - Thrawn Alliances-tmpfiles/1-Thrawn Alliance-finished.m4b' '-f' 'ffmetadata' '-'
'ffmpeg' '-hide_banner' '-i' 'inputtest/Timothy Zahn/Thrawn/2 - Thrawn Alliances/Thrawn Alliance.mp3' '-f' 'ffmetadata' '-'
tagFile - filename: Plex/Timothy Zahn/Thrawn/2 - Thrawn Alliances-tmpfiles/tmp_2 - Thrawn Alliances.m4b
full tag: {"encoder":"m4b-tool","title":"Thrawn Alliances","sortTitle":"Thrawn 2 - Thrawn Alliances","artist":"Timothy Zahn","sortArtist":null,"genre":"Audiobook","writer":null,"album":"Thrawn: Alliances (Star Wars) (Unabridged)","sortAlbum":"Thrawn 2 - Thrawn Alliances","disk":null,"disks":null,"albumArtist":"Timothy Zahn","year":{"date":"2020-05-13 20:18:00.000000","timezone_type":3,"timezone":"Europe\/Berlin"},"track":null,"tracks":null,"cover":{},"description":null,"longDescription":null,"comment":"On Batuu, at the edges of the Unknown Regions, a threat to the Empire is taking root - its existence little more than a glimmer, its consequences as yet unknowable. But it is troubling enough to the Imperial leader to warrant investigation by his most powerful agents....","copyright":"&#169;2018 Timothy Zahn;(P)2018 Random House Audio","encodedBy":null,"type":2,"performer":null,"language":null,"publisher":null,"lyrics":null,"chapters":[{"name":"1","start":0,"length":46901},{"name":"2","start":46901,"length":626753},{"name":"3","start":673654,"length":582217},{"name":"4","start":1255871,"length":2081297},{"name":"5","start":3337168,"length":2224565},{"name":"6","start":5561733,"length":2112830},{"name":"7","start":7674563,"length":2163728},{"name":"8","start":9838291,"length":2562554},{"name":"9","start":12400845,"length":1929950},{"name":"10","start":14330795,"length":2545232},{"name":"11","start":16876027,"length":2773717},{"name":"12","start":19649744,"length":2950745},{"name":"13","start":22600489,"length":2875374},{"name":"14","start":25475863,"length":1906590},{"name":"15","start":27382453,"length":2074192},{"name":"16","start":29456645,"length":1927303},{"name":"17","start":31383948,"length":2159223.0000000037},{"name":"18","start":33543171.000000004,"length":1981497.9999999963},{"name":"19","start":35524669,"length":2193590},{"name":"20","start":37718259,"length":1291075},{"name":"21","start":39009334,"length":2280525},{"name":"22","start":41289859,"length":2053294},{"name":"23","start":43343153,"length":2893067},{"name":"24","start":46236220,"length":851105},{"name":"25","start":47087325,"length":943798},{"name":"26","start":48031123,"length":66873}],"series":"Thrawn","seriesPart":"2","extraProperties":[],"removeProperties":[]}
'mp4tags' '-help'
'mp4tags' '-A' 'Thrawn: Alliances (Star Wars) (Unabridged)' '-a' 'Timothy Zahn' '-s' 'Thrawn Alliances' '-g' 'Audiobook' '-R' 'Timothy Zahn' '-y' '2020/05/13' '-c' 'On Batuu, at the edges of the Unknown Regions, a threat to the Empire is taking root - its existence little more than a glimmer, its consequences as yet unknowable. But it is troubling enough to the Imperial leader to warrant investigation by his most powerful agents....' '-C' '&#169;2018 Timothy Zahn;(P)2018 Random House Audio' '-E' 'm4b-tool' '-i' '2' '-sortname' 'Thrawn 2 - Thrawn Alliances' '-sortalbum' 'Thrawn 2 - Thrawn Alliances' 'Plex/Timothy Zahn/Thrawn/2 - Thrawn Alliances-tmpfiles/tmp_2 - Thrawn Alliances.m4b'
'mp4chaps' '-i' 'Plex/Timothy Zahn/Thrawn/2 - Thrawn Alliances-tmpfiles/tmp_2 - Thrawn Alliances.m4b'
'mp4art' '--remove' '--art-any' 'Plex/Timothy Zahn/Thrawn/2 - Thrawn Alliances-tmpfiles/tmp_2 - Thrawn Alliances.m4b'
'mp4art' '--add' 'inputtest/Timothy Zahn/Thrawn/2 - Thrawn Alliances/Cover.jpg' 'Plex/Timothy Zahn/Thrawn/2 - Thrawn Alliances-tmpfiles/tmp_2 - Thrawn Alliances.m4b'
tagged file tmp_2 - Thrawn Alliances.m4b (artist: Timothy Zahn, name: Thrawn Alliances, chapters: 26)
moved temporary tmp_2 - Thrawn Alliances.m4b to Plex/Timothy Zahn/Thrawn/2 - Thrawn Alliances.m4b
successfully merged 1 files to Plex/Timothy Zahn/Thrawn/2 - Thrawn Alliances.m4b

ffprobe then shows:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '2 - Thrawn Alliances.m4b':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    creation_time   : 2018-07-24T08:24:37.000000Z
    title           : Thrawn Alliances
    artist          : Timothy Zahn
    album_artist    : Timothy Zahn
    album           : Thrawn: Alliances (Star Wars) (Unabridged)
    comment         : On Batuu, at the edges of the Unknown Regions, a threat to the Empire is taking root - its existence little more than a glimmer, its consequences as yet unknowable. But it is troubling enough to the Imperial leader to warrant investigation by his most pow
    genre           : Audiobook
    date            : 2020/05/13
    sort_name       : Thrawn 2 - Thrawn Alliances
    sort_album      : Thrawn 2 - Thrawn Alliances
    copyright       : &#169;2018 Timothy Zahn;(P)2018 Random House Audio
    encoder         : m4b-tool
    media_type      : 2
  Duration: 13:21:38.11, start: 0.000000, bitrate: 66 kb/s
    Chapter #0:0: start 0.000000, end 46.901000
    Metadata:
      title           : 1
    Chapter #0:1: start 46.901000, end 673.654000
    Metadata:
      title           : 2
    Chapter #0:2: start 673.654000, end 1255.871000
 .........
 .........

Hope this gives some insight.

@Drudoo Unfortunately i was not able to reproduce this so far - my ffmetadata.txt worked in all my tests with the latest pre-release - BUT: I'm not giving up. On my way to reproduce i found another VERY NASTY bug in the php-strings library i wrote for m4b-tool which i had to fix and which was a lot of work (complete rewrite of the batch-pattern recognition logic).

After testing this i will retry on your issue, maybe it could already be fixed with the rewrite, but this is unlikely. I just would like to let you know: I'm still on it.

That's so weird that you cant reproduce it. I am using the .phar file on debian. But i get the same errors using the docker container.

If you have any new release to test, i am ready to give some feedback. This is literally the only thing left for my audiobook+plex+ios setup to work perfectly.

Ok, here comes the next pre-release. I did not reproduce the problem, even if i place the files and folders exactly in the same way, you did. (i ran exactly the same command with the same ffmetadata.txt file and it worked for me).

At least m4b-tool now shows logging information for every improver that is used (also ffmetadata), so we should see in the logs, if it is the ffmetadata improver or something else.

I tested the new version and i think it works. I also found the root-cause of the problem, which wasn't the code, but might be the encoding of my metadata file.

First i checked to make sure the ffmetadata file was utf-8 using tr -d \\000-\\177 < ffmetadata.txt | wc -c which showed that out of 3 test files, 1 was not correct encoding. This i fixed by replacing my copied ", ', . and ,. First problem solved.

Then ran ffmpeg -i src.m4b -f ffmetadata ffmetadata.txt.new on one of the books and checked the difference diff ffmetadata.txt ffmetadata.txt.new

This showed a different in the top of the file:

1c1
< ;FFMETADATA1
---
> ;FFMETADATA1

So i ran m4b-tool again using the new ffmetadata.txt.new file. This gave the correct result.

I ended up rewriting all my ffmetadata files from scratch using vim over ssh instead of writing them in sublime on windows and then scp'ing them to my server.

When running the same command as my last post:

./m4b-tool.phar merge -vvv --force --jobs=16 --output-file="Plex/" --batch-pattern="inputtest/%a/%s/%p - %n/" --batch-pattern="inputtest/%a/%n/" "inputtest/"

I can now see in the log that ffmetadata gets the right information:

ffmetadata.txt improved the following 9 properties: title, artist, writer, album, albumArtist, description, longDescription, copyright, type

and using ffprobe on the completed files, shows all the right tags are there.

Each of my metadata files contains:

;FFMETADATA1
genre=
album=
title=
artist=
description=
date=
copyright=
composer=
comment=

Ok, glad to hear m4b-tool was not the problem, but i would love to get one of those "bogus" ffmetadata.txt files from you... maybe i can harden m4b-tool to either work with these files or at least show an error message, that the file could not be parsed correctly. Would you mind sharing one of the bogus files as zip? (if you still got one)

Sure, i never delete anything. I have uploaded who textfiles here. One will be from ffmpeg (ffmetadata.txt.fromffmpeg), the other is the old one (.txt.homemade).

I am thinking it is the line endings or something like that, even though my setting in Sublime Text is set to using unix endings and not windows.

https://mega.nz/file/i81iiaBb#C3akgfsUlE0K69cCBhGnZ2dN8aib9aZV8hqPVam4xVw

@Drudoo The MAIN problem with your file was, that it started with ;FFMETADATA1 (with a trailing space) instead of ;FFMETADATA1 (trimmed). For now i added a trim to be more robust.

BUT: I'm not sure i'll keep the trim, because there is a clear specification that states, ffmetadata-files have to start with ;FFMETADATA1, and I bet that ffmpeg itself would not read your file format... if you are putting these files together manually, you should definitely check out the specs (with escaping, etc.): https://ffmpeg.org/ffmpeg-formats.html#Metadata-1

sandreas / m4b-tool

[question] Re-use batch-pattern fields #72