mdhiggins / sickbeard_mp4_automator

Automatically convert video files to a standardized format with metadata tagging to create a beautiful and uniform media library
MIT License
1.53k stars 201 forks source link

Proper Settings for Process Same Extension / Universal Audio #1364

Closed dramirvf closed 3 years ago

dramirvf commented 3 years ago

I have many videos which have once been processed by SMA (Through Radarr Post-Process), therefore "Universal Audio" is over there. But tagging has not been done properly due to different reasons:

  1. It has been previously processed by an earlier version of SMA (which missed some features such as the inclusion of name handlers, etc)
  2. There has been temporary internet disconnection so the script had not received any data from TMDB.
  3. There has been an update in the TMDB database, so I want to re-tag the video file, to include the updated information.

Since the issues are manually discovered after sometime, I don't bother to mess with the configured settings in Radarr and I process above issues manually through running the following command: python3 manual.py -a -i <Video File> -tmdb <TMDB Number>

What is the best AutoProcess.ini setting for such scenarios?

It look like a simple "re-tagging" results in many complications. Is there a less complicated solution to such scenarios which are indeed very common?

mdhiggins commented 3 years ago

Not 100% sure what you mean by "name handlers in tags" but the full tagging job is always performed if metadata is available. If the file was already encoded with SMA then things like stream-level metadata (IE stream titles, language etc) should already be correct in which case you're fine to have process-same-extensions set to False.

If you want to rewrite the stream level data, which I guess for old versions before this was implemented may be desired, then you're going to need to temporarily disable Universal Audio. If you want to preserve things exactly as they were then you should also turn off sort-streams or any other options that may filter out your audio channels you've previously created.

In short, there's no simple solution for this specific setup. But if you wanted you could create an alternate configuration file specifically for retagging and disable the features that cause output results to not be what you're looking for, and then when you do the retagging pass just point manual.py to that. That's what I do on my personal setup.

Also be aware the process-same-extension option does have another check that looks to see if the file was previously encoded with SMA (by checking the 'encoder' tag in the source metadata, only mp4 format supported) and will still prevent an FFMPEG pass unless force-convert is turned on as well

mdhiggins commented 3 years ago

Really the only situation where you'll need to do more than just the mutagen pass though is old files that don't have the stream level metadata you're looking for

dramirvf commented 3 years ago

Not 100% sure what you mean by "name handlers in tags" but the full tagging job is always performed if metadata is available. If the file was already encoded with SMA then things like stream-level metadata (IE stream titles, language etc) should already be correct in which case you're fine to have process-same-extensions set to False.

If you want to rewrite the stream level data, which I guess for old versions before this was implemented may be desired, then you're going to need to temporarily disable Universal Audio. If you want to preserve things exactly as they were then you should also turn off sort-streams or any other options that may filter out your audio channels you've previously created.

In short, there's no simple solution for this specific setup. But if you wanted you could create an alternate configuration file specifically for retagging and disable the features that cause output results to not be what you're looking for, and then when you do the retagging pass just point manual.py to that. That's what I do on my personal setup.

Also be aware the process-same-extension option does have another check that looks to see if the file was previously encoded with SMA (by checking the 'encoder' tag in the source metadata, only mp4 format supported) and will still prevent an FFMPEG pass unless force-convert is turned on as well

I mean 'handler_name' which has been discussed in this thread: https://github.com/mdhiggins/sickbeard_mp4_automator/issues/1351 and after that you released an update for the script.

Here is a sample file info before re-tagging (process-same-extension = False):

Microsoft Windows [Version 10.0.19041.630]
(c) 2020 Microsoft Corporation. All rights reserved.

C:\Program Files\MediaInfo_CLI_20.09_Windows_x64>MediaInfo.exe -i "Z:\Movies - Non-English\Period. End of Sentence. (2018)\Test.mp4"
General
Complete name                            : Z:\Movies - Non-English\Period. End of Sentence. (2018)\Test.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 785 MiB
Duration                                 : 25 min 35 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 4 290 kb/s
Movie name                               : Period. End of Sentence.
Genre                                    : Documentary
ContentType                              : Short Film
Recorded date                            : 2018-04-05
Writing application                      : Lavf58.54.100
Cover                                    : Yes
LongDescription                          : In an effort to improve feminine hygiene, a machine that creates low-cost biodegradable sanitary pads is installed in a rural village in Northern India. Using the machine, a group of local women is employed to produce and sell pads, offering them newfound independence and helping to destigmatize menstruation for all.

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 25 min 31 s
Source duration                          : 25 min 31 s
Bit rate mode                            : Variable
Bit rate                                 : 3 653 kb/s
Maximum bit rate                         : 20.0 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Variable
Frame rate                               : 23.976 (24000/1001) FPS
Minimum frame rate                       : 23.810 FPS
Maximum frame rate                       : 24.390 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.073
Stream size                              : 667 MiB (85%)
Source stream size                       : 667 MiB (85%)
Title                                    : FHD
Writing library                          : x264 core 148 r2715 7e75228
Encoding settings                        : cabac=1 / ref=4 / deblock=1:0:0 / analyse=0x3:0x111 / me=umh / subme=10 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=120 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=8 / lookahead_threads=2 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / stitchable=1 / constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=240 / keyint_min=121 / scenecut=0 / intra_refresh=0 / rc_lookahead=240 / rc=crf / mbtree=1 / crf=20.0 / qcomp=0.50 / qpmin=6 / qpmax=51 / qpstep=4 / vbv_maxrate=20000 / vbv_bufsize=25000 / crf_max=0.0 / nal_hrd=vbr / filler=0 / ip_ratio=1.40 / aq=1:0.80
mdhd_Duration                            : 1531614
Codec configuration box                  : avcC

Audio #1
ID                                       : 2
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Commercial name                          : Dolby Digital
Codec ID                                 : ac-3
Duration                                 : 25 min 35 s
Bit rate mode                            : Constant
Bit rate                                 : 256 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Compression mode                         : Lossy
Stream size                              : 46.9 MiB (6%)
Title                                    : Stereo
Language                                 : Hindi
Service kind                             : Complete Main
Default                                  : No
Alternate group                          : 1

Audio #2
ID                                       : 3
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Commercial name                          : Dolby Digital
Codec ID                                 : ac-3
Duration                                 : 25 min 35 s
Bit rate mode                            : Constant
Bit rate                                 : 384 kb/s
Channel(s)                               : 6 channels
Channel layout                           : L R C LFE Ls Rs
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Compression mode                         : Lossy
Stream size                              : 70.3 MiB (9%)
Title                                    : 5.1 Channel
Language                                 : Hindi
Service kind                             : Complete Main
Default                                  : Yes
Alternate group                          : 1

The same file info after re-tagging ('handler_name' info is missing):

Microsoft Windows [Version 10.0.19041.630]
(c) 2020 Microsoft Corporation. All rights reserved.

C:\Program Files\MediaInfo_CLI_20.09_Windows_x64>MediaInfo.exe -i "Z:\Movies - Non-English\Period. End of Sentence. (2018)\Test.mp4"
General
Complete name                            : Z:\Movies - Non-English\Period. End of Sentence. (2018)\Test.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 785 MiB
Duration                                 : 25 min 35 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 4 290 kb/s
Movie name                               : Period. End of Sentence.
Director                                 : Rayka Zehtabchi
Actor                                    : Arunachalam Muruganatham / Ajeya / Anita / Gouri Choudari / Shabana Khan
Producer                                 : Stacey Sher / Douglas Blush / Garrett K. Schiff / Guneet Monga / Rayka Zehtabchi
Genre                                    : Documentary
ContentType                              : Short Film
Recorded date                            : 2018-04-05
Writing application                      : SMA:Test.mp4
Cover                                    : Yes
ContentRating                            : mpaa|Not Rated|000
LongDescription                          : In an effort to improve feminine hygiene, a machine that creates low-cost biodegradable sanitary pads is installed in a rural village in Northern India. Using the machine, a group of local women is employed to produce and sell pads, offering them newfound independence and helping to destigmatize menstruation for all.

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 25 min 31 s
Source duration                          : 25 min 31 s
Bit rate mode                            : Variable
Bit rate                                 : 3 653 kb/s
Maximum bit rate                         : 20.0 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Variable
Frame rate                               : 23.976 (24000/1001) FPS
Minimum frame rate                       : 23.810 FPS
Maximum frame rate                       : 24.390 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.073
Stream size                              : 667 MiB (85%)
Source stream size                       : 667 MiB (85%)
Title                                    : FHD
Writing library                          : x264 core 148 r2715 7e75228
Encoding settings                        : cabac=1 / ref=4 / deblock=1:0:0 / analyse=0x3:0x111 / me=umh / subme=10 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=120 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=8 / lookahead_threads=2 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / stitchable=1 / constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=240 / keyint_min=121 / scenecut=0 / intra_refresh=0 / rc_lookahead=240 / rc=crf / mbtree=1 / crf=20.0 / qcomp=0.50 / qpmin=6 / qpmax=51 / qpstep=4 / vbv_maxrate=20000 / vbv_bufsize=25000 / crf_max=0.0 / nal_hrd=vbr / filler=0 / ip_ratio=1.40 / aq=1:0.80
mdhd_Duration                            : 1531614
Codec configuration box                  : avcC

Audio #1
ID                                       : 2
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Commercial name                          : Dolby Digital
Codec ID                                 : ac-3
Duration                                 : 25 min 35 s
Bit rate mode                            : Constant
Bit rate                                 : 256 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Compression mode                         : Lossy
Stream size                              : 46.9 MiB (6%)
Title                                    : Stereo
Language                                 : Hindi
Service kind                             : Complete Main
Default                                  : No
Alternate group                          : 1

Audio #2
ID                                       : 3
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Commercial name                          : Dolby Digital
Codec ID                                 : ac-3
Duration                                 : 25 min 35 s
Bit rate mode                            : Constant
Bit rate                                 : 384 kb/s
Channel(s)                               : 6 channels
Channel layout                           : L R C LFE Ls Rs
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Compression mode                         : Lossy
Stream size                              : 70.3 MiB (9%)
Title                                    : 5.1 Channel
Language                                 : Hindi
Service kind                             : Complete Main
Default                                  : Yes
Alternate group                          : 1

The same file (with another name) after two times of re-tagging which has proper 'handler_name' tagging in the expense of two additional audio streams (process-same-extension = True):

Microsoft Windows [Version 10.0.19041.630]
(c) 2020 Microsoft Corporation. All rights reserved.

C:\Program Files\MediaInfo_CLI_20.09_Windows_x64>MediaInfo.exe -i "Z:\Movies - Non-English\Period. End of Sentence. (2018)\Period. End of Sentence. (2018) WEBDL-1080p [8-Bit][x264][AC3 2.0][TMDB-524288].mp4"
General
Complete name                            : Z:\Movies - Non-English\Period. End of Sentence. (2018)\Period. End of Sentence. (2018) WEBDL-1080p [8-Bit][x264][AC3 2.0][TMDB-524288].mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 879 MiB
Duration                                 : 25 min 35 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 4 804 kb/s
Movie name                               : Period. End of Sentence.
Director                                 : Rayka Zehtabchi
Actor                                    : Arunachalam Muruganatham / Ajeya / Anita / Gouri Choudari / Shabana Khan
Producer                                 : Stacey Sher / Douglas Blush / Garrett K. Schiff / Guneet Monga / Rayka Zehtabchi
Genre                                    : Documentary
ContentType                              : Short Film
Recorded date                            : 2018-04-05
Writing application                      : SMA:Period. End of Sentence. (2018) WEBDL-1080p [8-Bit][x264][AC3 2.0][TMDB-524288].mp4
Cover                                    : Yes
ContentRating                            : mpaa|Not Rated|000
LongDescription                          : In an effort to improve feminine hygiene, a machine that creates low-cost biodegradable sanitary pads is installed in a rural village in Northern India. Using the machine, a group of local women is employed to produce and sell pads, offering them newfound independence and helping to destigmatize menstruation for all.

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 25 min 31 s
Source duration                          : 25 min 31 s
Bit rate mode                            : Variable
Bit rate                                 : 3 653 kb/s
Maximum bit rate                         : 20.0 Mb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Variable
Frame rate                               : 23.976 (24000/1001) FPS
Minimum frame rate                       : 23.810 FPS
Maximum frame rate                       : 24.390 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.073
Stream size                              : 667 MiB (76%)
Source stream size                       : 667 MiB (76%)
Title                                    : FHD / FHD
Writing library                          : x264 core 148 r2715 7e75228
Encoding settings                        : cabac=1 / ref=4 / deblock=1:0:0 / analyse=0x3:0x111 / me=umh / subme=10 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=120 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=8 / lookahead_threads=2 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / stitchable=1 / constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=240 / keyint_min=121 / scenecut=0 / intra_refresh=0 / rc_lookahead=240 / rc=crf / mbtree=1 / crf=20.0 / qcomp=0.50 / qpmin=6 / qpmax=51 / qpstep=4 / vbv_maxrate=20000 / vbv_bufsize=25000 / crf_max=0.0 / nal_hrd=vbr / filler=0 / ip_ratio=1.40 / aq=1:0.80
mdhd_Duration                            : 1531614
Codec configuration box                  : avcC

Audio #1
ID                                       : 2
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Commercial name                          : Dolby Digital
Codec ID                                 : ac-3
Duration                                 : 25 min 35 s
Bit rate mode                            : Constant
Bit rate                                 : 256 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Compression mode                         : Lossy
Stream size                              : 46.9 MiB (5%)
Title                                    : Stereo / Stereo
Language                                 : Hindi
Service kind                             : Complete Main
Default                                  : No
Alternate group                          : 1

Audio #2
ID                                       : 3
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Commercial name                          : Dolby Digital
Codec ID                                 : ac-3
Duration                                 : 25 min 35 s
Bit rate mode                            : Constant
Bit rate                                 : 384 kb/s
Channel(s)                               : 6 channels
Channel layout                           : L R C LFE Ls Rs
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Compression mode                         : Lossy
Stream size                              : 70.3 MiB (8%)
Title                                    : 5.1 Channel / 5.1 Channel
Language                                 : Hindi
Service kind                             : Complete Main
Default                                  : Yes
Alternate group                          : 1

Audio #3
ID                                       : 4
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Commercial name                          : Dolby Digital
Codec ID                                 : ac-3
Duration                                 : 25 min 35 s
Bit rate mode                            : Constant
Bit rate                                 : 256 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Compression mode                         : Lossy
Stream size                              : 46.9 MiB (5%)
Title                                    : Stereo / Stereo
Language                                 : Hindi
Service kind                             : Complete Main
Default                                  : No
Alternate group                          : 1

Audio #4
ID                                       : 5
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Commercial name                          : Dolby Digital
Codec ID                                 : ac-3
Duration                                 : 25 min 35 s
Bit rate mode                            : Constant
Bit rate                                 : 256 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 31.250 FPS (1536 SPF)
Compression mode                         : Lossy
Stream size                              : 46.9 MiB (5%)
Title                                    : Stereo / Stereo
Language                                 : Hindi
Service kind                             : Complete Main
Default                                  : No
Alternate group                          : 1
dramirvf commented 3 years ago

Not 100% sure what you mean by "name handlers in tags" but the full tagging job is always performed if metadata is available. If the file was already encoded with SMA then things like stream-level metadata (IE stream titles, language etc) should already be correct in which case you're fine to have process-same-extensions set to False.

If you want to rewrite the stream level data, which I guess for old versions before this was implemented may be desired, then you're going to need to temporarily disable Universal Audio. If you want to preserve things exactly as they were then you should also turn off sort-streams or any other options that may filter out your audio channels you've previously created.

In short, there's no simple solution for this specific setup. But if you wanted you could create an alternate configuration file specifically for retagging and disable the features that cause output results to not be what you're looking for, and then when you do the retagging pass just point manual.py to that. That's what I do on my personal setup.

Also be aware the process-same-extension option does have another check that looks to see if the file was previously encoded with SMA (by checking the 'encoder' tag in the source metadata, only mp4 format supported) and will still prevent an FFMPEG pass unless force-convert is turned on as well

How could I point manual.py to another AutoProcess.ini in the following command? python3 manual.py -a -i <Video File> -tmdb <TMDB Number>

I know how to do it through bash script, but since I pass other info (such as TMDB Number), through a pop-up input in WINSCP, I need to simplify the process into a command which could run through WINSCP Custom Commands.

mdhiggins commented 3 years ago

This is outlined in the readme

--config Specify an alternate configuration file location

dramirvf commented 3 years ago

This is outlined in the readme

--config Specify an alternate configuration file location

I had a video file which did not have proper "Audio Title". I tried to tag the file with process-same-extensions = False, so SMA could assign the proper audio title to the file. But it didn't assign any audio titles.

Then, according to your clue on --config, in my command, I included the path to AutoProcess-Special.ini which contains the following relevant info:

output-format = mp4
output-extension = mp4
delete-original = True
sort-streams = False
process-same-extensions = True
force-convert = False

[Audio]
codec = AC3, DTS, EAC3, AAC

[Universal Audio]
codec = 

Unfortunately I ended up with a video file which has no audio tracks!!! Additionally, the video does not show video track while playing (although it contains video track).

Here is the MediaInfo results after tagging is finished:

Microsoft Windows [Version 10.0.19041.630]
(c) 2020 Microsoft Corporation. All rights reserved.

C:\Program Files\MediaInfo_CLI_20.09_Windows_x64>MediaInfo.exe -i "Z:\Done\Motion Sickness (2010)\Motion Sickness (2010) WEBRip-1080p [8-Bit][x264][AAC 2.0][TMDB-193579].mp4"
General
Complete name                            : Z:\Done\Motion Sickness (2010)\Motion Sickness (2010) WEBRip-1080p [8-Bit][x264][AAC 2.0][TMDB-193579].mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 1.99 GiB
Duration                                 : 1 h 53 min
Overall bit rate                         : 2 502 kb/s
Movie name                               : Motion Sickness
Genre                                    : Fantasy
ContentType                              : Short Film
Recorded date                            : 2010-05-14
Writing application                      : SMA
Cover                                    : Yes
LongDescription                          : Lost on a train of thought tunneling through the dark night of his unconscious, a former rabbinical student struggles to resolve a paralyzing conflict between his mind and his body.

Video
ID                                       : 2
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : @L12.3 / @L24.2 / High@L4.1
MultiView_Count                          : 1 / 1
HDR format                               : SMPTE ST 2086, HDR10 compatible
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 1 h 53 min
Source duration                          : 1 h 53 min
Bit rate                                 : 2 396 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 (23976/1000) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.048
Stream size                              : 1.90 GiB (96%)
Source stream size                       : 1.98 GiB (100%)
Title                                    : FHD / FHD
Mastering display color primaries        : R: x=0.901260 y=0.032720, G: x=0.390580 y=0.728360, B: x=0.010700 y=0.446480, White point: x=1.295060 y=1.169880
Mastering display luminance              : min: 409121.5794 cd/m2, max: 384178 cd/m2
Maximum Content Light Level              : 30780 cd/m2
Maximum Frame-Average Light Level        : 54609 cd/m2
mdhd_Duration                            : 6818778
Codec configuration box                  : avcC

And here are SMA logs:

2020-11-23 10:27:37 - MANUAL - INFO - Manual processor started.
2020-11-23 10:27:37 - MANUAL - INFO - /usr/local/bin/python3
2020-11-23 10:27:37 - MANUAL - INFO - Loading config file /volume1/Amir Synology/Local SMA/config/autoProcess-Special.ini.
2020-11-23 10:27:51 - MANUAL - INFO - Matched movie title as: Motion Sickness 2010-05-14 (TMDB ID: 193579)
2020-11-23 10:27:51 - MANUAL - INFO - Processing Motion Sickness
2020-11-23 10:27:51 - MANUAL - INFO - Input and output extensions match and the file appears to have already been processed by SMA, enable force-convert to override [force-convert: False].
2020-11-23 10:27:51 - MANUAL - INFO - Bypassing conversion and setting outputfile to inputfile.
2020-11-23 10:27:51 - MANUAL - INFO - Tagging file: Motion Sickness (2010) WEBRip-1080p [8-Bit][x264][AAC 2.0][TMDB-193579].mp4.
2020-11-23 10:28:12 - MANUAL - INFO - Trying to write tags.
2020-11-23 10:28:18 - MANUAL - ERROR - Unable to guess movie information
Traceback (most recent call last):
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fbd9c05eee0>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.themoviedb.org', port=443): Max retries exceeded with url: /3/movie/193579?language=en&api_key=45e408d2851e968e6e4d0353ce621c66 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd9c05eee0>: Failed to establish a new connection: [Errno 110] Connection timed out'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/volume1/Amir Synology/Local SMA/manual.py", line 161, in guessInfo
    return movieInfo(guess, tmdbid=tmdbid, imdbid=imdbid)
  File "/volume1/Amir Synology/Local SMA/manual.py", line 189, in movieInfo
    metadata = Metadata(MediaType.Movie, tmdbid=tmdbid, imdbid=imdbid, language=settings.taglanguage, logger=log)
  File "/volume1/Amir Synology/Local SMA/resources/metadata.py", line 68, in __init__
    self.moviedata = query.info(language=self.language)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/tmdbsimple/movies.py", line 72, in info
    response = self._GET(path, kwargs)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/tmdbsimple/base.py", line 93, in _GET
    return self._request('GET', path, params=params)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/tmdbsimple/base.py", line 83, in _request
    response = requests.request(
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.themoviedb.org', port=443): Max retries exceeded with url: /3/movie/193579?language=en&api_key=45e408d2851e968e6e4d0353ce621c66 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd9c05eee0>: Failed to establish a new connection: [Errno 110] Connection timed out'))
2020-11-23 10:28:19 - MANUAL - INFO - Processing file Motion Sickness (2010) WEBRip-1080p [8-Bit][x264][AAC 2.0][TMDB-193579].mp4
2020-11-23 10:28:19 - MANUAL - INFO - Input Data
2020-11-23 10:28:19 - MANUAL - INFO - {
    "format": "mov,mp4,m4a,3gp,3g2,mj2",
    "format-fullname": "QuickTime / MOV",
    "video": {
        "index": 0,
        "codec": "h264",
        "bitrate": 2498638,
        "pix_fmt": "unknown",
        "profile": "unknown",
        "fps": 23.976,
        "dimensions": "1920x1080",
        "level": -9.9,
        "field_order": "unknown"
    },
    "audio": [
        {
            "index": 1,
            "codec": "ac3",
            "bitrate": 48000,
            "channels": 5,
            "samplerate": 44100,
            "language": "eng",
            "disposition": "-default-dub-original+comment-lyrics-karaoke-forced-hearing_impaired-visual_impaired"
        }
    ],
    "subtitle": [],
    "attachment": []
}
2020-11-23 10:28:19 - MANUAL - INFO - Reading video stream.
2020-11-23 10:28:19 - MANUAL - INFO - Video codec detected: h264.
2020-11-23 10:28:19 - MANUAL - INFO - Pix Fmt: unknown.
2020-11-23 10:28:19 - MANUAL - INFO - Profile: unknown.
2020-11-23 10:28:19 - MANUAL - INFO - Video codec parameters None.
2020-11-23 10:28:19 - MANUAL - INFO - Creating copy video stream from source stream 0.
2020-11-23 10:28:19 - MANUAL - INFO - Reading audio streams.
2020-11-23 10:28:19 - MANUAL - INFO - Audio detected for stream 1 - ac3 eng 5 channel.
2020-11-23 10:28:19 - MANUAL - INFO - Creating copy audio stream from source stream 1.
2020-11-23 10:28:19 - MANUAL - INFO - Default audio stream set to eng copy 5 channel stream [default-more-channels: True].
2020-11-23 10:28:19 - MANUAL - INFO - Reading subtitle streams.
2020-11-23 10:28:19 - MANUAL - INFO - Scanned for external subtitles and found 0 results in your approved languages.
2020-11-23 10:28:19 - MANUAL - INFO - Output Data
2020-11-23 10:28:19 - MANUAL - INFO - {
    "source": [
        "Motion Sickness (2010) WEBRip-1080p [8-Bit][x264][AAC 2.0][TMDB-193579].mp4"
    ],
    "format": "mp4",
    "video": {
        "codec": "copy",
        "map": 0,
        "bitrate": 2579.34785,
        "crf": -1,
        "maxrate": null,
        "bufsize": null,
        "level": 0.0,
        "profile": null,
        "preset": null,
        "pix_fmt": null,
        "field_order": "unknown",
        "width": null,
        "filter": null,
        "params": null,
        "title": "FHD",
        "debug": "video"
    },
    "audio": [
        {
            "map": 1,
            "codec": "copy",
            "channels": 5,
            "bitrate": 1280,
            "filter": null,
            "samplerate": null,
            "language": "eng",
            "disposition": "+default-dub-original+comment-lyrics-karaoke-forced-hearing_impaired-visual_impaired",
            "bsf": null,
            "title": "4.1 Channel (Commentary)",
            "debug": "audio"
        }
    ],
    "subtitle": [],
    "attachment": []
}
2020-11-23 10:28:19 - MANUAL - INFO - Preopts
2020-11-23 10:28:19 - MANUAL - INFO - []
2020-11-23 10:28:19 - MANUAL - INFO - Postopts
2020-11-23 10:28:19 - MANUAL - INFO - [
    "-threads",
    "0",
    "-metadata:g",
    "encoding_tool=SMA"
]
2020-11-23 10:28:19 - MANUAL - INFO - Subtitle Extracts
2020-11-23 10:28:19 - MANUAL - INFO - []
2020-11-23 10:28:19 - MANUAL - INFO - Starting conversion.
2020-11-23 10:28:19 - MANUAL - INFO - FFmpeg command:
2020-11-23 10:28:19 - MANUAL - INFO - ======================
2020-11-23 10:28:19 - MANUAL - INFO - /usr/local/bin/ffmpeg -i Motion Sickness (2010) WEBRip-1080p [8-Bit][x264][AAC 2.0][TMDB-193579].mp4.original -vcodec copy -map 0:0 -metadata:s:v title=FHD -metadata:s:v handler_name=FHD -c:a:0 copy -map 0:1 -metadata:s:a:0 title=4.1 Channel (Commentary) -metadata:s:a:0 handler_name=4.1 Channel (Commentary) -metadata:s:a:0 language=eng -disposition:a:0 +default-dub-original+comment-lyrics-karaoke-forced-hearing_impaired-visual_impaired-captions -f mp4 -threads 0 -metadata:g encoding_tool=SMA -y /volume1/video/Done/Motion Sickness (2010)/Motion Sickness (2010) WEBRip-1080p [8-Bit][x264][AAC 2.0][TMDB-193579].mp4
2020-11-23 10:28:19 - MANUAL - INFO - ======================
2020-11-23 10:28:19 - MANUAL - INFO - Tags written successfully using mutagen.
2020-11-23 10:28:19 - MANUAL - ERROR - Unable to set new file permissions.
Traceback (most recent call last):
  File "/volume1/Amir Synology/Local SMA/resources/mediaprocessor.py", line 1248, in setPermissions
    os.chmod(path, self.settings.permissions.get('chmod', int('0664', 8)))
FileNotFoundError: [Errno 2] No such file or directory: 'Motion Sickness (2010) WEBRip-1080p [8-Bit][x264][AAC 2.0][TMDB-193579].mp4'
2020-11-23 10:28:28 - MANUAL - INFO - /volume1/video/Done/Motion Sickness (2010)/Motion Sickness (2010) WEBRip-1080p [8-Bit][x264][AAC 2.0][TMDB-193579].mp4 created.
2020-11-23 10:28:29 - MANUAL - INFO - Relocating MOOV atom to start of file.

I don't have any backups of the file to show how it was before re-tagging. But I confirm that it was a normal video file with audio track and was playing normally without any issues.

Any ideas on what has happened?

dramirvf commented 3 years ago

Update: Since I manually did the re-tagging, Radarr still has the old file info as follows: image

mdhiggins commented 3 years ago

Did you have radarr rescan?

image

mdhiggins commented 3 years ago

As far as the file that didn't work, it does look like it gave it an audio track

    "audio": [
        {
            "map": 1,
            "codec": "copy",
            "channels": 5,
            "bitrate": 1280,
            "filter": null,
            "samplerate": null,
            "language": "eng",
            "disposition": "+default-dub-original+comment-lyrics-karaoke-forced-hearing_impaired-visual_impaired",
            "bsf": null,
            "title": "4.1 Channel (Commentary)",
            "debug": "audio"
        }

You also ran into permission errors which may be impacting things, though not 100% sure why you encountered that,

2020-11-23 10:28:19 - MANUAL - ERROR - Unable to set new file permissions.
Traceback (most recent call last):
  File "/volume1/Amir Synology/Local SMA/resources/mediaprocessor.py", line 1248, in setPermissions
    os.chmod(path, self.settings.permissions.get('chmod', int('0664', 8)))
FileNotFoundError: [Errno 2] No such file or directory: 'Motion Sickness (2010) WEBRip-1080p [8-Bit][x264][AAC 2.0][TMDB-193579].mp4'

But as far as the options generated it looks like you should have had an video and audio track that were both just remuxed

Turn on debug level logging and rerun the job and post your full autoProcess.ini and full conversion logs with DEBUG enabled

mdhiggins commented 3 years ago

https://github.com/mdhiggins/sickbeard_mp4_automator/wiki/Debug-Level-Logging

dramirvf commented 3 years ago

Did you have radarr rescan?

image

It's very weird that even after refresh, Radarr did not detect the absence of Audio Track in the file. Maybe Radarr checks detailed MediaInfo only during import.

I put the same file on my re-download list to see how it was originally.

I'll change the log level to Debug to have better data in re-tagging. Meanwhile, I have created different AutoProcess.ini files and I'll check re-tagging with different settings on other files.

dramirvf commented 3 years ago

As far as the file that didn't work, it does look like it gave it an audio track

    "audio": [
        {
            "map": 1,
            "codec": "copy",
            "channels": 5,
            "bitrate": 1280,
            "filter": null,
            "samplerate": null,
            "language": "eng",
            "disposition": "+default-dub-original+comment-lyrics-karaoke-forced-hearing_impaired-visual_impaired",
            "bsf": null,
            "title": "4.1 Channel (Commentary)",
            "debug": "audio"
        }

You also ran into permission errors which may be impacting things, though not 100% sure why you encountered that,

2020-11-23 10:28:19 - MANUAL - ERROR - Unable to set new file permissions.
Traceback (most recent call last):
  File "/volume1/Amir Synology/Local SMA/resources/mediaprocessor.py", line 1248, in setPermissions
    os.chmod(path, self.settings.permissions.get('chmod', int('0664', 8)))
FileNotFoundError: [Errno 2] No such file or directory: 'Motion Sickness (2010) WEBRip-1080p [8-Bit][x264][AAC 2.0][TMDB-193579].mp4'

But as far as the options generated it looks like you should have had an video and audio track that were both just remuxed

Turn on debug level logging and rerun the job and post your full autoProcess.ini and full conversion logs with DEBUG enabled

Is it possible to have modifications in the script, so when it Tags the Video Files without Conversion (process-same-extensions = False), it also assign "Titles" to currently available "Audio" and "Video" Tracks?

In fact, having "Titles" are generally regarded as "tags", but when SMA only tags the file, it tags everything except assigning titles to "Audio" and "Video" tracks. With the current format of script, assigning Titles for Audio and Video tracks is only achievable if we process the file again (which is not really necessary), and it results in further precautions to avoid double audio, losing arrangement of tracks, etc.

mdhiggins commented 3 years ago

Doesn't really matter what you regard it as, the method for writing tags to a stream is entirely different than the rest of the metadata. One is written using mutagen and the other is done during the FFMPEG creation of the file so there's no way to offload it and no way to just write that tag data to the file, a new file needs to be created to write the stream metadata.

Still think the best bet if this is your specific goal is to just make a special case autoProcess.ini. Should be quite straightforward and I think with a little testing you could easily bulk process to add this specific piece of data.

It's also an open source and object based program, you can totally write your own method to do this specific task and move the rest of the functionality if that's desired, but it should be easily achievable with the correct configuration.

dramirvf commented 3 years ago

I downloaded the same video file again and re-tagged it with a custom AutoProcess.ini. This time there was no issues and everything worked fine. I don't know what had happened during the earlier re-tagging.

I'll keep my logs at DEBUG level and continue my re-tagging with other files. If there was any issues, I'll post here again.

dramirvf commented 3 years ago

debug

Today the same thing happened for another video. The video became corrupted after being tagged with SMA. Before tagging, it was working fine with no issues. I had kept my logs at debug level the whole time. So, I have a better log here:

2020-12-16 16:48:36 - MANUAL - INFO - Manual processor started.
2020-12-16 16:48:36 - MANUAL - INFO - /usr/local/bin/python3
2020-12-16 16:48:36 - MANUAL - INFO - Loading config file /volume1/Amir Synology/Local SMA/config/autoProcess-OnlyTag.ini.
2020-12-16 16:48:36 - MANUAL - DEBUG - TMDBID: 606547
2020-12-16 16:48:36 - MANUAL - DEBUG - IMDBID: None
2020-12-16 16:48:36 - MANUAL - DEBUG - TVDBID: None
2020-12-16 16:48:36 - MANUAL - DEBUG - Using TMDB ID: 606547
2020-12-16 16:50:52 - MANUAL - ERROR - Unable to retrieve rating.
2020-12-16 16:50:52 - MANUAL - INFO - Matched movie title as: 52 Words for Love 2018-10-01 (TMDB ID: 606547)
2020-12-16 16:50:52 - MANUAL - INFO - Processing 52 Words for Love
2020-12-16 16:50:52 - MANUAL - DEBUG - Process started.
2020-12-16 16:50:53 - MANUAL - INFO - Input and output extensions are the same so passing back the original file [process-same-extensions: False].
2020-12-16 16:50:53 - MANUAL - INFO - Bypassing conversion and setting outputfile to inputfile.
2020-12-16 16:50:53 - MANUAL - DEBUG - 52 Words for Love (2018) WEBRip-1080p [10-Bit][x265][AAC 5.1][TMDB-606547].mp4 created from 52 Words for Love (2018) WEBRip-1080p [10-Bit][x265][AAC 5.1][TMDB-606547].mp4 successfully.
2020-12-16 16:50:53 - MANUAL - DEBUG - Height: 872
2020-12-16 16:50:53 - MANUAL - DEBUG - Width: 1920
2020-12-16 16:50:53 - MANUAL - INFO - Tagging file: 52 Words for Love (2018) WEBRip-1080p [10-Bit][x265][AAC 5.1][TMDB-606547].mp4.
2020-12-16 16:50:53 - MANUAL - INFO - Manual processor started.
2020-12-16 16:50:53 - MANUAL - INFO - /usr/local/bin/python3
2020-12-16 16:50:53 - MANUAL - INFO - Loading config file /volume1/Amir Synology/Local SMA/config/autoProcess-OnlyTag.ini.
2020-12-16 16:50:53 - MANUAL - DEBUG - TMDBID: 606547
2020-12-16 16:50:53 - MANUAL - DEBUG - IMDBID: None
2020-12-16 16:50:53 - MANUAL - DEBUG - TVDBID: None
2020-12-16 16:50:53 - MANUAL - DEBUG - Using TMDB ID: 606547
2020-12-16 16:50:56 - MANUAL - ERROR - Unable to retrieve rating.
2020-12-16 16:50:56 - MANUAL - INFO - Matched movie title as: 52 Words for Love 2018-10-01 (TMDB ID: 606547)
2020-12-16 16:50:56 - MANUAL - INFO - Processing 52 Words for Love
2020-12-16 16:50:56 - MANUAL - DEBUG - Process started.
2020-12-16 16:50:56 - MANUAL - INFO - Input and output extensions are the same so passing back the original file [process-same-extensions: False].
2020-12-16 16:50:56 - MANUAL - INFO - Bypassing conversion and setting outputfile to inputfile.
2020-12-16 16:50:56 - MANUAL - DEBUG - 52 Words for Love (2018) WEBRip-1080p [10-Bit][x265][AAC 5.1][TMDB-606547].mp4 created from 52 Words for Love (2018) WEBRip-1080p [10-Bit][x265][AAC 5.1][TMDB-606547].mp4 successfully.
2020-12-16 16:50:56 - MANUAL - DEBUG - Height: 872
2020-12-16 16:50:56 - MANUAL - DEBUG - Width: 1920
2020-12-16 16:50:56 - MANUAL - INFO - Tagging file: 52 Words for Love (2018) WEBRip-1080p [10-Bit][x265][AAC 5.1][TMDB-606547].mp4.
2020-12-16 16:51:07 - MANUAL - DEBUG - Unable to clear original tags, will proceed.
2020-12-16 16:51:09 - MANUAL - INFO - Trying to write tags.
2020-12-16 16:51:10 - MANUAL - INFO - Trying to write tags.
2020-12-16 16:51:15 - MANUAL - INFO - Tags written successfully using mutagen.
2020-12-16 16:51:15 - MANUAL - INFO - Relocating MOOV atom to start of file.
2020-12-16 16:51:15 - MANUAL - WARNING - QT FastStart did not run - perhaps moov atom was at the start already or file is in the wrong format.
2020-12-16 16:51:15 - MANUAL - DEBUG - Final output file: 52 Words for Love (2018) WEBRip-1080p [10-Bit][x265][AAC 5.1][TMDB-606547].mp4.
2020-12-16 16:51:15 - MANUAL - ERROR - There was an error writing the tags.
Traceback (most recent call last):
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/mutagen/mp4/__init__.py", line 427, in __save
    path = atoms.path(b"moov", b"udta", b"meta", b"ilst")
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/mutagen/mp4/_atom.py", line 165, in path
    path.append(path[-1][name, ])
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/mutagen/mp4/_atom.py", line 123, in __getitem__
    raise KeyError("%r not found" % remaining[0])
KeyError: "b'udta' not found"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/volume1/Amir Synology/Local SMA/resources/metadata.py", line 245, in writeTags
    video.save()
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/mutagen/mp4/__init__.py", line 1221, in save
    super(MP4, self).save(*args, **kwargs)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/mutagen/_util.py", line 156, in wrapper
    return func(self, h, *args, **kwargs)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/mutagen/_file.py", line 132, in save
    return self.tags.save(filething, **kwargs)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/mutagen/_util.py", line 185, in wrapper
    return func(*args, **kwargs)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/mutagen/_util.py", line 156, in wrapper
    return func(self, h, *args, **kwargs)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/mutagen/mp4/__init__.py", line 423, in save
    self.__save(filething.fileobj, atoms, data, padding)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/mutagen/mp4/__init__.py", line 429, in __save
    self.__save_new(fileobj, atoms, data, padding)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/mutagen/mp4/__init__.py", line 465, in __save_new
    self.__update_offsets(fileobj, atoms, len(data), offset)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/mutagen/mp4/__init__.py", line 548, in __update_offsets
    self.__update_offset_table(fileobj, ">%dI", atom, delta, offset)
  File "/var/packages/py3k/target/usr/local/lib/python3.8/site-packages/mutagen/mp4/__init__.py", line 524, in __update_offset_table
    offsets = struct.unpack(fmt, data[4:])
struct.error: unpack requires a buffer of 0 bytes
2020-12-16 16:51:15 - MANUAL - INFO - Relocating MOOV atom to start of file.
2020-12-16 16:51:15 - MANUAL - WARNING - QT FastStart did not run - perhaps moov atom was at the start already or file is in the wrong format.
2020-12-16 16:51:15 - MANUAL - DEBUG - Final output file: 52 Words for Love (2018) WEBRip-1080p [10-Bit][x265][AAC 5.1][TMDB-606547].mp4.
mdhiggins commented 3 years ago

This isn't the same error as before and just looks like your source file is missing important headers that's causing mutagen to crash. I see that the file wasn't created by FFMPEG

Input and output extensions are the same so passing back the original file [process-same-extensions: False]

So its probably just a bad source file

dramirvf commented 3 years ago

It's definitely not a bad source file because it has been working fine and even tagged by SMA before. In the earlier tag, information to fetch from TMDB were missing due to the internet connection issue, so I fixed the connection issue and started the process again.

I remember that the earlier issue was also due to the same issue with internet connection which had resulted in the missing TMDB info and I started re-tagging after manual fix in the connection.

It looks like if a process has already been started by SMA, and it is not still finished due to different reasons such as internet connection issues, it is still possible to initiate another process by the same SMA engine through another Putty session for the same file.

In these cases, a file is being processed simultaneously from 2 different sessions which results in a corrupted video output.

Although the first session looks like to be closed, but the process still continues in the background and it takes several minutes for SMA to consider the task as finished.

On Wed., Dec. 16, 2020, 21:58 Michael Higgins, notifications@github.com wrote:

This isn't the same error as before and just looks like your source file is missing important headers that's causing mutagen to crash. I see that the file wasn't created by FFMPEG

Input and output extensions are the same so passing back the original file [process-same-extensions: False]

So its probably just a bad source file

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/mdhiggins/sickbeard_mp4_automator/issues/1364#issuecomment-746780524, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJVMNQ42TO5MOJUL4E2LQYDSVD33FANCNFSM4T62BN5Q .

mdhiggins commented 3 years ago

Well the error here KeyError: "b'udta' not found" is a mp4 atom error and udta is one of the primary parent keys where metadata is stored. Its not specific to any one element or tag. If that's missing then its a corrupt mp4 container. Whether playback worked or not before is irrelevant. Playback may still be fine but at some point that moov data for that file was corrupted

It also looks like QT FastStart did not run - perhaps moov atom was at the start already or file is in the wrong format.

If you're repeatedly tagging movies where you've moved the MOOV data to the front of the file things may eventually be overwritten or run out of space resulting in this behavior and potentially impacting playback if enough data is lost. Its the whole reason the tagging data is written before QTFS is run

dramirvf commented 3 years ago

I downloaded the same file again and it is not being reported as corrupt by mediaInfo, MkvToolnix, or any other video processing application. So similar to the earlier issue, the problem has happened somewhere during the multiple tagging process by SMA.

What is your suggestion to avoid such behaviour in the future?

On Wed., Dec. 16, 2020, 22:55 Michael Higgins, notifications@github.com wrote:

Well the error here KeyError: "b'udta' not found" is a mp4 atom error and udta is one of the primary parent keys where metadata is stored. Its not specific to any one element or tag. If that's missing then its a corrupt mp4 container. Whether playback worked or not before is irrelevant. Playback may still be fine but at some point that moov data for that file was corrupted

It also looks like QT FastStart did not run - perhaps moov atom was at the start already or file is in the wrong format.

If you're repeatedly tagging movies where you've moved the MOOV data to the front of the file things may eventually be overwritten or run out of space resulting in this behavior and potentially impacting playback if enough data is lost. Its the whole reason the tagging data is written before QTFS is run

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/mdhiggins/sickbeard_mp4_automator/issues/1364#issuecomment-746875167, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJVMNQ2SUCVUQUWNIYFG6SLSVECTDANCNFSM4T62BN5Q .

mdhiggins commented 3 years ago

Is the file originally an MP4 or MKV?

Is it being tagged multiple times? tagging really should only happen once so if the tagging is being done multiple times I would adjust how you are handling your workflow. At least make sure the tags are written before QTFS is run

Does it fail every time or is it fine after replacing it? If replacing the file with the original source works perfectly then it was probably just a fluke corruption and you probably don't need to do too much. If its a reproducible and persistent issue then see the above tips and post some more information about the source file

dramirvf commented 3 years ago

My videos are being tagged by SMA through Radarr. For almost 10% of files, I see that the files have not been tagged due to Connection Timeout to TMDB Server according to SMA logs (probably temporary TMDB server overload, temporary internet connection issue, etc).

From my experience on over 10,000 videos being tagged so far, I assume the following scenario applies to this case:

After failed tagging through Radarr, when I try to re-tag manually, if the first attempt is successful (without Connection Timeout Error to TMDB Server), all files would be tagged without any issues.

If the first attempt is again unsuccessful due to connection timeout, my temporary SSH Session becomes unresponsive which results in the termination of inactive session. If another session for tagging is initiated, the output video would become corrupted.

I believe similar issue has happened around 30-40 times for me so far, but I could only focus on 2 of them and missed the other cases since I have been tagging a lot of videos.

On Wed., Dec. 16, 2020, 23:28 Michael Higgins, notifications@github.com wrote:

Is the file originally an MP4 or MKV?

Is it being tagged multiple times? tagging really should only happen once so if the tagging is being done multiple times I would adjust how you are handling your workflow. At least make sure the tags are written before QTFS is run

Does it fail every time or is it fine after replacing it? If replacing the file with the original source works perfectly then it was probably just a fluke corruption and you probably don't need to do too much. If its a reproducible and persistent issue then see the above tips and post some more information about the source file

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/mdhiggins/sickbeard_mp4_automator/issues/1364#issuecomment-746921270, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJVMNQ2PBAF3GDWCSPBDBMTSVEGNZANCNFSM4T62BN5Q .

dramirvf commented 3 years ago

The original file has been MP4.

On Wed., Dec. 16, 2020, 23:28 Michael Higgins, notifications@github.com wrote:

Is the file originally an MP4 or MKV?

Is it being tagged multiple times? tagging really should only happen once so if the tagging is being done multiple times I would adjust how you are handling your workflow. At least make sure the tags are written before QTFS is run

Does it fail every time or is it fine after replacing it? If replacing the file with the original source works perfectly then it was probably just a fluke corruption and you probably don't need to do too much. If its a reproducible and persistent issue then see the above tips and post some more information about the source file

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/mdhiggins/sickbeard_mp4_automator/issues/1364#issuecomment-746921270, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJVMNQ2PBAF3GDWCSPBDBMTSVEGNZANCNFSM4T62BN5Q .

mdhiggins commented 3 years ago

And the original file tags without issue?

A python script timing out should never cause your SSH session to become unresponsive.

I guess one option to consider would be if tagging fails or errors I can block QTFS from running

mdhiggins commented 3 years ago

160f1c8a8c5dfdca2a8cf08454ebe4e3f23b962c

dramirvf commented 3 years ago

It becomes unresponsive because I run the session through WinSCP commands, which is very convenient to use.

If the same original files is downloaded again, it gets tagged without any issue.

On Wed., Dec. 16, 2020, 23:52 Michael Higgins, notifications@github.com wrote:

And the original file tags without issue?

A python script timing out should never cause your SSH session to become unresponsive.

I guess one option to consider would be if tagging fails or errors I can block QTFS from running

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/mdhiggins/sickbeard_mp4_automator/issues/1364#issuecomment-746954961, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJVMNQ3XGANLM5HUO7NNCCLSVEJJ5ANCNFSM4T62BN5Q .

mdhiggins commented 3 years ago

If you use a standard SSH client (not sure specifically what winSCP does, I only use that for FTP functions) you can just ctrl+c to cancel the process if its stuck or hanging

dramirvf commented 3 years ago

WinSCP SSH commands feature is a game changing facility for such custom scenarios. There, you could define a custom command with your desired input. If you click on the created command button in the toolbar, a menu pops up and asks for your desired "TMDB ID" and/or your preferred "AutoProcess.ini" for the current file. Then it runs the command on the remote linux server.

On Wed., Dec. 16, 2020, 23:59 Michael Higgins, notifications@github.com wrote:

If you use a standard SSH client (not sure specifically what winSCP does, I only use that for FTP functions) you can just ctrl+c to cancel the process if its stuck or hanging

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/mdhiggins/sickbeard_mp4_automator/issues/1364#issuecomment-746963420, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJVMNQ2ZX2NKHMXVIGBY4FDSVEKCBANCNFSM4T62BN5Q .