Post processing fails after renaming in progress hook on status=finished

[jvs3]

What is the purpose of your issue?

• [x] Bug report (encountered problems with youtube-dl)
• [ ] Site support request (request for adding support for a new site)
• [ ] Feature request (request for a new functionality)
• [ ] Question
• [ ] Other

  If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

WARNING:tldextract:unable to cache TLDs in file /usr/local/lib/python2.7/dist-packages/tldextract/.tld_set: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/tldextract/.tld_set'
[debug] Encodings: locale UTF-8, fs UTF-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2016.04.13
[debug] Python version 2.7.9 - Linux-3.16.0-4-amd64-x86_64-with-debian-8.4
[debug] exe versions: avconv 2.6.8, avprobe 2.6.8, ffmpeg 2.6.8, ffprobe 2.6.8, rtmpdump 2.4
[debug] Proxy map: {}
[debug] Public IP address: 109.201.154.177
Finished downloading /home/user/pub/video/www/youtube/Dexbonus/20160414 - ♡ TY! ♡ - March 2016.mp4
Renaming to: /home/user/pub/video/www/youtube/Dexbonus/20160414 - TY - March 2016.mp4
Renaming to: /home/user/pub/video/www/youtube/Dexbonus/20160414 - TY - March 2016.annotations.xml
Renaming to: /home/user/pub/video/www/youtube/Dexbonus/20160414 - TY - March 2016.description
Renaming to: /home/user/pub/video/www/youtube/Dexbonus/20160414 - TY - March 2016.info.json
Renaming to: /home/user/pub/video/www/youtube/Dexbonus/20160414 - TY - March 2016.jpg
Finished downloading /home/user/pub/video/www/youtube/Dexbonus/20130330 - LONDON DAY 2 - Westminster, Buckingham Palace, and Friendship.mp4
Renaming to: /home/user/pub/video/www/youtube/Dexbonus/20130330 - LONDON DAY 2 - Westminster Buckingham Palace and Friendship.mp4
Renaming to: /home/user/pub/video/www/youtube/Dexbonus/20130330 - LONDON DAY 2 - Westminster Buckingham Palace and Friendship.annotations.xml
Renaming to: /home/user/pub/video/www/youtube/Dexbonus/20130330 - LONDON DAY 2 - Westminster Buckingham Palace and Friendship.description
Renaming to: /home/user/pub/video/www/youtube/Dexbonus/20130330 - LONDON DAY 2 - Westminster Buckingham Palace and Friendship.info.json
Renaming to: /home/user/pub/video/www/youtube/Dexbonus/20130330 - LONDON DAY 2 - Westminster Buckingham Palace and Friendship.jpg
Traceback (most recent call last):
  File "/home/user/dev/youtube-dl-automate/youtube-dl-automate/youtube-dl-automate.py", line 158, in <module>
    main()
  File "/home/user/dev/youtube-dl-automate/youtube-dl-automate/youtube-dl-automate.py", line 152, in main
    ydl.download([line])
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1730, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 682, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 734, in process_ie_result
    extra_info=extra_info)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 682, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 842, in process_ie_result
    extra_info=extra)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 734, in process_ie_result
    extra_info=extra_info)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 682, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 727, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1376, in process_video_result
    self.process_info(new_info)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1712, in process_info
    self.post_process(filename, info_dict)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1776, in post_process
    files_to_delete, info = pp.run(info)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 470, in run
    self.run_ffmpeg(filename, temp_filename, options)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 172, in run_ffmpeg
    self.run_ffmpeg_multiple_files([path], out_path, opts)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 146, in run_ffmpeg_multiple_files
    os.stat(encodeFilename(path)).st_mtime for path in input_paths)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 146, in <genexpr>
    os.stat(encodeFilename(path)).st_mtime for path in input_paths)
OSError: [Errno 2] No such file or directory: '/home/user/pub/video/www/youtube/Dexbonus/20130330 - LONDON DAY 2 - Westminster, Buckingham Palace, and Friendship.mp4'

...
<end of log>

Description of your issue, suggested solution and other information

When renaming a file at status=finished using a progress hook youtube-dl tries to post process the file using the old filename. Would it be possible to delay status=finished until post processing is also finished? Or to provide another status for the progress hook (status=postprocessed)?

Other information

Relevant program code

def rename_files(path):
    #TODO: os.rename() doesn't automatically overwrite on Windows
    if not os.path.isfile(path):
        print('Could not find file: ' + filename)
        sys.exit()
    # Extensions for extra files (which should also be renamed)
    extensions = [ '.annotations.xml', '.description', '.info.json', '.jpg' ]
    # We only want to change the filename
    filename = os.path.basename(path)
    # Don't need the extension until later
    newfilename = os.path.splitext(filename)[0]
    # Replace all '_\/' characters with spaces
    newfilename =  re.sub('[_\/]+',' ',newfilename)
    # Remove all characters not in 'A-Za-z0-9 -'
    newfilename =  re.sub('[^A-Za-z0-9 -]+','',newfilename)
    # Filename should not contain multiple spaces in a row
    newfilename = " ".join(newfilename.split())
    # Filename should not contain spaces at the beginning or end
    newfilename = newfilename.strip()
    # Filename should not contain trailing dashes
    newfilename = newfilename.rstrip('-')
    # Filename should not contain trailing spaces after removing trailing dashes
    newfilename = newfilename.rstrip()
    # Put back the extension
    newfilename = newfilename + os.path.splitext(filename)[1]
    # Append new filename to path
    newfilepath = os.path.dirname(path) + os.path.sep + newfilename
    os.rename(path, newfilepath)
    print('Renaming to: ' + newfilepath)
    # Rename extra files
    for extension in extensions:
        filename = os.path.splitext(path)[0] + extension
        newfilename = os.path.splitext(newfilepath)[0] + extension
        if os.path.isfile(filename):
            os.rename(filename, newfilename)
            print('Renaming to: ' + newfilename)
        else:
            print('Missing file: ' + filename)
            sys.exit()

def my_hook(d):
    if d['status'] == 'downloading':
        pass
    if d['status'] == 'finished':
        print('Finished downloading ' + d['filename'])
        rename_files(d['filename'])
    if d['status'] == 'error':
        print('Error during downloading ' + d['filename'])
        sys.exit()

def main():
    ....
    # See YoutubeDL.py for options
    ydl_opts = {
        'sleep_interval': 10,
        #'playlistreverse': 'true',
        'format': 'best',
        'age_limit': 30,
        'writedescription': 'true',
        'writeinfojson': 'true',
        'writeannotations': 'true',
        'writethumbnail': 'true',
        'download_archive': outputdir + os.path.sep + sitename +
            os.path.sep + archivefile,
        'outtmpl': outputdir + os.path.sep + sitename + os.path.sep +
            '%(uploader)s' + os.path.sep +
            '%(upload_date)s - %(title)s.%(ext)s',
        'logger': MyLogger(),
        'progress_hooks': [my_hook],
        'prefer_ffmpeg': 'true',
        'call_home': 'true',
        'verbose': 'true',
        #'simulate': 'true',
    }

    with youtube_dl.YoutubeDL(ydl_opts) as ydl:
        ydl.download([line])
    ...

[itkfilelor]

I know this is old, but since google brought me here as well, I'll answer for the future.

Unfortunately, one cannot rename in this fashion as ffmpeg is looking for d['filename'] to do it's conversions. The way I see it, there are 3 options to fix this issue. 1.) (I believe this only works for YouTube)Simulate the playlist with the same ydl_opts as if downloading, grab the desired video title and video id, pass the title through rename_files() then ydl.download(https://www.youtube.com/watch?v=<your_video_id> without the "<>". 2.)rename all of the files after the download call you can use something like: with youtube_dl.YoutubeDL(ydl_opts) as ydl: playlist_dict = ydl.extract_info(podcast['url'], download=False) for video in playlist_dict['entries']: title = video.get('title') 3.) skip pre defined post and import ffmpeg on your own and run that after the rename in the hook function.