Problems with automatically setting the video's mtime to the server's modification time

[epitron]

Youtube-DL recently changed its behaviour in a way that broke how I was using it. It used to set the mtime to when the file was downloaded; now it's set to when the file was uploaded to the video site.

This means that I can't sort my videos by download time anymore. It's problematic when you have a large download directory, or multiple collections, which you slowly process over time.

The only other downloader I've found with this default behaviour is wget. All the other downloaders -- browsers, torrent programs, curl, scp -- set the mtime to the time that you downloaded the file. (This is the the definition of mtime.)

I can see how people would want to save the video creation date. Luckily, there's a pull request in the pipeline (#1708) that writes this information -- and more -- to the file's xattrs.

So, you can have it all: files sorted by download date, and their creation date!

How 'bout it?

[jaimeMF]

Using the --no-mtime option disables this behaviour, does it work for you?

[epitron]

Yeah, it works.

It seems like it would be better if the default was --no-mtime, and that there was an --mtime option instead.

[SS64]

I think you can argue this either way, but it confused me at first having the download file's date/time appear to be wrong, in most cases the date always seems to be about 23 hours ago for some reason - would be nice if the --no-mtime option was a little more prominent in the docs/online help.

[epitron]

I think the strongest argument is that every other program (except wget) sets the mtime to the current time, and the ctime to the remote file's time.

Knowing the order you downloaded files in a directory helps you process them more easily.

[dsoprea]

+1 on all (changing the default, setting the ctime to this value instead of the ctime, setting the xattrs).

What's it going to take to move forward? This conversation has been stalled for a couple of years.

[yan12125]

+1 for making --no-mtime as the default. IMO youtube-dl is a video/audio downloader. All other actions besides downloading the contents of video/audio streams shouldn't be there unless explicitly requested.

setting the xattrs

-1 on this. xattrs shouldn't be added without --xattrs

[dsoprea]

In keeping consistent with your remarks, I understand where you are coming from.

I'm indifferent, though setting these doesn't operationally or visually interfere with anyone or anything (so we might as well).

Dustin

On Mon, Mar 6, 2017 at 12:53 PM, Yen Chi Hsuan notifications@github.com wrote:

+1 for making --no-mtime as the default. IMO youtube-dl is a video/audio downloader. All other actions besides downloading the contents of video/audio streams shouldn't be there unless explicitly requested.

setting the xattrs

-1 on this. xattrs shouldn't be added without --xattrs

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rg3/youtube-dl/issues/1709#issuecomment-284476310, or mute the thread https://github.com/notifications/unsubscribe-auth/AArrajIVUC2OLAjiR3RL6Z6UB0fg2ZLJks5rjEgqgaJpZM4BKgA2 .

[untotren]

+1

[allanlaal]

+1

I was expecting this behaviour and completely lost track of what I've downloaded because it was using random mtime dates for files

[NickSto]

+1 million

I think having the default date modified be the upload date violates the principle of least astonishment, since most tools set it to the current time. People will assume it operates like most other tools, as I did, and it can cause real problems.

I went years without realizing what it was actually doing. I really like to have a record of when I first found and downloaded a video, and now there's years of videos I've saved where I have no idea what period of my life they're from. The worst part is, it effectively poisoned the date modifieds on all the videos downloaded before that as well. If I see a video with a date from before that period, I don't know if it's actually from then, or if it's one downloaded with youtube-dl with a date that artificially put it back then.

It wasn't until I was trawling through the list of options and saw --no-mtime that I realized what happened. At the very least, I'd recommend putting a note somewhere prominent about what the default date modified will be.

[polonskiy]

--no-mtime definitely should be default. I have almost deleted new videos with find /youtube -type f -mtime +7 -delete.

[SS64]

The current behaviour would make sense if the server time related to the original video's creation date, but it's not. I'm not even sure that it is reliably related to the upload date?

[graywolf]

I'm not sure if you are counting votes or anything, but +1 for this. I've just out of habit tried ls -rtl to find the downloaded video from youtube and was bit confused it was nowhere to be find.

While --mtime could be possible handy, I think vast majority of people would expect --no-mtime to be the default, same way as other downloaders work.

But that's just, like, my opinion ^_^

[shivdhar]

+1 for --no-mtime to be the default, and --xattrs and, in fact, --add-metadata too.

These are flags that add information without breaking anything AFAIK, which is a good thing and definitely should be on by default.

--mtime by default is just plain confusing.

[yomajo]

Problem: music folder is arranged by last modified, so latest downloaded songs are at the top. Currently youtube-dl messes this up using last modified which is ~5days off.

Suggestion: use scrape timestamp instead of headers.

[dirkf]

What is a potential bug is that --no-mtime exists but there is no --mtime to override that if set as a default in a site or user config. Until this is fixed the advice that, if you like --no-mtime, you can make it the default, is not completely sound.

--- old/youtube_dl/options.py
+++ new/youtube_dl/options.py
@@ -734,8 +734,12 @@
         action='store_true', dest='nopart', default=False,
         help='Do not use .part files - write directly into output file')
     filesystem.add_option(
+        '--mtime',
+        action='store_true', dest='updatetime', default=True,
+        help='Use the Last-modified header to set the file modification time (default)')
+    filesystem.add_option(
         '--no-mtime',
-        action='store_false', dest='updatetime', default=True,
+        action='store_false', dest='updatetime',
         help='Do not use the Last-modified header to set the file modification time')
     filesystem.add_option(
         '--write-description',

Then:

$ python -m youtube_dl --help
...
  Filesystem Options:
...
    --mtime                              Use the Last-modified header to set
                                         the file modification time (default)
    --no-mtime                           Do not use the Last-modified header to
                                         set the file modification time
...
$ devscripts/cli_to_api.py --no-mtime --mtime
{}
$

[dirkf]

Finally, on platforms with POSIX touch, you could use --exec touch -m {} to set the file modification time to the current (ie, download) time, or in Windows cmd use some other hack with --exec ....