URLDownloader redownloads files even when they haven't changed

[jgstew]

URLDownloader will redownload the file if either the etag header changed or the last_modified header changed.

I would like the option to check any combination of ["etag", "last-modified", "content-length"] and skip downloads if any one or any 2 of them have not changed.

From what I can tell, URLDownloader redownloads if the etag alone changed. If you specify the check filesize only option, it still redownloads the file in order to do that check it seems.

Using the CHECK_FILESIZE_ONLY option will stop the process for continuing like there is a new download when there isn't, but it will still download the full file to check it seems.

The sending headers for etag / last modified are handled in URLGetter:

https://github.com/autopkg/autopkg/blob/5792fcc09d2c3862dd5e430329d4a9ac88b5d019/Code/autopkglib/URLGetter.py#L79

From here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match

"When used in combination with If-Modified-Since, If-None-Match has precedence (if the server supports it)."

Related:

• https://github.com/jgstew/jgstew-recipes/blob/main/BigFix/FixletDebugger.download.recipe.yaml
• https://github.com/autopkg/homebysix-recipes/blob/master/GoToMeeting/GoToMeeting.download.recipe
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-None-Match
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Modified-Since
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Match
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/If-Unmodified-Since
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag

[jgstew]

The following does work to NOT get the download again:

curl --head -H "If-Modified-Since: Wed, 20 Jan 2021 21:30:18 GMT" https://software.bigfix.com/download/bes/100/util/QNA10.0.2.52.zip

[jgstew]

Seems like there are 2 options:

• Conditionally configure URLGetter to NOT send the If-None-Match: _etag_ header
  • this seems best
  • Seems like I can accomplish this by clearing the etag variable from URLDownloader
  • best way to do this without modifying URLDownloader would be to clear the etag from the file xattr
  • see the xattr: xattr -l -v /Users/USER/Library/AutoPkg/Cache/RECIPE/downloads/FILENAME
• Get only the headers, then implement logic in Python to do the header comparisons
  • This option appears to be required for GoToMeeting and any other servers that don't support If-Modified-Since, If-None-Match headers.

[jgstew]

I resolved this for the following: https://github.com/jgstew/jgstew-recipes/blob/main/BigFix/FixletDebugger.download.recipe.yaml

By clearing the saved eTag since it is not valid to stop it from being sent in subsequent curl commands:

https://github.com/jgstew/jgstew-recipes/blob/main/SharedProcessors/ClearFileXattr.py

This does NOT resolve the issue for GoToMeeting since their CDN does not support etags or last modified correctly. The only way to resolve this for that case is to fetch the headers only, then check if the download has changed in python, then download the actual file if and only if it changed.