whitesource / unified-agent-distribution

51 stars 48 forks source link

conditional-jar-downloads: avoid downloading files, if their remote content hasn't changed since the last download #37

Open svendr opened 2 years ago

svendr commented 2 years ago

This PR tries to avoid constant re-downloading of the files wss-unified-agent.jar and wss-unified-agent.config, if the local last-modified-date is equal or newer than the last-modified-date of the remote file. This concept is known as Conditional GET Request and is commonly used to save download times and reduce unnecessary network usage. curl's -z option takes care of the heavy work here, see man curl.

r0bb3n commented 1 year ago

I would recommend to make use of ETag instead, see Check by modification of content.

Using ETags, curl can check for remote updates without having to rely on times or file dates. It also then makes the check able to detect sub-second changes, which the timestamp based checks cannot.

curl --etag-compare etag.txt --etag-save etag.txt \
  https://example.com/file -o output

UPDATE FYI: There seems to be a bug in curl prior to v7.70.0, that prevents using etags in this constellation, see this