search

codeship / scripts

Scripts for use on Codeship Basic
https://codeship.com
MIT License
217 stars 172 forks source link

Use -nc to check cache before downloading elasticsearch #274

Closed aandis closed 1 year ago

aandis commented 3 years ago

Fix tested.

rof@bionic_daafeb4e-dec9-40a4-b32f-1de67b76fed5_c9fd48beedd3:~$ wget -nc --output-document "${CACHED_DOWNLOAD}" https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.6/elasticsearch-2.4.6.tar.gz File ‘/home/rof/cache/elasticsearch-2.4.6.tar.gz’ already there; not retrieving.

rof@bionic_daafeb4e-dec9-40a4-b32f-1de67b76fed5_c9fd48beedd3:~$ wget -nc --continue --output-document "${CACHED_DOWNLOAD}" https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.6/elasticsearch-2.4.6.tar.gz File ‘/home/rof/cache/elasticsearch-2.4.6.tar.gz’ already there; not retrieving.

rof@bionic_daafeb4e-dec9-40a4-b32f-1de67b76fed5_c9fd48beedd3:~$ wget -nc --continue --output-document "${CACHED_DOWNLOAD}" https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.6/elasticsearch-2.4.6.tar.gz File ‘/home/rof/cache/elasticsearch-2.4.6.tar.gz’ already there; not retrieving.

denis-chmel commented 3 years ago

Hi @aandis, I've just created a PR https://github.com/codeship/scripts/pull/275 that workarounds the problem with 404 from elasticsearch, there will be a conflict with this one. But check that out, maybe this one is not needed even. Cause I guess this one helps to workaround the same issue, but theoretically helps only if file was successfully downloaded at least once before. My PR covers this too.

denis-chmel commented 3 years ago

The tiny potential issue with current PR I see that wget --continue will continue to download the file if it was disconnected in the middle (previous time). In that scenario the file will be present, but it is truncated (the size gonna be less than expected). It won't unzip in that case. But I don't know, maybe it doesn't happen anymore. However somebody did add --continue flag before, so this could be still the case, in theory.

joesiewert commented 1 year ago

Thanks, this has not continued to be an issue so going to close. We can revisit if it becomes an issue again.