Download stuck for long running jobs

[ani-ghosh]

Sometimes file downloads getting stuck in midway. Need ctrl+c to escape and move to next download. One example situation shown below.

Possible solutions and new feature request:

1. Add a time-out in the download function
2. Compare size of downloaded file and content-length from the httr::post request and attempt to re-download if the numbers don't match

3. Use another alternative to the progress bar showing download status

   |======================================================================| 100% |================== | 25%^C Warning messages: 1: In cmr_download(urls = fileurls, path = path, username = cred$user, : failure:https://e4ftl01.cr.usgs.gov//DP107/MOLA/MYD13Q1.006/2004.06.17/MYD13Q1.A2004169.h18v07.006.2015154155539.hdf 2: In cmr_download(urls = fileurls, path = path, username = cred$user, : failure:https://e4ftl01.cr.usgs.gov//DP107/MOLA/MYD13Q1.006/2009.10.08/MYD13Q1.A2009281.h18v07.006.2015195132705.hdf 3: In cmr_download(urls = fileurls, path = path, username = cred$user, : failure:https://e4ftl01.cr.usgs.gov//DP107/MOLA/MYD13Q1.006/2009.10.08/MYD13Q1.A2009281.h18v08.006.2015195135818.hdf 4: In cmr_download(urls = fileurls, path = path, username = cred$user, : failure:https://e4ftl01.cr.usgs.gov//DP107/MOLA/MYD13Q1.006/2009.10.08/MYD13Q1.A2009281.h17v07.006.2015195135725.hdf 5: In cmr_download(urls = fileurls, path = path, username = cred$user, : failure:https://e4ftl01.cr.usgs.gov//DP107/MOLA/MYD13Q1.006/2009.10.08/MYD13Q1.A2009281.h17v08.006.2015195135815.hdf |======================================================================| 100% |======================================================================| 100% |============ | 18%^C Warning message: In cmr_download(urls = fileurls, path = path, username = cred$user, : failure:https://e4ftl01.cr.usgs.gov//DP107/MOLA/MYD13Q1.006/2013.05.17/MYD13Q1.A2013137.h17v07.006.2015260174324.hdf

   |======================================================================| 100%

[wildintellect]

• We should look at timeout options for the download, which may depend on download backend (e.g. we might need to specifically force curl).
• We can get the expected file size from the header 'Content-Length' and compare to file.size(), there may also be a check method in the download backends.
• If the Data Source provides a hash, then we could compare hash values.

Note: Some backends actually support resume, so maybe on timeout we should attempt to resume the download of a particular file.