CulverLab / sparcd

https://wildcatresearch.arizona.edu/
GNU General Public License v3.0
4 stars 7 forks source link

Upload failure detections and retries #40

Closed Chris-Schnaufer closed 3 years ago

Chris-Schnaufer commented 4 years ago

Added a TransferStatusCallbackListener derived class to detect when Jargon fails on uploads Reduced the count of files per tar to a maximum of 50 (allows faster retries w/ less lost time when errors occur) Added a checksum check between the local file and the uploaded file w/ automatic retry (per file) if the checksum fails Added additional exception detection for catching certain upload issues with upload retry (on a per file basis) Added a retry loop for files that failed to upload Added a backoff algorithm that waits before retrying uploads after a failure (either checksum or other)

dlebauer commented 3 years ago

fixes https://github.com/CulverLab/sparcd/issues/27

smalusa commented 3 years ago

Thank you for taking the time to meet today. Sue

Sent from my iPhone

On Jan 6, 2021, at 1:44 PM, David LeBauer notifications@github.com wrote:



External Email

fixes #27https://github.com/CulverLab/sparcd/issues/27

— You are receiving this because your review was requested. Reply to this email directly, view it on GitHubhttps://github.com/CulverLab/sparcd/pull/40#issuecomment-755665877, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFABTXYNUB2UFMAWD4D6CWTSYTDTBANCNFSM4T34UT5Q.

julianpistorius commented 3 years ago

@Chris-Schnaufer Is this PR ready to merge? If so I might merge https://github.com/CulverLab/sparcd/pull/41 first, then rebase this branch over CulverLab:Sanimal-No-ES and try to build again with Java 15.

Chris-Schnaufer commented 3 years ago

@Chris-Schnaufer Is this PR ready to merge? If so I might merge #41 first, then rebase this branch over CulverLab:Sanimal-No-ES and try to build again with Java 15.

PR 41 should be building off this one. I don't have a strong opinion on which to merge first - I think merging 41 before this one would result in more testing however.