Intermittent SVN errors associated with TECA data

[taobrienlbl]

TravisCI builds are intermittently failing with the following error:

svn: E000060: Can't read from connection: Operation timed out

This is causing build failures on builds that would otherwise be successful. I noticed this, for example, the other day when navigating to the TECA home page on github, and the 'build' badge showed 'build:error' on master. On going into TravisCI's logs, I saw the above error, and restarting the build fixed the problem.

The current workaround is to simply re-run the build, but this is problematic for two reasons:

• It makes TECA's master branch to intermittently appear to have broken code, which could give potential new users the false impression that the code quality is poor.
• It increases the time between pushing new code and seeing whether the builds are successful or not, which makes the PR process take longer than it otherwise would. @elbashandy and I just encountered this in the review process for #274 .

(n.b., I see from searching the internet that this has been raised with TravisCI before, by @burlen : https://github.com/travis-ci/travis-ci/issues/7435 )

The intermittency of this is notable. A few ideas come to mind as to what could be the cause:

• Intermittent network connection failure on missmarple (faulty wiring, faulty network hub, ...?)
• Intermittent DNS resolution delays - perhaps LBL's DNS serve isn't totally reliable?
• Intermittent server overloading on missmarple - could there be cron job or some OS service that occasionally runs and bogs the system down to the point where it has delayed responses to network requests?
• Could there be an inherent behavior (flaw?) in SVN itself, or in the implementations on either the client or server or combination of the two, that makes it vulnerable to network variability?

[elbashandy]

Would using storage services like AWS S3 service be considered as a sustainable option?

[taobrienlbl]

Could be: good suggestion. Depending on the source of the problem, LBL ScienceIT offers container-based solutions (this wouldn't help if LBL's DNS resolution is the root of the problem though).

On Tue, Aug 6, 2019 at 9:20 AM Abdelrahman Elbashandy < notifications@github.com> wrote:

Would using storage services like AWS S3 service be considered as a sustainable option?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/LBL-EESA/TECA/issues/275?email_source=notifications&email_token=ACDDUFTJHR7HPPP62HHN773QDGQCVA5CNFSM4IJXJE72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3VVZRA#issuecomment-518741188, or mute the thread https://github.com/notifications/unsubscribe-auth/ACDDUFWLUIPTPPRUXAW746DQDGQCVANCNFSM4IJXJE7Q .

-- Travis A. O'Brien

http://eesa.lbl.gov/profiles/travis-a-obrien/ +1 (510) 495-8047

[burlen]

Yeah, this has been an on going issue for quite some time. we have some of the fastest network infrastructure in the world here at the lab, and if it were a problem with missmarple then linux builds would also fail in the same way. I am confident that the issue is on the travis side & I don't think there is anything we can do about it beyond reaching out to travis ci support.

however the work around is fairly easy - re-submit the build via travis web site. Usually things run smoothly the second time.

[burlen]

@taobrienlbl Sorry, I misspoke. In the conversation w/ travis ci you linked seems that I found a solution that was never merged into teca. In that link I mention that svn is also slow on my local apple system and that I thought that this slowness may be an svn + homebrew/mac issue. i'm not entirely sure why I didn't merge the work around. I will try it.

[burlen]

merged the fix mentioned above, if it helps we can close the issue. we should probably let it run for a while to be sure given the intermittent nature of the problem

[burlen]

haven't seen this is a while, I think we are all set.