ssc-oscar / gather

For harvesting latest repos
10 stars 9 forks source link

Connection to da getting closed + Repos to split up #10

Closed sylviesworld closed 3 years ago

sylviesworld commented 3 years ago

1) For some reason my connection to the da servers keep getting closed.

2) Could you give me a list of the repos that need to be broken up to run in parallel? (I'm guessing that some are small enough that they don't need to be broken up, while some do) Also, how many collections do you think I can run at a time?

audrism commented 3 years ago
  1. run screen on da servers to not worry about your connection
  2. all date/repo splits are specified in run2109.sh please fix the dates to current, however, e.g: python3 bbRepos.py 2020-05-03 bitbucket$DT 2021-05-01 &> bbRepos${DT}9.out &

only

sylviesworld commented 3 years ago

I'm not worried about the connection for the scripts. It's just annoying that anytime I try to do anything on the da servers that my connection gets closed. Is there anything that can be done to fix it?

audrism commented 3 years ago

Need more details: is that a result of some activity or inactivity and, also, how do you connect, what is from and what is to?

sylviesworld commented 3 years ago

Activity/Inactivity doesn't play a part. No matter what I'm doing, after a couple of minutes of being connected is kicks me off. I connect from my personal computer with the following in my config:

image

audrism commented 3 years ago

It appears this may be a home router or a windows issue (my connections never get dropped, I tried connecting as you and no dropping). The following

https://www.sweharris.org/post/2016-12-24-ssh-disconnect/

suggests to try ServerAliveInterval 600

sylviesworld commented 3 years ago

Do you think that using a VPN could cause the issue?

audrism commented 3 years ago

possibly, I'd still add ServerAliveInterval 600