search

fioan89 / sourcesync

A powerful plugin for performing one-way file synchronization for Intellij Idea and Pycharm projects
Other
89 stars 36 forks source link

Simultaneous connections #2

Closed BourneLoser closed 10 years ago

BourneLoser commented 10 years ago

Please add option to allow/disallow simultaneous connections to target server. When I manually select two or three files and click "Sync selected files to remote target" the transfer succeeds. However when I click "Sync changed files to remote target" (I have 16 changed files) I get a bunch of the following errors:

Sync warning: com.jcraft.jsch.JSchException: Session.connect: java.net.SocketException: Connection reset
Sync warning: com.jcraft.jsch.JSchException: connection is closed by foreign host
Sync error: com.jcraft.jsch.JSchException: session is down

This happens with both scp and sftp. I can see the simultaneous connection attempts in my target server's /var/log/secure file.

fioan89 commented 10 years ago

This sounds quite odd! The sync selected files runs multiple syncs at the same time just like the sync changed files. It sounds more like if there are more than "x" connections at the same time, the socket will fail. Can you try with different numbers of changed files to see what is the accepted number?

BourneLoser commented 10 years ago

Yes, I agree. I don't think it has to do with selected vs changed. So my target machine seems to allow around 7 simultaneous connections but on other machines that number could be anything - it all depends on the target machine and how busy it happens to be at the moment. What I had in mind when I made the request was that when a user unchecked "allow simultaneous connections" that all files are transferred in a loop through one connection. I guess that's not possible for scp but for the other protocols it should be possible.

fioan89 commented 10 years ago

That could be possible. In fact I'm thinking that if "Allow simultaneous connections" is checked to limit the upload to just two or three threads, otherwise the system could be stressed too much. Good thinking. I would do it. Probably by the end of the weekend it would be implemented.

fioan89 commented 10 years ago

This is fixed with the latest commits on the master branch.

fioan89 commented 10 years ago

I will close the issue. Feel free to open it again if problems arrives.

BourneLoser commented 10 years ago

I installed version 1.2 from the JetBrains repository. I am now able to successfully transfer all my changed files to my target machine. Thank you. I consider the issue closed too.

That said, I noticed that it takes longer than expected to transfer my 16 changed files with the "Allowed number of simultaneous sync jobs" set to 1. When I inspected my /var/log/secure file, I noticed that there were still 16 sessions being opened, they just came one after the other this time. Which is fine but doing it that way adds the overhead of session buildup/teardown for each file transferred. My original thought was to open 1 sftp session and transfer all the files through that one session. It should be noticeably faster that way. But anyways, after figuring out the behavior of the "Ok" button (see below), I changed the simultaneous sync jobs to 4, and retried. 16 sessions were still opened but it did bring the time down from 1m45s to 41s which is good enough for me.

As a side note on the buttons on the "sourcesync connection configuration" popup dialog, I'm used to "Apply" meaning: submit and keep window open. And I'm used to "Ok" meaning: submit and dismiss window. The way the "Ok" button behaves now is just to dismiss the window which is fine - I just think it should be labeled "Dismiss" instead. Otherwise some people might not understand what's going on and think something is broken.

fioan89 commented 10 years ago

Good point about OK/Apply buttons! I will implement that - it shouldn't be that hard. About the first part - it's quite a thing! In order to do that I need to redesign a few things around. So it might take a while! One other thing that was pointed out by a friend of mine is that currently - sensitive information like passwords and user-names are plain stored in map that later is serialized on the disk. So I have to prioritize this two bugs and see which one must be addressed faster. Since I don't have quite a lot of experience in the field of security I want to deal first with this problem ( I think it will take a little longer order to finish this)

fioan89 commented 10 years ago

Also please feel free to issue a bug based on your observation. And thank you for the feedback.

fioan89 commented 10 years ago

@BourneLoser Hi again! So I've tried to force SourceSync to reuse existing opened sessions. Can you please give this a try https://www.wetransfer.com/downloads/5aa6f4b9f52b95a5573bca3200a9d17920140412124300/ab29b2872a58745da60527dd510c4d3a20140412124300/ae5f9e before I will upload it to JetBrains website?

With 4 connections allowed, netstat -atn | grep ':22' gives me tcp 0 0 127.0.0.1:45020 127.0.0.1:22 TIME_WAIT
tcp 0 0 127.0.0.1:45022 127.0.0.1:22 TIME_WAIT
tcp 0 0 127.0.0.1:45023 127.0.0.1:22 TIME_WAIT
tcp 0 0 127.0.0.1:45021 127.0.0.1:22 TIME_WAIT

Also the OK/Apply issue should be fixed, plus a few additional visual glitches.

Many thanks, Ioan!

BourneLoser commented 10 years ago

Works good. I can see the connection is being reused. Great job thanks!