(s)ftp issues - Githubissues

babobski commented 8 years ago

Short Summary

Im having issues with opening remote directories. When I try to open a remote directory. Komodo get's stuck and doesnt react for a while.

In my error log I get the following errors:

.. does not have a local path (2) in chrome://komodo/content/library/uriparse.js:168
Traceback (most recent call last):
  File "chrome://komodo/content/library/uriparse.js", line 168, in

Naatan commented 8 years ago

Could you share the full error log? Also, could you share the path that you are trying to open?

babobski commented 8 years ago

Tryed to reproduce, but I don't have lot of remote projects at home. Will come back on it tomorrow.

th3coop commented 8 years ago

That error is coming from:https://github.com/Komodo/KomodoIDE/blob/master/src/chrome/komodo/content/library/uriparse.js#L164 and it seems very odd that that function is even being called.

Can you clarify how you're opening the remote directory? Is it mounted locally? Or are you opening a remote directory in Places?

babobski commented 8 years ago

I'm opening the file from the places widget, but I had the same errors in the error log in #1635 2016-06-29 20_48_48

babobski commented 8 years ago

@Naatan send you an Email with the complete error log

babobski commented 8 years ago

Found out what is causing this issue, and how to fix it (local). This issue occurs when you have a project set up with a remote dir over ftp, and the server setting is using sftp.

Switch my server settings to use sftp, but my project(s) settings was still using ftp. I had the same problem with my less compiler getting stuck on the browseForRemoteDir action. Setting my project to also use sftp fixed this issue.

babobski commented 8 years ago

😢 didn't fixed it.

babobski commented 8 years ago

Maybe it's Sftp releated, got some leading sftp errors before the display path erros begin

ERROR:xpcom:Unhandled exception calling 'int8 * newURI(in utf8string &, in string, in nsISomething, out retval nsISomething);'
Traceback (most recent call last):
  File "C:\Program Files (x86)\ActiveState Komodo IDE 10\lib\mozilla\python\xpcom\server\policy.py", line 300, in _CallMethod_
    return 0, func(*params)
  File "C:\Program Files (x86)\ActiveState Komodo IDE 10\lib\mozilla\components\koSFTP.py", line 47, in newURI
    self.defaultPort, aSpec, aOriginCharset, aBaseURI)
  File "<XPCOMObject method 'init'>", line 3, in init
Exception: 2152398858 (NS_NET_STATUS_WAITING_FOR)
[2016-07-04 15:56:30,967] [ERROR] console-logger: Error: 'sftp://location' does not have a local path (2) in chrome://komodo/content/library/uriparse.js:168
Traceback (most recent call last):
  File "chrome://komodo/content/library/uriparse.js", line 168, in

Naatan commented 8 years ago

Excerpt from @babobski's error log:

[2016-06-29 13:22:54,135] [ERROR] koFTPConnection: FTP CLOSE ERROR: 421 Idle timeout (600 seconds): closing control connection
Exception in thread Thread-4:
Traceback (most recent call last):
  File "C:\Program Files (x86)\ActiveState Komodo IDE 9\lib\python\lib\threading.py", line 808, in __bootstrap_inner
    self.run()
  File "C:\Program Files (x86)\ActiveState Komodo IDE 9\lib\mozilla\python\komodo\watchdog\observers\api.py", line 186, in run
    self.queue_events(self.timeout)
  File "C:\Program Files (x86)\ActiveState Komodo IDE 9\lib\mozilla\python\komodo\watchdog\observers\read_directory_changes.py", line 73, in queue_events
    self.watch.is_recursive)
  File "C:\Program Files (x86)\ActiveState Komodo IDE 9\lib\mozilla\python\komodo\watchdog\observers\winapi_common.py", line 124, in read_directory_changes
    None)
  File "C:\Program Files (x86)\ActiveState Komodo IDE 9\lib\mozilla\python\komodo\watchdog\observers\winapi.py", line 106, in _errcheck_bool
    raise ctypes.WinError()
WindowsError: [Error 5] Toegang geweigerd.

[2016-06-29 17:06:34,617] [WARNING] views: Inconsistent view count in view_closed event
[2016-06-29 17:06:34,793] [ERROR] codeintel.komodo.KoCodeIntelManager: Error reading data from codeintel
Traceback (most recent call last):
  File "C:\Program Files (x86)\ActiveState Komodo IDE 9\lib\mozilla\components\koCodeIntel.py", line 1167, in run
    ch = self.pipe.read(1)
  File "C:\Program Files (x86)\ActiveState Komodo IDE 9\lib\mozilla\python\komodo\win32_named_pipe.py", line 212, in read
    raise ctypes.WinError(ctypes.get_last_error())
WindowsError: [Error 109] De pipe is beëindigd.

Perhaps the issue is that it still calls the newURI stuff despite the connection having failed?

@babobski can you reproduce the issue in safe-mode?

babobski commented 8 years ago

@Naatan I'm currently trying to switch to sftp with my servers. But I'm still having issues with this. I switched a server to sftp and updated the project to switch to sftp. I wasn't able to connect over sftp so i switched it back to ftp.

I've updated the server to use ftp and updated the project to ftp. But when I saved the project settings I get a exception it can not connect trough sftp. My question is why it tries to connect trough sftp, if I updated the server and the project settings? Is there a project preference I'm missing?

Naatan commented 8 years ago

Could it be that your server does not support sftp?

babobski commented 8 years ago

@Naatan The problem lies in the project settings. When I have a server configured over ftp, create projects with this server. When I now change the server to sftp, my projects will still work, only when I then use the "open remote directory" in the places widget, komodo will crash.

When I change my project to sftp and try the "open remote directory" komodo will still crash. When I start komodo again my project settings is back to ftp. When i set my project settings to ftp and then reboot komodo, my project settings is back to ftp. If I then change the project settings to sftp, change it to a other project and then reboot open the project again, komodo has finally saved the settings and my project is sftp and if i now use the "open remote directory" in the places widget it will work again.

th3coop commented 8 years ago

Thanks for the effort in finding repro steps @babobski. I'm taking a look meow.

th3coop commented 8 years ago

When i set my project settings to ftp and then reboot komodo, my project settings is back to ftp.

Finally looking at this now. There is no Project level Sever settings. What exactly are you setting here?

th3coop commented 8 years ago

Ignoring the Project Settings stuff, this sounds like it might be related to the connection caching that Komodo does since restart Komodo seems to solve the issue. I'll see if I can figure out how to reset a specific connections cache when it's prefs are changed...i suspect this is going to be easier said than done.

Would it be reasonable to reset all cached connections when server settings are updating? Note I said "reasonable", not "optimal". @Naatan @babobski?

Naatan commented 8 years ago

I'm not sure what a connection cache entails, is that just that it holds the prefs (info) or is it actually keeping the connection alive?

th3coop commented 8 years ago

I'm not sure what a connection cache entails,

Me neither. If you have the Remote Files dialog open you will see a message that says cached at 00:00:00 or something like that. I don't actually KNOW what it's doing but I can guess. I think I found a more likely cause though.

I was investigating this yesterday and found that Komodo tries to load the last URL that was opened (makes sense). In some cases I have no idea where it's getting that value from (haven't dug deep enough into the Places Code). I'm assuming that variable is the issue. I'm guessing at some point it's populated by a value from your prefs file and could be stale if your server config changed.

Naatan commented 8 years ago

Sounds like a good lead. The caching angle would need more research imo, because it could affect both stability and performance.

th3coop commented 7 years ago

ping @babobski https://github.com/Komodo/KomodoEdit/issues/1758#issuecomment-245125483

babobski commented 7 years ago

I'm using a remote url for most of my projects:

2016-11-07 19_22_06

The remote directories are cached, when i use the relative includes addon I alswas see the cached at ..

th3coop commented 7 years ago

relative includes addon

I don't know what that is but the caching happens regardless of any addons.

Thanks for the update.

babobski commented 7 years ago

I'ts uses the ko.filepicker.remoteFileBrowser for generating a relative url. But the remote window is (almost) always showing the cached at, that was all.

np.

th3coop commented 7 years ago

Are you still able to reproduce this @babobski? I can't get Komodo to crash with your steps and I never have an issue with the Project Base Dir property not staying updated. If I update it to SFTP, it stays as FTP, even after a restart.

babobski commented 7 years ago

@cgchoffman will look in it tomorrow. Need tot test this on work, as I have the issues here. Can remember I had a crash a week ago, witch was using the wrong setting, what causes komodo to crash.

babobski commented 7 years ago

@cgchoffman Just tested it, but this is still an issue. Step i did today to reproduce:

Set up an connection trough sftp ( server )
Create a project with a remote url with the server you just created
Refresh you're project by double clicking on it
Open project settings and change the url from sftp to ftp and save the settings
Refresh you're project by double clicking on it
Use "Open remote directory" in the places widget

Komodo will now get stuck or hang for a long time.

And the errors that it triggers:

[2016-11-12 13:35:31,105] [ERROR] console-logger: Error: 'ftp://remote_server/httpdocs' does not have a local path (2) in chrome://komodo/content/library/uriparse.js:168
Traceback (most recent call last):
  File "chrome://komodo/content/library/uriparse.js", line 168, in 

[2016-11-12 13:35:31,121] [ERROR] console-logger: Error: 'ftp://remote_server/httpdocs' does not have a local path (2) in chrome://komodo/content/library/uriparse.js:168
Traceback (most recent call last):
  File "chrome://komodo/content/library/uriparse.js", line 168, in 

[2016-11-12 13:35:31,131] [ERROR] console-logger: Error: 'ftp://remote_server/httpdocs' does not have a local path (2) in chrome://komodo/content/library/uriparse.js:168
Traceback (most recent call last):
  File "chrome://komodo/content/library/uriparse.js", line 168, in 

[2016-11-12 13:35:31,454] [ERROR] console-logger: Error: 'ftp://remote_server/httpdocs' does not have a local path (2) in chrome://komodo/content/library/uriparse.js:168
Traceback (most recent call last):
  File "chrome://komodo/content/library/uriparse.js", line 168, in 

[2016-11-12 13:35:31,469] [ERROR] console-logger: Error: 'ftp://remote_server/httpdocs' does not have a local path (2) in chrome://komodo/content/library/uriparse.js:168
Traceback (most recent call last):
  File "chrome://komodo/content/library/uriparse.js", line 168, in 

[2016-11-12 13:35:31,480] [ERROR] console-logger: Error: 'ftp://remote_server/httpdocs' does not have a local path (2) in chrome://komodo/content/library/uriparse.js:168
Traceback (most recent call last):
  File "chrome://komodo/content/library/uriparse.js", line 168, in 

[2016-11-12 13:36:04,509] [ERROR] koFTPConnection: FTP do_openSocket ERROR: Connection timed out

th3coop commented 7 years ago

What is "Komodo will now get stuck or hang for a long time."? How long? Is it the timeout you've set in the Preferences > Servers? I thought this was a crash, ie. Komodo stops working all together and you must restart.

babobski commented 7 years ago

The default connection timeout is set to 30 seconds. The problem I am having ( at the moment less intense ) is that when I switched the settings of a test server from ftp to sftp, it felt like komodo was falling apart.

Even after I switched most project to sftp, when using the "open remote location", in the places widget. It sometimes still gets stuck while the current project settings where sftp.

Also in some addons I'm calling the browseForRemoteDir and the remoteFileBrowser functions, but these also suffer of a ""cached url". Getting komodo stuck for no real reason.

It try's to connect trough ftp while the server settings has changed to sftp. Also it fails, while previous it connected trough ftp with almost no problems, so it seems to me that the only part cached is the url and not the server settings. Otherwise it would also connect trough ftp with the old server settings. Looks like it's calling the old ftp url with the sftp settings causing komodo to get stuck until the connection fails.

30 seconds is a pretty long sit, doing nothing. So it might be I killed komodo most of the time, before the 30 seconds where over. So it's probably only getting stuck, but it feels like crashing when komodo blanks out.

On the worst days Komodo was getting stuck multiple times a day, that really felt like things where falling apart. I think it should not be that hard to switch a server to sftp.

th3coop commented 7 years ago

So far I can't reproduce anything. I followed your reproduction steps and nothing bad happens. I also note a possible issue with your reproduction steps. This isn't to say the issue doesn't exist. This means that the project has nothing to do with the issue: My steps:

I created two server setups to the same server. One SFTP and one FTP
The server only supports SFTP
Create a project and set Project Base Dir property to my SFTP connection using the Browse Remote button.
open/reload project by double clicking Here is where the problem arises I mentioned above. You did one of two things here:
- open prefs again, click Browse Remote to set the FTP connection ( but how?*Server doesn't support FTP so I can never open the connection. How did you set this if the FTP connection doesn't work?) OR
- You just deleted the s from the sftp in path string (this doesn't do anything. I'm not 100% sure but I think that portion of the string is just decoration. Note, that is not the server URL after it, it's the server preferences configuration name)

Either way, I tried both of the above and I never have Komodo hang when I select Open Remote Location.

Could you try this in more sterile conditions? (ie. restart komodo or your OS, reset profile, etc) I feel like there is something else going on here. I'm 99% the project is a false lead.

babobski commented 7 years ago

Thanks, will try if i can reproduce here at home.

babobski commented 7 years ago

Just a note, but should Komodo not only look at the server settings?

You just deleted the s from the sftp in path string (this doesn't do anything. I'm not 100% sure but I think that portion of the string is just decoration. Note, that is not the server URL after it, it's the server preferences configuration name)

if the path in my project is ftp and I open files from the places widget. The files are opened from a ftp location, if I change it to sftp the files are opened from a sftp location. Also it allows me to open the same file I have opened ( because for komodo it is not the same file anymore ).

It's a bug that if i have my project location settings on ftp and updated my server to use sftp, that it still tries to open my files over ftp. The project location setting is intended to point to a location on the server and should not control the connection protocol.

th3coop commented 7 years ago

Just a note, but should Komodo not only look at the server settings?

This is what it does. That is what I said in the text you quoted; I think that portion of the string is just decoration.

if the path in my project is ftp and I open files from the places widget. The files are opened from a ftp location

I'm seeing this too but as I said above, I'm pretty sure this is just decoration (ie. has no effect on the actual protocol used). I am 100% sure the file is not being opened over FTP because the server I'm connecting to does not support FTP connections so it would have refused. I think there is some magic path creation going on in the background.

The project location setting is intended to point to a location on the server and should not control the connection protocol.

Agreed and I don't think is. You said yourself that your server doesn't support FTP so how could Komodo be opening a file over FTP?

Having said that, that's a good find! That does seem like a likely culprit to be cause this BUT I can only reproduce the wrong protocol shown in open file but I cannot reproduce Komodo hanging when I try to open remote directory. We're missing something here.

Note, i've filed #2155

babobski commented 7 years ago

@cgchoffman I think we have to agree to disagree, I think it does something with the protocol part of the project settings.

You said yourself that your server doesn't support FTP so how could Komodo be opening a file over FTP?

I can't remember saying such a thing. I used to connect over FTP but recently switched to SFTP, is still can connect to the server over FTP, but also over SFTP.

The things that let me think it still is calling the FTP protocol, is that the "stuck" behavior only occurs when my project settings is still using FTP, but my server settings is SFTP. When I then look in my error log, is see errors like this:

[2016-11-12 13:35:31,105] [ERROR] console-logger: Error: 'ftp://remote_server/httpdocs' does not have a local path (2) in chrome://komodo/content/library/uriparse.js:168
Traceback (most recent call last):
  File "chrome://komodo/content/library/uriparse.js", line 168, in

In the error it is still parsing as a FTP protocol. When i hover over my file tabs in the tooltip it's using the FTP protocol.

When I switch my project settings to SFTP, the stuck behavior disappears, and all the urls are using the SFTP protocol. Komodo should not hang using FTP because i still can connect using FTP, the only difference is the port I set it to use. It looks like it fails because it using the SFTP settings on a FTP url.

I set the default connection timeout to 10 seconds, this has a positive effect. When I now have the issue, Komodo pops out of the "stuck" screen about 10 seconds.

Maybe something missing is that the FTP connection was using a active connection, can it be it still does something with this flag?

th3coop commented 7 years ago

@cgchoffman I think we have to agree to disagree, I think it does something with the protocol part of the project settings.

Agree to disagree on what? I've followed your steps and they do not reproduce the issue. I'm not disagreeing with an opinion, i'm saying your steps don't work.

The things that let me think it still is calling the FTP protocol, is that the "stuck" behavior only occurs when my project settings is still using FTP, but my server settings is SFTP.

I did this. It has no effect on me clicking "Open Remote Directory".

RE Parse error: I get that error too. I still do not get a hanging Komodo. That is unrelated as far as I can tell.

Please see the comment I made 2 days ago, https://github.com/Komodo/KomodoEdit/issues/1758#issuecomment-260468765. We're going in circles here and wasting a lot of time. I cannot reproduce the issue with the steps you've provide, it's as simple as that. You need to find new steps in STERILE conditions so I can reproduce the issue.

Naatan commented 7 years ago

@babobski I don't think you ever responded to this; are you able to reproduce these steps under safe-mode?

babobski commented 7 years ago

@Naatan tried to reproduce at home, but i don't have a remote here that supports both FTP and SFTP so I'm unable to reproduce.

Could you try this in more sterile conditions? (ie. restart komodo or your OS, reset profile, etc)

For on at work resetting my profile folder, is something I don't want to do. Restarting in safe mode is something I can do at work, so i will try it tomorrow.

The only thing bothering me with the reproduction steps of @cgchoffman is that the server only supports SFTP, in my case it supports FTP and SFTP. If it supports only SFTP you can not reproduce the issue as you can not refresh the places widget ( as you can not connect over FTP ).

Making the field not editable like the request in #2155 , will make it worse in my case. If I have a url with the wrong protocol in my projects and use browse remote button, I will get the stuck behavior.

As it is a editable field, my workaround is changing the protocol by hand so it will not get stuck.

th3coop commented 7 years ago

If it supports only SFTP you can not reproduce the issue as you can not refresh the places widget ( as you can not connect over FTP ).

See my comment from 3 days ago. I've already addressed this and need your response on how you actually edit the pref: https://github.com/Komodo/KomodoEdit/issues/1758#issuecomment-260468765 It does reload for me.

will make it worse in my case. If I have a url with the wrong protocol in my projects and use browse remote button

You would use the Browse Remote button to set the path. It would be impossible to have the wrong protocol then. You shouldn't be setting that by typing, it doesn't do anything when it comes to the protocol as far as I can tell.

As it is a editable field, my workaround is changing the protocol by hand so it will not get stuck.

I thought this is what was causing the issue, you manually editing that field. You never said this was a workaround did you?

Naatan commented 7 years ago

@babobski safe-mode doesnt require resetting anything. Please use Help > Troubleshooting > Restart in Safe-mode, then reproduce the issue. Please try this first as knowing whether it is dependant on something in your profile will give us a lot more to go on.

babobski commented 7 years ago

Please try this first as knowing whether it is dependant on something in your profile will give us a lot more to go on.

Ok, will do.

babobski commented 7 years ago

See my comment from 3 days ago. I've already addressed this and need your response on how you actually edit the pref: #1758 (comment) It does reload for me.

The problem occurred to me when I switched a test-server preferences > servers from FTP to SFTP. The projects I created with the project url pointing to this remote, still kept the FTP protocol. This is when the real problems occurred, and I'm getting issues with opening the "browse for remote dir" options throughout Komodo.

In the reproduction steps i provided, I did it the other-way around but gave me the same results. The problem started by just changing my server to SFTP and still have projects pointing at the server using FTP.

You would use the Browse Remote button to set the path. It would be impossible to have the wrong protocol then.

I know but as my project url was still pointing to FTP, using the browse remote button gave me the same stuck behavior. I pretty quick switched by editing the protocol from FTP to FTPS by hand. As this would bypass the stuck behavior, but also fixes the problem.

I did not get to test it on work yet, but got the same stuck behavior reproduced at home.

komodo-getting-stuck

This server does not support SFTP but it is the same stuck behavior, when I then look in my error log I get a SFTP connection error (reproduced by changing the url by hand to sftp and then use the browse remote button ):

[2016-11-19 05:26:30,903] [ERROR] koSFTPConnection: SFTP do_openSocket ERROR: Een verbindingspoging is mislukt omdat de verbonden party niet correct heeft geantwoord na een bepaalde tijd, of de gemaakte verbinding is mislukt omdat de verbonden host niet heeft geantwoord

Naatan commented 7 years ago

@cgchoffman couldn't this simply be that it's trying to access the directory MRU which still has an ftp:// reference?

th3coop commented 7 years ago

@babobski was this testing you did in your last comment, using a fresh profile?

@cgchoffman couldn't this simply be that it's trying to access the directory MRU which still has an ftp:// reference?

@Naatan possibly. That seems like a reasonable guess, but I can't reproduce this issue at all. I've tried all the methods of trying to trick Komodo into opening the wrong old server config and it just doesn't work. It always just opens the Open Remote Dialog without issue. The server config is not selected in the dialog though and I need to open it manually from the drop down. So Komodo isn't trying to load anything until I tell it to.

babobski commented 7 years ago

@cgchoffman it was not a clean profile, but using Edit. Just tested it with a clean profile, but at first when switching the project url to sftp made it hang but really short. It also did not reproduced the same error.

[2016-11-21 20:39:23,012] [WARNING] koSFTPConnection: do_authenticateWithPassword:: SSHException: Error reading SSH protocol banner
[2016-11-21 20:39:23,012] [ERROR] koSFTPConnection: SFTP do_authenticateWithPassword ERROR: ftp.bladiebla.nl:21 is not a recognized SSH server. Please recheck your SFTP server and port configuration.
[2016-11-21 20:39:23,128] [ERROR] console-logger: NS_ERROR_NOT_AVAILABLE: Cannot call openModalWindow on a hidden window (2) in file:///C:/Program%20Files%20(x86)/ActiveState%20Komodo%20IDE%2010/lib/mozilla/components/nsPrompter.js:356
Traceback (most recent call last):
  File "file:///C:/Program%20Files%20(x86)/ActiveState%20Komodo%20IDE%2010/lib/mozilla/components/nsPrompter.js", line 356, in

edited the server settings and omitted the server port, if I then open the project settings and browse remote with a sftp url I get the same stuck behavior and errors. Looks like the magic is in the server port.

I have to check at work how the server ports are configured, but I'm pretty sure I set it to 22 for the SFTP connection.

babobski commented 7 years ago

Just checked, but I have a server port configured for my SFTP server. Probability the behavior is different between FTP -> SFTP then FTP -> SFTP

Naatan commented 7 years ago

Please see if this makes a difference;

Make a backup of your prefs.xml (in your komodo profile folder)
Execute the following in your Komodo console

var prefs = require("ko/prefs");
prefs.deletePref("mruFileList");
prefs.deletePref("mruProjectList");

Then see if you can still reproduce the issue.

babobski commented 7 years ago

@Naatan just tested it, but same result using the above.

executed above command
opened a project using FTP
Changes the url to sftp by typing
Hit the browse remote button

Same result stuck for 30 seconds: 2016-11-23 20_10_24

And after hitting it for the second time for the screenshot. After the stuck behavior komodo didn't react to anything anymore, and needed to kill it using the task-manager.

Naatan commented 7 years ago

Just to clarify, when you change ftp to sftp, is the url in the input field 100% correct? (ie. including the port).

I don't believe Komodo is smart enough to automatically infer the port from an sftp url.

babobski commented 7 years ago

At work for the SFTP server I have the ports set to 22, but having this behavior when my old project still has FTP.

Naatan commented 7 years ago

Could you record a short screencast that shows your server settings as well as your project settings (specifically the base directory field). Please ensure we can see all settings properly and in their entirety (navigate through the input field), apart from of course the password.

You could send it so support@activestate.com to keep the server information private.

I'm very curious what the similarities are between your settings, as Komodo tries to auto detect what server config to use from the file URI.

babobski commented 7 years ago

@Naatan send you a mail.

I'm very curious what the similarities are between your settings, as Komodo tries to auto detect what server config to use from the file URI.

Does this only looks at the hostname? I'm always using the ip for the hostname. As most of the website I work on run on the same server they share the same ip/hostname, maybe this is causing the issue?

Naatan commented 7 years ago

Honestly it's not as easy to tell as I'd like. It'd be quicker for you to simply test by deleting one of the servers and seeing if that resolves the issue.

Komodo / KomodoEdit

(s)ftp issues #1758

Short Summary