Fix freeze when opening repo on unsupported URL scheme.

real-yfprojects commented 1 year ago

It seems like not all file explorers respect the supported scheme property, that's why Vorta shouldn't rely on that. However currently Vorta will freeze when opening an unsupported scheme like smb or sftp. This should be fixed. A small error message should be displayed in the dialog instead.

You can open smb:// and other URLs when adding a repository although borg only supports normal paths (file scheme) and the ssh scheme. This is because vorta is missing to set the supported schemes attribute.

See https://doc.qt.io/qt-5/qfiledialog.html#supportedSchemes-prop

Originally posted by @ThomasWaldmann in https://github.com/borgbase/vorta/issues/1628#issuecomment-1459041597

i1sm3ky commented 1 year ago

@real-yfprojects @ThomasWaldmann Would dialog.setSupportedSchemes(["file", "ssh"]) work?

real-yfprojects commented 1 year ago

I think so.

ThomasWaldmann commented 1 year ago

Not sure whether the file dialog will be able to let you "select" a ssh: url somehow, but we can try.

What it for sure won't do, is the special ssh: path styles borg uses to indicate current directory and home directory on the server, usually ssh:// urls always give absolute paths only.

real-yfprojects commented 1 year ago

Not sure whether the file dialog will be able to let you "select" a ssh: url somehow, but we can try.

I was thinking that there might be a file selector somewhere out there that allows just that, so why not tell it that vorta supports ssh://.

What it for sure won't do, is the special ssh: path styles borg uses to indicate current directory and home directory on the server, usually ssh:// urls always give absolute paths only.

Are normal ssh: paths supported by borg?

ThomasWaldmann commented 1 year ago

ssh://user@host:port/some/abs/path is the usual and supported form.

ssh://user@host:port/./some/rel/path and ssh://user@host:port/~/some/home/path is supported by borg, but likely nowhere else.

i1sm3ky commented 1 year ago

So should we just allow the local path to be added using the dialog box and leave the ssh to be filled manually?

i1sm3ky commented 1 year ago

@real-yfprojects so I just add dialog.setSupportedSchemes(["file", "ssh"]) and merge it right?

ThomasWaldmann commented 1 year ago

@i1sm3ky i guess this needs trying interactively. try it before any change, check how the dialogue behaves and what it offers (esp. considering non-file: stuff), then change the schemes and try again how it behaves.

this is platform dependent, so try on all platforms you have access to.

i1sm3ky commented 1 year ago

@i1sm3ky i guess this needs trying interactively. try it before any change, check how the dialogue behaves and what it offers (esp. considering non-file: stuff), then change the schemes and try again how it behaves.

this is platform dependent, so try on all platforms you have access to.

Ok, I'll test that and work on it accordingly.

i1sm3ky commented 1 year ago

I tested it for local repos on MacOS Ventura 13.2.1 and it worked without any errors for both the cases(with and without the schemes). Unfortunately I can't figure out a way to test it for ssh and smb. Is there a way to test it for smb and ssh(without a paid host like borgbase) on the same machine?

Also while installing Borg, I faced some issues as I'm using a m1 chip and wasn't able to install the arm version. I had to install it using Rosetta 2 on x86_64 architecture. I think an update on https://vorta.borgbase.com/install/macos/ instructing on how to install it for m1 would be good.

ThomasWaldmann commented 1 year ago

There is nothing you need to "test" for smb except that:

before your modification you can select "smb:" stuff somehow
after your modification it will not let you choose "smb:" stuff any more

ThomasWaldmann commented 1 year ago

For ssh, you could run a sshd on localhost or some other (e.g. linux) machine you have access to.

borgbase has a for-free offer btw.

ThomasWaldmann commented 1 year ago

borg runs fine (and natively, without rosetta) on apple silicon, I have a MBA M1 myself.

for vorta, it is a bit more difficult, but it also can be compiled for apple silicon, see #727 about that.

real-yfprojects commented 1 year ago

You should be able to use a VM or a different machine to set up a test smb share and test ssh access.

i1sm3ky commented 1 year ago

before your modification you can select "smb:" stuff somehow after your modification it will not let you choose "smb:" stuff any more

Ya I kinda was asking for how can I do that as I didn't have another machine with me at that point of time. But I'll use one of my friends to test.

For ssh, you could run a sshd on localhost or some other (e.g. linux) machine you have access to. borgbase has a for-free offer btw.

Ok I'll look more into that. Thanks ;)

borg runs fine (and natively, without rosetta) on apple silicon, I have a MBA M1 myself.

Ok... Maybe there was something wrong on my end.

You should be able to use a VM or a different machine to set up a test smb share and test ssh access.

I tried that with a VM but was unable to do so, I'll try it once again.

ThomasWaldmann commented 1 year ago

I just tried on my MBA and I was not able to get a "smb:" URL result from the file dialogue (when adding a new repo):

either I click on the "world" icon, then it just gives me a ssh: prototype url to edit (no file dialogue)
or i click on the "folder" icon, then I can select misc simple paths from the filesystem (not: URLs) using the file dialogue
even when trying to select a SMB share via the file dialogue, the SMB server needs to be "connected" first and that will result in the share being mounted below `/Volumes/..." and it will also just result in a normal fs path (not an URL).
i was not able to select something from the SMB server without "connecting" (mounting) it first.

I could not discover any way to select a ssh: URL via the file dialogue.

So, as far as macOS and smb: / ssh: is concerned, I did not find the problem we are trying to "solve" here.

vorta 0.8.10 borg 1.2.3 macOS Ventura

real-yfprojects commented 1 year ago

I did not find the problem we are trying to "solve" here.

The guy in #1628 used Gnome and Fedora 37.

i1sm3ky commented 1 year ago

i was not able to select something from the SMB server without "connecting" (mounting) it first.

Yes same, I was able to use a mounted server but was not able to select a smb:// or a URL by itself on mac.

The guy in #1628 used Gnome and Fedora 37.

Yeah I'll try on those to see if I can reproduce the issue and does the fix work.

i1sm3ky commented 1 year ago

Sorry, I had my exams so I was unable to work on the issue.

I got one of my friend's pc to test.

While testing, I found some more issues-

When selecting ssh from the file dialog it also froze vorta
Manually typing the ssh or smb in the text field, both gave error: Unable to add your repository

Unfortunately I was not able to check with the fix enabled as I'm having some issues with linux. I'll try to resolve the issue and test the fix tomorrow.

I think it may also be a issue with Flatpak. It uses Sandboxing to increase security, On their site they have mentioned - by default, applications that are run with Flatpak have extremely limited access to the host environment. This includes: no access to the network.

And I also would love to know what would be the chances of getting GSOC under your mentorship. I am really looking forward to work more with you guys. What should be my next step and how should I apply to GSOC for your organisation? @real-yfprojects @m3nu @ThomasWaldmann Any help or recommendations would really be appreciated. :)

Edit: And when should I apply?

real-yfprojects commented 1 year ago

And I also would love to know what would be the chances of getting GSOC under your mentorship. I am really looking forward to work more with you guys. What should be my next step and how should I apply to GSOC for your organisation?

Chances are good. You can find information about our application process on our GSoC wiki page. You have already completed steps one to three.

When selecting ssh from the file dialog it also froze vorta

What file manager / dialog allows you to select ssh urls?

Manually typing the ssh or smb in the text field, both gave error: Unable to add your repository

A valid ssh address to a borg repository should work. The behaviour regarding smb is expected.

i1sm3ky commented 1 year ago

Chances are good. You can find information about our application process on our GSoC wiki page. You have already completed steps one to three.

Yeah Thanks :) I have read that earlier, I'll again have look into that and apply shortly. One more thing, if I send you the application before uploading to google, could any of you please check it once and tell me the corrections if any is need?

What file manager / dialog allows you to select ssh urls?

The default Fedora one.

A valid ssh address to a borg repository should work.

It didn't in my case.

real-yfprojects commented 1 year ago

One more thing, if I send you the application before uploading to google, could any of you please check it once and tell me the corrections if any is need?

Yes, that is the recommended procedure. This is also laid out on the linked page.

i1sm3ky commented 1 year ago

Yes, that is the recommended procedure. This is also laid out on the linked page.

Yup, Thanks again ;)

I think regarding the file dialog, We should only allow local files to be selected and ssh and other supported URLs could be added using the text box itself. So only allowing local files to be selected using the file dialog by adding dialog.setSupportedSchemes(["file", ]) should get us the expected functionality we are looking for. I'll just check it once and confirm in about half an hour or so.

i1sm3ky commented 1 year ago

Setting dialog.setSupportedSchemes(["file", ]) didn't helped either. I was still able to select ssh and smb URLs from the file dialog. I tried on Ubuntu as well, it took the ssh URL as sftp URLs and didn't prevent me from selecting a file over a network. And on Ubuntu, I didn't really found a way to select smb file from the file dialog by default, the file dialog didn't had the connect to a network tab on there.

Edit- Another solution would be to check for substrings containing unsupported URL types in the selected URL and pushing a QMessageBox informing the user about the supported URL types.

ThomasWaldmann commented 1 year ago

If the file dialogues are misbehaved and do not respect parameters given:

a bug should be filed at the respective projects
we could just not use the file dialogue - typing in a fs path or a ssh:// URL isn't that hard and preferable to a file dialogue crashing/hanging vorta just because the user clicked on something unexpected.

real-yfprojects commented 1 year ago

we could just not use the file dialogue - typing in a fs path or a ssh:// URL isn't that hard and preferable to a file dialogue crashing/hanging vorta just because the user clicked on something unexpected.

This would make for a very bad UX though. Vorta shouldn't crash/freeze instead.

i1sm3ky commented 1 year ago

So should I work on a prototype for the unsupported MessageBox?

real-yfprojects commented 1 year ago

So should I work on a prototype for the unsupported MessageBox?

I think we don't need a messagebox. We already have a label for showing errors in the add repo dialog.

i1sm3ky commented 1 year ago

I think we don't need a messagebox. We already have a label for showing errors in the add repo dialog.

So instead of a messagebox, we should just check the URL and put the warning if needed in the label?

ThomasWaldmann commented 1 year ago

BTW, not sure how repo validation is actually implemented, but maybe some code from borg could be copied for that purpose (maybe later, in a separate PR).

real-yfprojects commented 1 year ago

BTW, not sure how repo validation is actually implemented, but maybe some code from borg could be copied for that purpose (maybe later, in a separate PR).

Vorta does some general checks (e.g. whether the repo was added already). Then it runs borg info.

i1sm3ky commented 1 year ago

I was firstly thinking to use RegEx to check for the unsupported URLs, but now when I think of it, I think a simple if else consisting of a check for :// and file:// should work. Any suggestions on which path should I continue on? And if I might be missing out on anything using the later method?

i1sm3ky commented 1 year ago

I might start with the simpler solution first and see if it fulfills all the requirements which it might in theory.

i1sm3ky commented 1 year ago

Do any of you know if the file dialog with return a file:// type path for a local file under any circumstance?

i1sm3ky commented 1 year ago

I made the following changes to fix the issue of Vorta freezing. Added:

def check_url(self, URL):
        if ":" not in URL:
            return True
        else:
            URL = URL.split(":")
            protocols = ''
            for url in URL[:-1]:
                protocols += url.split("/")[-1] + ", "
                self._set_status(f"Selected filetype(s): ({protocols[:-2]}) cannot be opened through file dialog, Please try typing the URL in the text field manually.")

And changed if folder: to if folder and self.check_url(folder[0]): in repo_add_dialog.py.

Tested it in Ubuntu and Fedora and it works and prevents Vorta from freezeing Could anyone of you please ensure it's correct and ready to be merged? @real-yfprojects @ThomasWaldmann

ThomasWaldmann commented 1 year ago

Be careful, : is a valid char in UNIX filenames.

i1sm3ky commented 1 year ago

Be careful, : is a valid char in UNIX filenames.

Ya that might cause some problem. At first I was using :// to separate but linux takes ssh files as sftp:172.something.something, So I think I might separate using :// with an additional search for sftp:.

i1sm3ky commented 1 year ago

def check_url(self, URL):
        if "sftp:" not in URL:
            if "://" not in URL:
                return True
            else:
                URL = URL.split("://")
                protocols = ''
                for url in URL[:-1]:
                    protocols += url.split("/")[-1] + ", "
                    self._set_status(f"Selected filetype(s): ({protocols[:-2]}) cannot be opened through file dialog, Please try typing the URL in the text field manually.")
        else:
            self._set_status(f"Selected filetype: (sfpt:) cannot be opened through file dialog, Please try typing the URL in the text field manually.")

I think this won't have any issues with filenames containing :.

ThomasWaldmann commented 1 year ago

see the borg docs for the acceptable repo path/url style. note: the scp-style was removed for borg2.

i1sm3ky commented 1 year ago

see the borg docs for the acceptable repo path/url style. note: the scp-style was removed for borg2.

Thanks for that. I read the docs and my implementation would require some changes which I might do during the day, gotta sleep now.

i1sm3ky commented 1 year ago

import re
from urllib.parse import urlparse
from os.path import exists as local_path_exists

def check_url(self, URL):
    if ("sftp:host=" not in URL and "smb-share:server=" not in URL):
        if not re.match(r'^[-_a-zA-Z0-9]+@((\d{1,3}\.){3})\d{1,3}:', URL):
            parsed_url = urlparse(URL)
            if parsed_url.scheme in ('file', 'ssh', ''):
                return local_path_exists(parsed_url.path)
    self._set_status("Selected filetype cannot be opened through file dialog, Please try typing the URL in the text field manually.")

I tested it and it works in preventing any unsupported url type that any platform file dialog would return which borg and borg 2 won't support. @ThomasWaldmann Would this work?

ThomasWaldmann commented 1 year ago

Not sure what you are doing there, but it looks problematic.

i1sm3ky commented 1 year ago

Not sure what you are doing there, but it looks problematic.

What could cause problems?

if ("sftp:host=" not in URL and "smb-share:server=" not in URL):

This checks for the ssh and the smb connected files through the file dialog on linux.

if not re.match(r'^[-_a-zA-Z0-9]+@((\d{1,3}.){3})\d{1,3}:', URL):

This checks for the user@host: type urls.

parsed_url = urlparse(URL) if parsed_url.scheme in ('file', 'ssh', ''):

This checks for any other url types other than file:// or ssh://

return local_path_exists(parsed_url.path)

This checks if the path exists or not.

i1sm3ky commented 1 year ago

It currently only accepts local urls or file:// urls and ssh:// ones if any file dialog allows those. All other types are not supported by borg and hence aren't allowed to add via the file dialog.

i1sm3ky commented 1 year ago

Hey @real-yfprojects @m3nu @Hofer-Julian when should I start creating the application and the proposal?

m3nu commented 1 year ago

As per the official deadline. Should be submitted by April 4, I believe.

i1sm3ky commented 1 year ago

As per the official deadline. Should be submitted by April 4, I believe.

@m3nu Ya I know but I was asking this as my code hasn't been reviewed and merged for the Code Contribution part of the application yet so would that be a problem? And if yes then what should be my next step?

Edit: In the PSF checklist they have mentioned Your code doesn't have to be accepted and merged, but it does have to be visible to the public and it does have to be your own work so a link to the issues where the code have been discussed should work right?

m3nu commented 1 year ago

Yes, just link to the issue.

i1sm3ky commented 1 year ago

Yup sure, Thanks!

ThomasWaldmann commented 1 year ago

@i1sm3ky if you decide to write a own url/path validator, please do it as a boolean function and write unit tests for it.

but as i said, it might be simpler to either copy that from borg code (or invoke a borg command to let it do the testing).

if you write your own REs and conditions, it is rather likely that you miss(match) something.

borgbase / vorta

Fix freeze when opening repo on unsupported URL scheme. #1631