Symlinks are not supported

[NightMachinery]

I am running a server on a local dir:

sftpgo portable -d ~/Base/shared --permissions 'list,download' --username 'x' --password 'x' --webdav-port 8114 --sftpd-port 8115 --ftpd-port 8116 --log-verbose --log-file-path sftpgo.log

I symlink other directories I want to share into ~/Base/shared, but sftpgo does not handle the symlinks correctly. (I know sftpgo supports virtual dirs, but I use that ~/Base/shared for other services, too, and I do not want to duplicate my config.)

PS: That command also does not start the webdav server, saying

{"level":"debug","time":"2021-03-05T10:49:41.514","sender":"service","message":"WebDAV server not started, disabled in config file"}

[drakkan]

Hi,

webdav support for portable mode is now fixed.

Escaping home dir via symlinks is not allowed, you have to use the virtual folders for this

[NightMachinery]

How do I install the master?

go get -u -v github.com/drakkan/sftpgo
go: downloading golang.org/x/sys v0.0.0-20210305034016-7844c3c200c3

golang.org/x/sys/internal/unsafeheader
golang.org/x/sys/cpu
golang.org/x/crypto/blake2b
golang.org/x/sys/unix
golang.org/x/crypto/argon2
github.com/alexedwards/argon2id
github.com/prometheus/procfs
github.com/fsnotify/fsnotify
golang.org/x/net/internal/socket
github.com/spf13/viper
golang.org/x/net/ipv6
golang.org/x/net/ipv4
github.com/miekg/dns
github.com/prometheus/client_golang/prometheus
github.com/drakkan/sftpgo/vfs
# github.com/drakkan/sftpgo/vfs
../../go/pkg/mod/github.com/drakkan/sftpgo@v1.2.2/vfs/azblobfs.go:199:52: not enough arguments in call to blobBlockURL.BlobURL.Download
    have (context.Context, int64, number, azblob.BlobAccessConditions, bool)
    want (context.Context, int64, int64, azblob.BlobAccessConditions, bool, azblob.ClientProvidedKeyOptions)
../../go/pkg/mod/github.com/drakkan/sftpgo@v1.2.2/vfs/azblobfs.go:307:50: not enough arguments in call to dstBlobURL.GetProperties
    have (context.Context, azblob.BlobAccessConditions)
    want (context.Context, azblob.BlobAccessConditions, azblob.ClientProvidedKeyOptions)
../../go/pkg/mod/github.com/drakkan/sftpgo@v1.2.2/vfs/azblobfs.go:686:45: not enough arguments in call to blobBlockURL.BlobURL.GetProperties
    have (context.Context, azblob.BlobAccessConditions)
    want (context.Context, azblob.BlobAccessConditions, azblob.ClientProvidedKeyOptions)
../../go/pkg/mod/github.com/drakkan/sftpgo@v1.2.2/vfs/azblobfs.go:823:37: not enough arguments in call to blockBlobURL.StageBlock
    have (context.Context, string, *bytes.Reader, azblob.LeaseAccessConditions, nil)
    want (context.Context, string, io.ReadSeeker, azblob.LeaseAccessConditions, []byte, azblob.ClientProvidedKeyOptions)
../../go/pkg/mod/github.com/drakkan/sftpgo@v1.2.2/vfs/azblobfs.go:844:40: not enough arguments in call to blockBlobURL.CommitBlockList
    have (context.Context, []string, azblob.BlobHTTPHeaders, azblob.Metadata, azblob.BlobAccessConditions, azblob.AccessTierType, nil)
    want (context.Context, []string, azblob.BlobHTTPHeaders, azblob.Metadata, azblob.BlobAccessConditions, azblob.AccessTierType, azblob.BlobTagsMap, azblob.ClientProvidedKeyOptions)
github.com/grandcat/zeroconf

[NightMachinery]

@drakkan How do I add virtual folders to the portable mode?

[drakkan]

Virtual folders are not supported in portable mode. Since I'm working to add virtual folders to non local fs too this will add a lot of new configuration options. You could use a dump file and the memory provider directly. This is how portable mode works under the hood.

Installation via go get will fail since we need to use some forks (for example for crypto/ssh etc.) to support features not yet merged upstream. To install git main you can do something like this:

git clone https://github.com/drakkan/sftpgo.git
cd sftpgo
go build -ldflags "-s -w -X github.com/drakkan/sftpgo/version.commit=`git describe --always --dirty` -X github.com/drakkan/sftpgo/version.date=`date -u +%FT%TZ`" -o sftpgo

[NightMachinery]

@drakkan Can’t you just add an option to follow symlinks without reservations in portable mode? “-force-unsafe-symlinks” or something. This much config for something this simple that is only accessible on my local LAN seems a bit excessive ...

[drakkan]

For your use case I think the simplest thing to do is to enable webdav and ftp inside sftpgo.json, for example:

diff --git a/sftpgo.json b/sftpgo.json
index 0d041c5..5cadff8 100644
--- a/sftpgo.json
+++ b/sftpgo.json
@@ -28,7 +28,7 @@
   "sftpd": {
     "bindings": [
       {
-        "port": 2022,
+        "port": 8115,
         "address": "",
         "apply_proxy_config": true
       }
@@ -54,7 +54,7 @@
   "ftpd": {
     "bindings": [
       {
-        "port": 0,
+        "port": 8116,
         "address": "",
         "apply_proxy_config": true,
         "tls_mode": 0,
@@ -82,7 +82,7 @@
   "webdavd": {
     "bindings": [
       {
-        "port": 0,
+        "port": 8114,
         "address": "",
         "enable_https": false,
         "client_auth_type": 0,

start sftpgo in standard mode, for example:

./sftpgo serve -l ""

point your browser to the web admin login using admin/password and create a user.

From now on you can simply start sftpgo using ./sftpgo serve -l "" and use the defined user via SFTP/FTP/WebDAV.

I'll consider your request but I have other priorities for now, it could take some time.

[NightMachinery]

@drakkan stfpgo does not run if I symlink its binary to another location:

{"level":"error","time":"2021-03-08T13:15:28.652","sender":"utils","message":"error loading required template: open templates/base.html: no such file or directory"}
panic: open templates/base.html: no such file or directory

Update: It seems the templates directory needs to be copied to config-dir. I think this is a bug and sftpgo should create these default files if not found.

[drakkan]

Hi,

you need to set the config dir to the directory containing the configuration file (sftpgo.json or sftogo.yaml etc..). For example:

./sftpgo serve -l "" -c /home/nicola/goprojects/sftpgo

The config dir is the current dir by default. On Linux the config file is searched in $HOME/.config/sftpgo and /etc/sftpgo too.

The config directory will also be used as base dir so if it contains templates and static folders this is enough. If you want to customize the things you can set different paths inside sftpgo.json. You can also set templates_path and static_files_path to empty strings. In this case sftpgo will start with the web admin disabled (but rest api will be available).

You can also explictly set the config file using --config-file

[drakkan]

@drakkan stfpgo does not run if I symlink its binary to another location:

{"level":"error","time":"2021-03-08T13:15:28.652","sender":"utils","message":"error loading required template: open templates/base.html: no such file or directory"}
panic: open templates/base.html: no such file or directory

Update: It seems the templates directory needs to be copied to config-dir. I think this is a bug and sftpgo should create these default files if not found.

Are you manually installing sftpgo? If you use a distro package all this things will be done for you, if you want to install sftpgo manually you have to handle them yourself. For manual installation this doc could help

[NightMachinery]

@drakkan I am now trying to create a user using the web admin, but it gives me this error:

Validation error: folder name "/Volumes/hyper-diva/video" is not valid, the following characters are allowed: a-zA-Z0-9-_.~

[drakkan]

@drakkan I am now trying to create a user using the web admin, but it gives me this error:

Validation error: folder name "/Volumes/hyper-diva/video" is not valid, the following characters are allowed: a-zA-Z0-9-_.~

Please create the folder first and then reference it by name, thank you

[NightMachinery]

@drakkan I am now trying to create a user using the web admin, but it gives me this error:

Validation error: folder name "/Volumes/hyper-diva/video" is not valid, the following characters are allowed: a-zA-Z0-9-_.~

Please create the folder first and then reference it by name, thank you

Thanks, I have finally succeeded! ^_^

[drakkan]

Thank you for confirming. If you think some existing documentation could be improved or you want to write an howto style guide that could help others a pull request is very appreciated

[NightMachinery]

@drakkan I did create a guide for my own later reference, but it's really too concise for general use ...

manual install

see https://github.com/drakkan/sftpgo/blob/main/docs/service.md

git clone <https://github.com/drakkan/sftpgo.git> cd sftpgo go build
-ldflags \"-s -w -X github.com/drakkan/sftpgo/version.commit=\`git
describe --always --dirty\` -X
github.com/drakkan/sftpgo/version.date=\`date -u +%FT%TZ\`\" -o sftpgo

setup the config and launch the server

1. the default config file (which lives at the root of the git repo of sftpgo) only needs the ports and addresses of the different services

   ## setup config-dir:
   cp -r static templates ~/Base/keys/sftpgo
   ## launch
   sftpgo serve --config-file "$NIGHTDIR/launchers/sftpgo_darwin.json" --config-dir ~/Base/keys/sftpgo --log-file-path sftpgo.log

open the web admin http://localhost:8080/web/users, and login using admin:password

create the needed virtual folders in http://localhost:8080/web/folders

create a user and give it its root (home) dir and some of the virtual dirs set before

/dl::dl::-1::-1
/sg4::sg4::-1::-1
/hyperdiva::hyperdiva::-1::-1

1. perms

   /sg4/Pocket/forbidden::
   /writable::*

   1. default perms should be only list, download

2. set Allowed IP/Mask to 192.168.1.0/16

[drakkan]

Thank you! I'll redirect here if other users have similar issues.

P.S. you can download prebuilt binaries for both stable releases or for each commit, for example the binaries for the latest commit are available here

[ajl000]

Supporting symlinks would be very useful and would avoid strange errors in the directory listing that could be accidentally deleted
+1

[drakkan]

@ajl000 can you please elaborate? You should be not able to remove a file pointing outside the user's home directory.

Symlinks are supported if they point within the user home directory. Allowing to escape the home dir via a symlink is a significant security issues. Other similar software behave like SFTPGo and they recommend bind mounts (unsupported on macOS). For example take a look here. The only similar software that allows to escape home dir via symlinks is Pure-FTPD if you compile it using the --with-virtualchroot. Please take a look at "shared directories and chroot" section here. SFTPGo supports this feature using the virtual folders.

Following symlinks outside home dir can be accomplished quite easily in SFTPGo, I have a local patch for this

git diff --stat
 common/actions_test.go           |  2 +-
 common/common_test.go            |  2 +-
 common/connection_test.go        | 12 ++++++------
 common/transfer_test.go          | 10 +++++-----
 dataprovider/user.go             |  2 +-
 ftpd/internal_test.go            |  6 +++---
 httpd/httpd_test.go              |  4 ++--
 httpd/schema/openapi.yaml        | 14 ++++++++++++++
 httpd/webadmin.go                |  3 +++
 sftpd/internal_test.go           | 20 ++++++++++----------
 sftpd/sftpd_test.go              |  8 ++++----
 templates/webadmin/fsconfig.html | 16 ++++++++++++++++
 vfs/azblobfs.go                  |  2 +-
 vfs/cryptfs.go                   |  9 +++++----
 vfs/filesystem.go                | 14 ++++++++++++--
 vfs/folder.go                    |  2 +-
 vfs/gcsfs.go                     |  2 +-
 vfs/osfs.go                      | 17 +++++++++++------
 vfs/s3fs.go                      |  2 +-
 vfs/sftpfs.go                    |  2 +-
 vfs/vfs.go                       | 18 ++++++++++++++++++
 webdavd/internal_test.go         | 20 ++++++++++----------
 webdavd/webdavd_test.go          |  4 ++--
 23 files changed, 129 insertions(+), 62 deletions(-)

I'm not sure if I'll ever push this patch: it adds a new per-user setting to skip paths validation and so follow symlinks outside the home dir. If I push something like this I'm sure other users will ask a way to globally disable this behaviour (because it is a security issue and they don't want inadvertently enable it), so basically I'll add a new setting and another one to disable it ...

I'll think a bit more about this request but I'm really tempted to close it as won't fix

[drakkan]

Here is the WIP patch, I'll not merge it for now

0001-WIP-add-a-per-user-option-to-follow-symlinks.zip

[ajl000]

thanks Nicola, I understand the security concerns. I still however think it would be good outside the home directory in various circumstances to provide a seamless experience for the user.

This said it is not a feature I specifically need, given that there is a user authentication 'end point' on login, and I can add one or two virtual folders then. (It is not as seamless as would be symlinks outside the home directory, but should do)

The only other feature that I was thinking of was user impersonation, able to be set via the API. Its not really a feature for most users, unless they have logging at the filesystem level as well or possible ACL at that level too.

I will subscribe annually should I ever use the software (which I may well do).

[drakkan]

Please take a look at the last comment here.

I'll close this issue for now, the patch here can be used as starting point if I change my mind in future

[NightMachinery]

@drakkan The virtual folders can't overlap:

/my-music::my-music::-1::-1
/my-music/Songs::Songs::-1::-1

Validation error: invalid virtual folder "/my-music/Songs" overlaps with virtual folder "/my-music"

So ... what do I do? The only way to fix this seems to be to create virtual folders for every single directory in my-music/, and even then I would lose files in the my-music/ directory itself.

I can't mount Songs to /Songs, as I have playlists that suppose the my-music/Songs exists.

Update: I hacked my way around this limit, but it is crude and fragile, and I will much appreciate it if overlapping virtual folders or symbolic links were supported.

[drakkan]

It seems to work for me

can you please share the filesystem paths for your virtual folders?

[NightMachinery]

@drakkan

---

As I already had a Songs symlink in my-music,

~/my-music
❯ l       
.DS_Store
.fzfhist
cache
d -> /Users/evar/Downloads
LilanMusic -> /Volumes/LILAN/Music
ls
mmisc
playlists
Songs -> /Volumes/hyper-diva/Songs
V_hyp -> /Volumes/hyper-diva/video/V

I tried using a different name for the virtual folder, but that didn't help:

---

I am on version 2.0.2, should I update?

[drakkan]

In 2.0.x branch we had this, in 2.1.x I have changed the code this way, so updating could help

[NightMachinery]

@drakkan Ok, I did that, but now I get:

> sftpgo initprovider
2021-09-28T17:48:57.000 INF Initializing provider: "sqlite" config file: ""
2021-09-28T17:48:57.000 ERR database version 9 is too old, please see the upgrading docs
2021-09-28T17:48:57.000 WRN Unable to initialize/update the data provider: database version 9 is too old, please see the upgrading docs

While the docs say it is possible to upgrade from 2.0 to 2.1? So what am I missing?

[drakkan]

Please post the output from ./sftpgo --version, you should have this error if you are using the development version (the future v2.2), v2.1.x should handle database versions from 8 to 10

[NightMachinery]

Indeed, I installed from master. I restored my json backup, so I'm good.

[NightMachinery]

The virtual folders indeed can overlap in the latest version. Thanks!

[drakkan]

Good, thanks for sharing