tormodwill commented 3 years ago

Hello

I have compiled sshfs for osxfuse ( now macFUSE ) I am now running SSHFS version 2.9 on macOS Mojave 10.14.6

Last login: Mon Jan 25 09:37:58 on console Tormods-MacBook-Pro:~ tormod$ sshfs --version SSHFS version 2.9 FUSE library version: 2.9.9 fuse: no mount point Tormods-MacBook-Pro:~ tormod$

Why is it that version 2.5.0 is listed as the latest release ? The SSHFS version 2.9 is working fine.

I did the following changes to the sshfs.c

force ssh-fs>sync_read = 0 in line 1827 ( https://github.com/osxfuse/sshfs/issues/57 )

changed # include to #include ( https://github.com/osxfuse/osxfuse/issues/751 )

Would it be possible to have 2.9 as the latest release ?

The folder on https://github.com/osxfuse Should be changed to https://github.com/macfuse

I also find that the text on the page https://github.com/osxfuse/sshfs is misleading. According to the page : "The latest version and more information can be found on http://github.com/libfuse/sshfs" Well, this is not correct if you use a mac. And if you don´t use a mac you would not be on the page in the first place. The latest version for mac is 2.9, and the latest version can be found on the page : https://github.com/osxfuse/sshfs

The latest version on the https://github.com/libfuse/sshfs will not work for mac. ( There is no FUSE 3 for OS X, FUSE library version: 2.9.9 for macFUSE macFUSE 4.0.4 )

Best regards

Tormod Willassen

Metacomet10 commented 3 years ago

Hi,

I was wondering what the proper procedure to compile and install v2.9 (as you have done) would be. I know how to use Terminal, but I'm very lost on how to take the files here and make them into a package that I can install. Any advice or pointers would be immensely helpful.

Thanks, Metacomet10

tormodwill commented 3 years ago

How to compile and install SSHFS version 2.10.0 on macOS

The various steps described below will demand some knowledge in use of terminal.

For this tutorial I use the following versions of software: Xcode_10.1 - for High Sierra 10.13.6 Command_Line_Tools_for_Xcode_10.1 - for High Sierra 10.13.6

Xcode_11.3.1 - for Mojave 10.14.6 Command_Line_Tools_for_Xcode_11.3.1 - For Mojave 10.14.6

If you run a newer version of macOS, you will find the correct version of Xcode here : https://xcodereleases.com/

macFUSE - use latest release. Download from here : https://github.com/osxfuse/osxfuse/releases Use .dmg file ( already compiled )

glib-2.66.4 Download glib from here : https://ftp.acc.umu.se/pub/GNOME/sources/glib/2.66/

SSHFS version 2.10.0 Download SSHFS from here : https://github.com/libfuse/sshfs/releases/tag/sshfs-2.10 As of june 2021 this is the latest version that will run on mac. The latest version on the https://github.com/libfuse/sshfs will not work for mac. The latest version of libfuse shipping with macFUSE is 2.9.9. This means you are limited to SSHFS 2.x. ( There is no FUSE 3 for OS X, FUSE library version: 2.9.9 for macFUSE macFUSE 4.x.x )

Pyton 3.9.1 Download python from here : https://www.python.org/downloads/release/python-391/

pkg-config 0.29.2 Download https://pkgconfig.freedesktop.org/releases/?C=M;O=D Description : https://opensource.ncsa.illinois.edu/confluence/display/DESDM/Installing+pkg-config+from+source+for+OSX

Start the install / process

To get this working, you need glib. in order to get glib working, you need meson and ninja. In order to start the process, you can have a look here:

https://mesonbuild.com/SimpleStart.html

Scroll down to the macOS part.

Install Xcode.
" Installing XCode is not sufficient by itself. You also need to start XCode' GUI application once. This will make XCode download and install more files that are needed for compilation." This is a slow download. And takes a long time to install.

Install Command Line Tools for Xcode

Install Python 3 https://www.python.org/downloads/mac-osx/

Install meson and ninja. pip3 install --user meson ninja

Your .bash_profile should have a PATH for meson and ninja : Edit your .bash_profile using your favorite editor. ( I use vim ) $ vim .bash_profile

Setting PATH for meson an ninja

export PATH=$PATH:/Users/_YOUR_USERNAMEHERE/Library/Python/3.9/bin

Setting PATH for Python 3.9

PATH="/Library/Frameworks/Python.framework/Versions/3.9/bin:${PATH}" export PATH

When this is done, you can install glib. This is probably the most difficult part. Here is the instructions for installing glib :

% tar xf glib-@GLIB_VERSION@.tar.gz # unpack the sources % cd glib-@GLIB_VERSION@ # change to the toplevel directory % meson _build # configure the build % ninja -C _build # build GLib [ Become root if necessary ] % ninja -C _build install # install GLib

Install sshfs

Edit sshfs.c Edit the sshfs.c before compiling: ( #57 ) Line 1724 in sshfs.c ( SSHFS version 2.10 ) change to sshfs.sync_read = 0 This will force sync read. This change helped the speed on my line.
```
#if FUSE_VERSION >= 26
/* Readahead should be done by kernel or sshfs but not both */
if (conn->async_read)
    sshfs.sync_read = 0;
#endif
```
Edit sshfs.c changed # include to #include ( osxfuse/osxfuse#751 )
Edit test/meson.build https://github.com/libfuse/sshfs/commit/813b75b8c2577a4d5b08eb38f7b3a9c3dd200495 --preserve=mode Yes, works for linux. For BSD : no go. The following parameter should be used for BSD / OSX : -fp . Change the line beginning with command: to read: command: ['cp', '-fp', '@input@', meson.current_build_dir() ])

( Compiling sshfs ) To build and install, use Meson (version 0.38 or newer) and Ninja. After extracting the sshfs tarball, create a (temporary) build directory and run Meson:

$ mkdir build; cd build $ meson .. To build, test and install SSHFS, you then use Ninja (running the tests requires the py.test Python module): $ ninja $ python3 -m pytest test/ # optional, but recommended $ sudo ninja install

Best regards

Metacomet10 commented 3 years ago

Thank you for the in-depth walkthrough! I haven't been able to do this yet, but I'll let you know if I run into any problems.

luzik commented 3 years ago

There is also 2.10 release

tormodwill commented 3 years ago

What is the advantage by using 2.10 ? Why is 2.1 better than 2.5 or 2.9 ?

luzik commented 3 years ago

Please read release notes https://github.com/libfuse/sshfs/releases?after=sshfs-3.3.0\

tormodwill commented 3 years ago

Thanks, luzik

So, it seems that version 2.10 is a newer ( more up to date ) version than version 2.5 or 2.9. Is this correct ?

I have been reading on this page : https://github.com/osxfuse/sshfs/blob/master/ChangeLog

Reading the release notes was usefull.

I have one question : In the release notes, i find the following for version SSHFS 3.6.0 : "Improved performance under OS X"

So, I suppose this means that it is possible to run also the latest versions of SSHFS
on OS X / masOS. How can this be done ?

My understanding was that the latest FUSE library for OS X / masOS is 2.9.9.
My understanding was that the SSHFS 3.x versions needed fuse library version 3 or later on OS X / masOS ?

Release notes from SSHFS 3.0.0: "sshfs now requires libfuse 3.1.0 or newer"

How can I run FUSE library version 3 or later on OS X / masOS ?
Currently I run FUSE library version: 2.9.9. This version is in the "LATEST VERSION" of macFUSE. ( macFUSE 4.0.5 )

tormodwill commented 3 years ago

Ping, @Nikratio

bfleischer commented 3 years ago

The latest version of libfuse shipping with macFUSE is 2.9.9. This means you are limited to SSHFS 2.x.

Is there any important feature available in SSHFS 3.7.1, but not in version 2.5?

tormodwill commented 3 years ago

The advantages running a newer version : SSHFS Version 3.6.0: Added --verbose option. Fixed a number of compiler warnings. *Improved performance under OS X.

Some serious bugs have not been handled : force ssh-fs>sync_read = 0 in line 1827 ( #57 ) changed # include to #include ( osxfuse/osxfuse#751 )

I also find a list of various features here : https://github.com/libfuse/sshfs/releases If the features are important or not to the user, depends on the various user senario. A feature that is urgent and important for me, might not be of interest for you.

You are also not answering my question :
"How can I run FUSE library version 3 or later on OS X / masOS ?"

According to the text under release notes this seems to be possible : libfuse 3.9.3 Fixed compilation under OS X and µClibc.

bfleischer commented 3 years ago

The changes you listed seem to boil down to performance improvements. Backporting the relevant code changes to version 2.10 should be doable. Don't get me wrong, porting libfuse 3 to macOS is on my to do list, but other items on the list have a higher priority.

According to the text under release notes this seems to be possible : libfuse 3.9.3 Fixed compilation under OS X and µClibc.

libfuse 3 might compile on macOS, but this does not mean that libfuse 3 will actually work. libfuse 3 currently lacks the required patches for supporting macOS. From what I can tell, the author of the comment you mentioned only added an #ifdef.

Currently the macFUSE kernel extension comes with (partial) support for FUSE ABI 7.18. This is not going to change from updating libfuse to version 3. In other words, most new libfuse 3 features will not be available on macOS. In case libfuse 3 requires newer ABI features, it will not work at all. What I'm trying to say is that updating libfuse makes most sense when updating the kernel extension in tandem.

You are also not answering my question : "How can I run FUSE library version 3 or later on OS X / masOS ?"

You would have to add support for macOS to libfuse 3.

tormodwill commented 3 years ago

Thanks for the answer, Benjamin Fleischer.

The version 2.5 can not mount under /Volumes on my machines. When I run version 2.9, I am able to mount under /Volumes. This is the behavior I see. ( Maybe I have a strange config )

It is not obvious that you have to do manual changes to the sshfs.c file. If you don´t do these changes it will not compile. ( osxfuse/osxfuse#751 )

And also had to make manual adjustments to make a temporary fix for the preformance. ( #57 )

Making a .dmg file of the SSHFS version 2.10 will fix some issues for a lot of users.

When I started using the sshfs program on macOS, I wanted it to work as seamlessly as my pcloud drive. I wanted the ssh program to connect by itself at boot. I wanted to see the icon for the ssh drive in finder, and I wanted to be able to disconnect and connect using an icon on the desktop ( without any passwords ). To be able to do this a lot of fiddling and scripts has been needed. None of this has been working at first attempt.

I suppose the mac users are demanding. We are used to things that "just works".

Timmmm commented 3 years ago

How does SSHFS 3.7.0 improve performance under OS X if osxfuse/macFUSE/FUSE for Mac only supports SSHFS 2.x? Is there a way to run SSHFS 3.x on Mac?

The proliferation of old/incompatible versions and renamed projects makes this very confusing!

knaccc commented 2 years ago

@tormodwill Thank you!! This worked for me, and SSHFS performance increased for me from 20MB/s read to 115MB/s read over a 1GbE LAN.

I have brew and Xcode installed, so the instructions were simpler for me:

brew install python3 meson ninja glib wget
wget https://ftp.acc.umu.se/pub/GNOME/sources/glib/2.66/glib-2.66.8.tar.xz
wget https://github.com/libfuse/sshfs/releases/download/sshfs-2.10/sshfs-2.10.tar.gz
wget https://pkgconfig.freedesktop.org/releases/pkg-config-0.29.2.tar.gz
tar xf glib-2.66.8.tar.xz
tar xf sshfs-2.10.tar.gz
tar xf pkg-config-0.29.2.tar.gz

edit sshfs-2.10/sshfs.c and replace: # include <fuse_darwin.h> with # include <fuse.h> and on line 1724 change sshfs.sync_read = 1; to sshfs.sync_read = 0;

edit sshfs-2.10/test/meson.build and change the line beginning with command: to read: command: ['cp', '-fPp',

cd sshfs-2.10
mkdir build; cd build
meson ..
ninja

And voila, a brand new sshfs is built.

knaccc commented 2 years ago

One thing - on MacOS, for some reason when I read from an sshfs mount connected to a local LAN server via 1GbE, it can be as slow as 17-25 MB/s for the first 30 seconds, then rises to 110MB/s for the remainder of the read. This happens consistently.

My options are big_writes,iosize=1048576,reconnect,ServerAliveInterval=15,ServerAliveCountMax=3,allow_other,noapplexattr,noappledouble,defer_permissions

Does anyone have any suggestions please?

ricfregian commented 2 years ago

My machine is a macOS BigSur 11.6 and I followed your steps @tormodwill to install sshfs 2.10. However in the pytest step I get this:

`giancaceresvera@~/Downloads/sshfs-2.10/build$ python3 -m pytest test/ ========================================================================= test session starts ========================================================================== platform darwin -- Python 3.9.10, pytest-7.1.1, pluggy-1.0.0 -- /usr/local/opt/python@3.9/bin/python3.9 cachedir: .pytest_cache rootdir: /Users/giancaceresvera/Downloads/sshfs-2.10/build/test, configfile: pytest.ini collected 4 items

test/test_sshfs.py::test_sshfs[0-False] SKIPPED (Can't find fusermount executable) [ 25%] test/test_sshfs.py::test_sshfs[0-True] SKIPPED (Can't find fusermount executable) [ 50%] test/test_sshfs.py::test_sshfs[1-False] SKIPPED (Can't find fusermount executable) [ 75%] test/test_sshfs.py::test_sshfs[1-True] SKIPPED (Can't find fusermount executable) [100%]

=========================================================================== warnings summary =========================================================================== test/conftest.py:66 /Users/giancaceresvera/Downloads/sshfs-2.10/build/test/conftest.py:66: PytestDeprecationWarning: @pytest.yield_fixture is deprecated. Use @pytest.fixture instead; they are the same. @pytest.yield_fixture(autouse=True)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ======================================================================= short test summary info ======================================================================== SKIPPED [4] test/test_sshfs.py:32: Can't find fusermount executable ==================================================================== 4 skipped, 1 warning in 0.01s ===================================================================== ` I suppose the fusermount has to be found but I'm not sure. Any suggestions please ?

tormodwill commented 2 years ago

@tormodwill Thank you!! This worked for me, and SSHFS performance increased for me from 20MB/s read to 115MB/s read over a 1GbE LAN.

I have brew and Xcode installed, so the instructions were simpler for me:
brew install python3 meson ninja glib wget
wget https://ftp.acc.umu.se/pub/GNOME/sources/glib/2.66/glib-2.66.8.tar.xz
wget https://github.com/libfuse/sshfs/releases/download/sshfs-2.10/sshfs-2.10.tar.gz
wget https://pkgconfig.freedesktop.org/releases/pkg-config-0.29.2.tar.gz
tar xf glib-2.66.8.tar.xz
tar xf sshfs-2.10.tar.gz
tar xf pkg-config-0.29.2.tar.gz
edit sshfs-2.10/sshfs.c and replace: # include <fuse_darwin.h> with # include <fuse.h> and on line 1724 change sshfs.sync_read = 1; to sshfs.sync_read = 0;

edit sshfs-2.10/test/meson.build and change the line beginning with command: to read: command: ['cp', '-fPp',
cd sshfs-2.10
mkdir build; cd build
meson ..
ninja
And voila, a brand new sshfs is built.

The correct edit of test/meson.build change the line beginning with command: to read: command: ['cp', '-fp', '@input@', meson.current_build_dir() ])

knaccc commented 2 years ago

@tormodwill thanks - what is the consequence of changing from '-fPp' to '-fp'?

tormodwill commented 2 years ago

This is my Interpretation

@tormodwill thanks - what is the consequence of changing from '-fPp' to '-fp'?

This is my interpretation : OSX is based on BSD. https://www.freebsd.org/cgi/man.cgi?cp -P is not needed, and I was not able to compile with this flag

-P No symbolic links are followed. This is the default if the -R option is specified.

-p Cause cp to preserve the following attributes of each source file in the copy: modification time, access time, file flags, file mode, ACL, user ID, and group ID, as allowed by permissions.

   If the user ID and group ID cannot be preserved, no error message
   is displayed and the exit value is not altered.

   If the source file has its set-user-ID bit on and the user ID can-
   not be preserved, the set-user-ID bit is not preserved in the
   copy's permissions.  If the source file has its set-group-ID bit on
   and the group ID cannot be preserved, the set-group-ID bit is not
   preserved in the copy's permissions.  If the source file has both
   its set-user-ID and set-group-ID bits on, and either the user ID or
   group ID cannot be preserved, neither the set-user-ID nor set-
   group-ID bits are preserved in the copy's permissions.

https://man7.org/linux/man-pages/man1/cp.1.html -P, --no-dereference never follow symbolic links in SOURCE

-p same as --preserve=mode,ownership,timestamps

tormodwill commented 2 years ago

https://github.com/tormodwill/macSSHFS

osxfuse / sshfs

Update "latest release" - sshfs for macFUSE #64

How to compile and install SSHFS version 2.10.0 on macOS

Start the install / process

Setting PATH for meson an ninja

Setting PATH for Python 3.9