Mac OS X support - Githubissues

rfjakob commented 8 years ago

Go support Mac OS X, as does the FUSE library we use, go-fuse.

gocrypts may actually work out of the box on OSX, but there probably are small issues that have to be sorted out.

At the very least, gocryptfs has to be tested on OSX. As I do not have a Mac to test on, this would be an opportunity for somebody from the Mac community to step up. Please comment here if you are interested.

rfjakob commented 7 years ago

No, should work without splice as well. Seems the go-fuse lib should check if splice is available before using it.

JayBrown commented 7 years ago

 ❯ go get github.com/rfjakob/gocryptfs
# github.com/rfjakob/gocryptfs/internal/stupidgcm
go/src/github.com/rfjakob/gocryptfs/internal/stupidgcm/locking.go:11:10: fatal error: 'openssl/evp.h' file not found
#include <openssl/evp.h>
         ^
1 error generated.
>>> elapsed time 2m13s

10.11.6

How can I go about to installing gocryptfs correctly?

rfjakob commented 7 years ago

Looks like you don't have openssl installed - please try

./build-without-openssl.bash

JayBrown commented 7 years ago

openssl is actually native to macOS. I also have a homebrewed version of openssl, which is normally keg only: I just linked it to one of the bin directories, but gocryptfs didn't install either. (evp.h is actually present in the openssl include directory; so I don't know what the problem could be… have to continue trying.)

rfjakob commented 7 years ago

I can't comment on that, but note that if your cpu has aes acceleration there is NO downside in building without openssl.

(edit on Dec 10, 2016: the "NO" was missing)

bailey27 commented 7 years ago

I have a Mac that I rarely use. I tried gocryptfs on it a while back, and it seemed to work fine then.

I upgraded it to the lastest OSX (Sierra) and tried gocryptfs on it again.

gocryptfs wouldn't compile (with go 1.7.3) because

# github.com/rfjakob/gocryptfs/internal/ctlsock
internal/ctlsock/ctlsock_serve.go:74: undefined: syscall.PathMax

So I changed syscall.PathMax to 4096.

        // 2*PATH_MAX is definitely big enough for requests to decrypt or
        // encrypt paths.
-       buf := make([]byte, 2*syscall.PathMax)
+       //buf := make([]byte, 2*syscall.PathMax)
+       buf := make([]byte, 2*4096)

It looks like syscall.PathMax is normally a const 0x1000 (4096) in go. https://golang.org/pkg/syscall/. On Ubuntu, PATH_MAX is 4096 in <linux/limits.h>.

I wonder why it's missing in OSX? In syslimits.h, OSX has PATH_MAX defined to 1024. But I think here you just want to make sure the buffer is big enough, so I think it wouldn't hurt to use 4096 there.

So I compiled it, but then I had this weird problem. When go-fuse tried to glob /dev/osxfuse*, it didn't find any fuse device files, so it said "no FUSE devices found".

I tried re-installing "FUSE for macOS" aka osxfuse, from both the project's installer and from brew. I tried disabling Apple SIP (System Integrity Protection) thinking that was preventing osxfuse from loading.

But that wasn't it.

Then I mounted something with encfs, and it worked. And then I could see the osxfuse files in /dev.

So then gocryptfs worked too. And turning SIP back on didn't make it not work, as long as encfs mounted something before gocryptfs did.

The problem seems to be that somebody needs to run

/Library/Filesystems/osxfuse.fs/Contents/Resources/load_osxfuse

before go-fuse tries to open a fuse device file.

I put logic in mount_darwin.go to, if there were no /dev/osxfuse* files, then run load_osxfuse and try again.

But then it couldn't open any of them. It said they were all busy.

So I made it run load_osxfuse, then, in a loop, try to open any, and if none were opened, then sleep 100ms and try again (50 times, so 5 secs). It still didn't work. I mean it never worked the first time I ran gocryptfs, but running gocryptfs again worked.

So then I put this code in main.go in gocryptfs to make it run load_osxfuse before it did anything else (with no changes to go-fuse)

And now it works (including the first time after a fresh boot).


+func loadFuseIfNeeded() {
+       if runtime.GOOS == "darwin" {
+               const oldLoadOsxFuseBin = "/Library/Filesystems/osxfusefs.fs/Support/load_osxfusefs"
+               const newLoadOsxFuseBin = "/Library/Filesystems/osxfuse.fs/Contents/Resources/load_osxfuse"
+               bin := oldLoadOsxFuseBin
+                if _, err := os.Stat(newLoadOsxFuseBin); err == nil {
+                        bin = newLoadOsxFuseBin
+                }
+                cmd := exec.Command(bin);
+                cmd.Run();
+       }
+}
+
 func main() {
+       loadFuseIfNeeded()

bailey27 commented 7 years ago

I think this is maybe better (more directed). It works as well.


diff --git a/main.go b/main.go
index 8ab180e..a4a46f7 100644
--- a/main.go
+++ b/main.go
@@ -3,7 +3,7 @@ package main
 import (
        "fmt"
        "os"
-
+       "os/exec"
        "path/filepath"
        "runtime"
        "runtime/pprof"
@@ -118,12 +118,33 @@ func printVersion() {
                tlog.ProgramName, GitVersion, buildFlags, GitVersionFuse, built)
 }

+func loadFuseIfNeeded() {
+       if runtime.GOOS == "darwin" {
+               if _, err := os.Stat("/dev/osxfuse0"); err != nil {
+                       const oldLoadOsxFuseBin = "/Library/Filesystems/osxfusefs.fs/Support/load_osxfusefs"
+                       const newLoadOsxFuseBin = "/Library/Filesystems/osxfuse.fs/Contents/Resources/load_osxfuse"
+                       bin := oldLoadOsxFuseBin
+                       if _, err := os.Stat(newLoadOsxFuseBin); err == nil {
+                               bin = newLoadOsxFuseBin
+                       }
+                       cmd := exec.Command(bin);
+                       cmd.Run();
+               }
+       }
+}
+
 func main() {
        runtime.GOMAXPROCS(4)
        var err error
        // Parse all command-line options (i.e. arguments starting with "-")
        // into "args". Path arguments are parsed below.
        args := parseCliOpts()
+
+       // On some OSes we might need to load fuse if we are mounting a
+       // filesystem.
+       if (flagSet.NArg() == 2) {
+               loadFuseIfNeeded()
+       }
        // Fork a child into the background if "-fg" is not set AND we are mounting
        // a filesystem. The child will do all the work.
        if !args.fg && flagSet.NArg() == 2 {

rfjakob commented 7 years ago

Nice detective work! When you run load_osxfuse manually, does it work as well?

spaghetti2514 commented 7 years ago

When you run load_osxfuse manually, does it work as well?

After applying bailey27's fix for syscall.PathMax and then manually running load_osxfuse, gocryptfs works fine for me. I did not try it without manually running load_osxfuse to try and reproduce bailey27's issue, but assuming it's accurate, running manually fixes it.

Also of note, test.bash fails to run on OS X due to the flock utility not existing

openssl is actually native to macOS.

OS X ships an (old) openssl binary and libcrypto, but has not shipped openssl headers since before El Capitan. The headers are necessary to build against it. Even if you could build against it, it's too old to support some of the functions required by stupidgcm

Homebrew recently started refusing to link their modern openssl to system locations, citing concerns that software built against their headers would end up linked to OS X's old openssl. They now recommend explicitly passing include dirs for homebrew's openssl when building anything against it.

Edit: I am currently accomplishing this by adding

// #cgo LDFLAGS: -L/usr/local/opt/openssl/lib
// #cgo CFLAGS: -I/usr/local/opt/openssl/include

to stupidgcm.go

bailey27 commented 7 years ago

@rfjakob,

Yes, it worked when I ran load_osxfuse manually before running gocryptfs.

@spaghetti2514 ,

I seem to be linking with the openssl that came with brew.


$ otool -L gocryptfs
gocryptfs:
    /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)

But I didn't have to do anything special with include or library paths to get it to compile.

spaghetti2514 commented 7 years ago

@bailey27 Is there something in your environment allowing you to do that? Or maybe some old symlinks in a more system-wide location? Without include and library paths I get the same error about openssl/evp.h that @JayBrown got.

bailey27 commented 7 years ago

@spaghetti2514 ,

I re-installed my mac from scratch about two years ago. I think that's maybe why I don't have any openssl besides what comes with brew (installed/linked into /usr/local).

I don't know why my go build environment is looking in /usr/local/include. I haven't set any environment variables about it. My go isn't installed from bew (I downloaded the mac installer from the golang.org).

It looks like when I compile something from the command line with cc it doesn't look for headers in /usr/local/include. So I think it isn't system wide.

rfjakob commented 7 years ago

@bailey27 I have dropped syscall.PathMax and now just use 5000 hardcoded ( https://github.com/rfjakob/gocryptfs/commit/12374be9c51b0298c85751e4652fe4e852c85546 ). Unless Linux raises the 4096-byte limit, that should be good enough.

rfjakob commented 7 years ago

@spaghetti2514 I guess virtually all Macs that are running today have AES acceleration, so I'd just completely disregard openssl and build using ./build-without-openssl.bash.

You can check the impact of using openssl by commenting out line 51 in benchmark.bash and enabling one of the other two lines below: https://github.com/rfjakob/gocryptfs/blob/master/benchmark.bash#L51

spaghetti2514 commented 7 years ago

@rfjakob benchmark.bash

Testing gocryptfs at /tmp/benchmark.bash.fk2
md5sum: stat '/tmp/linux-3.0.tar.gz': No such file or directory
Downloading linux-3.0.tar.gz
/tmp/linux-3.0.tar.gz                 100%[========================================================================>]  92.20M  3.16MB/s    in 29s     
2016-12-10 15:14:43 URL:https://cdn.kernel.org/pub/linux/kernel/v3.0/linux-3.0.tar.gz [96675825/96675825] -> "/tmp/linux-3.0.tar.gz" [1]
WRITE: dd: bs: illegal numeric value
rm: zero: No such file or directory
./benchmark.bash: line 62: fusermount: command not found

rfjakob commented 7 years ago

@spaghetti2514 Fixed in 2bacbdf99de27a132c236003953574a321b0cdbc . Also, I have added proper option parsing, so you can now run

./benchmark.bash -openssl=false

or

./benchmark.bash -openssl=true

spaghetti2514 commented 7 years ago

@rfjakob

$ ./benchmark.bash -openssl=false
Testing gocryptfs at /tmp/benchmark.bash.SzA
md5sum: stat '/tmp/linux-3.0.tar.gz': No such file or directory
Downloading linux-3.0.tar.gz
/tmp/linux-3.0.tar.gz                 100%[========================================================================>]  92.20M  2.95MB/s    in 32s     
2016-12-13 17:24:47 URL:https://cdn.kernel.org/pub/linux/kernel/v3.0/linux-3.0.tar.gz [96675825/96675825] -> "/tmp/linux-3.0.tar.gz" [1]
WRITE: 262144000 bytes transferred in 14.460788 secs (18127919 bytes/sec)
UNTAR: /usr/bin/time: illegal option -- f
usage: time [-lp] command.
./benchmark.bash: line 76: fusermount: command not found

Removing the -f option from time:

$ ./benchmark.bash -openssl=false   
Testing gocryptfs at /tmp/benchmark.bash.gjd
WRITE: 262144000 bytes transferred in 16.244092 secs (16137806 bytes/sec)
UNTAR: linux-3.0/arch/microblaze/boot/dts/system.dts: Can't set permissions to 0755Can't update time for linux-3.0/arch/microblaze/boot/dts/system.dts
linux-3.0/sound/soc/davinci/Makefile: Can't create 'linux-3.0/sound/soc/davinci/Makefile'
linux-3.0/sound/soc/davinci/davinci-evm.c: Can't create 'linux-3.0/sound/soc/davinci/davinci-evm.c'
linux-3.0/sound/soc/davinci/davinci-i2s.c: Can't create 'linux-3.0/sound/soc/davinci/davinci-i2s.c'
[snip 694 lines]
linux-3.0/virt/kvm/irq_comm.c: Can't create 'linux-3.0/virt/kvm/irq_comm.c'
linux-3.0/virt/kvm/kvm_main.c: Can't create 'linux-3.0/virt/kvm/kvm_main.c'
tar: Error exit delayed from previous errors.
      245.72 real         3.28 user        12.95 sys
./benchmark.bash: line 76: fusermount: command not found

There was a very long pause after the first permission/utime error, and then the flood of create errors all came at once.

mhogomchungu commented 7 years ago

Greeting to all who have participated in this discussion.

I have a project called "SiriKali"[1] and i recently added support for OSX and would appreciate if somebody here who have gocryptfs running in OSX would try it out and report if it works just fine with gocryptfs. Git versions of both gocryptfs and SiriKali will have to be used for this test.

[1] https://github.com/mhogomchungu/sirikali

bits commented 7 years ago

At first look, gocryptfs appears to be working for me on OS X El Capitan 10.11.6. I'm able to init, mount, write, unmount, mount, read and get back what was written. There is still work to be done to pass all the tests.

Here are some notes:

Build and run gocryptfs on Mac

Build gocryptfs "without openssl" on OS X / mac OS

Install go from https://golang.org/dl/ -or- using MacPorts: port install go -or- Homebrew: brew install go. I used MacPorts.
Install FUSE for macOS from https://osxfuse.github.io/
export GOPATH="${GOPATH=$HOME}" # set GOPATH to ~ if it's not already set
go get -d github.com/rfjakob/gocryptfs
$GOPATH/src/github.com/rfjakob/gocryptfs/build-without-openssl.bash # compiles to $GOPATH/bin/gocryptfs

Initialize an encrypted directory

test -e /dev/osxfuse0 || /Library/Filesystems/osxfuse.fs/Contents/Resources/load_osxfuse # initialize FUSE for macOS once per boot until @bailey27's suggestion is merged in.
GOCRYPTFS_ENCRYPTED_DIR=~/secrets.encrypted
GOCRYPTFS_MOUNT_DIR=~/secrets
mkdir $GOCRYPTFS_ENCRYPTED_DIR $GOCRYPTFS_MOUNT_DIR
$GOPATH/bin/gocryptfs -init "$GOCRYPTFS_ENCRYPTED_DIR" # choose a passphrase

Unlock and mount encrypted directory and write to it

$GOPATH/bin/gocryptfs "$GOCRYPTFS_ENCRYPTED_DIR" "$GOCRYPTFS_MOUNT_DIR" # supply passphrase
echo "secret text" > $GOCRYPTFS_MOUNT_DIR/test
tree $GOCRYPTFS_MOUNT_DIR
tree $GOCRYPTFS_ENCRYPTED_DIR

Unmount and lock

umount $GOCRYPTFS_MOUNT_DIR

Unlock and remount

$GOPATH/bin/gocryptfs "$GOCRYPTFS_ENCRYPTED_DIR" "$GOCRYPTFS_MOUNT_DIR" # supply passphrase
test "$(cat $GOCRYPTFS_MOUNT_DIR/test)" == "secret text" && echo "It works!" || echo "Not working :("

It works!

Adapt tests and benchmarks for Mac

The following changes were made to get gocryptfs passing more tests and the benchmarks. No effort has yet been made to make these changes portable across operating systems so the same codebase works on Linux and Mac (and maybe the other BSDs?). My goal was to get it minimally working and run the tests to get a better handle on what's missing.

fusermount → umount

There is no fusermount with FUSE for macOS, they recommend using umount. https://github.com/osxfuse/osxfuse/wiki/FAQ#48-how-should-i-unmount-my-fuse-for-os-x-file-system-i-cannot-find-the-fusermount-program-anywhere

perl -p -i -e 's/fusermount -u -z/umount/g' \
   $GOPATH/src/github.com/rfjakob/gocryptfs/benchmark.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/benchmark-reverse.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/reverse/linux-tarball-test.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/stress_tests/extractloop.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/stress_tests/fsstress-gocryptfs.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/stress_tests/fsstress-loopback.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/stress_tests/pingpong-rsync.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/stress_tests/pingpong.bash

perl -p -i -e 's/fusermount -u/umount/g' \
   $GOPATH/src/github.com/rfjakob/gocryptfs/test.bash

perl -p -i -e 's/"fusermount", "-u", "-z"/"umount"/g' \
   $GOPATH/src/github.com/rfjakob/gocryptfs/mount.go \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/test_helpers/helpers.go

time

OS X / macOS's time does not support -f to set the format string, so let's use grep to parse out the elapsed "real" time

perl -p -i -e 's!LC_ALL=C /usr/bin/time -f %e 2>&1 $@ > /dev/null!LC_ALL=C /usr/bin/time -p 2>\&1 $@ > /dev/null | grep real | grep -Eo "[0-9.]+"!g' \
   $GOPATH/src/github.com/rfjakob/gocryptfs/benchmark-reverse.bash \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/canonical-benchmarks.bash

stat

Linux uses stat -c %s to get size in bytes, while OS X / macOS uses stat -f %z

perl -p -i -e 's/stat -c %s/stat -f %z/g' \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/dl-linux-tarball.bash

cat /proc/mount → mount

Instead of cat /proc/mount, parse mount, where the 3rd field is the mounted directory

perl -p -i -e 's|cat /proc/mounts|mount|'  $GOPATH/src/github.com/rfjakob/gocryptfs/test.bash
perl -p -i -e 's|cut -f2 -d" "|cut -f3 -d" "|'  $GOPATH/src/github.com/rfjakob/gocryptfs/test.bash

syscall.PathMax → 5000

Hardcode a value since there's no syscall.PathMax on Mac

perl -p -i -e 's|syscall.PathMax|5000|'  \
   $GOPATH/src/github.com/rfjakob/gocryptfs/tests/test_helpers/helpers.go

Eliminate variable warnings

Declare variables to eliminate warnings

perl -p -i -e '
   s|\) \(err error\) \{|) error {|g;
   s|\(\w+ int, err error\)|(int, error)|g;
   s|\(\w+ string, err error\)|(string, error)|g;
   s|oldpath, err = dirfdAbs\(olddirfd, oldpath\)|oldpath, err := dirfdAbs(olddirfd, oldpath)|;
   s|path, err = dirfdAbs\(dirfd, path\)|path, err := dirfdAbs(dirfd, path)|g;
   undef $/; # for multiline grep
   s|\}\n\terr = syscall.Fchdir\(dirfd\)|}\n\terr := syscall.Fchdir(dirfd)|;
' $GOPATH/src/github.com/rfjakob/gocryptfs/internal/syscallcompat/sys_darwin.go

Benchmark

Benchmark on a MacBook Pro (Retina, 15-inch, Mid 2015) with 2.8 GHz Intel Core i7

$GOPATH/src/github.com/rfjakob/gocryptfs/benchmark.bash

Testing gocryptfs at /tmp/benchmark.bash.25P
WRITE: 262144000 bytes transferred in 0.787978 secs (332679266 bytes/sec)
UNTAR: 32.33
LS:    7.93
RM:    13.31

Build gocryptfs "with openssl" on OS X / mac OS

To build with OpenSSL, install it from Homebrew brew install openssl, then follow @spaghetti2514's tip and reference it in stupidgcm.go. According to @rfjakob, building with OpenSSL is not necessary since modern Macs all have AES instructions. But it seems possible to compile with OpenSSL as well for the speedup that OpenSSL offers over go's built-in crypto when your CPU doesn't have AES extensions.

perl -p -i -e 's|// +build !without_openssl|// +build !without_openssl\n// #cgo LDFLAGS: -L/usr/local/opt/openssl/lib\n// #cgo CFLAGS: -I/usr/local/opt/openssl/include|' \
   $GOPATH/src/github.com/rfjakob/gocryptfs/internal/stupidgcm/stupidgcm.go

Run tests

$GOPATH/src/github.com/rfjakob/gocryptfs/test.bash -v

Warning: unmounting leftover filesystem: /private/tmp/gocryptfs-test-parent/591257554/v1.1-reverse.B
Warning: unmounting leftover filesystem: /private/tmp/gocryptfs-test-parent/591257554/v1.1-reverse-plaintextnames.B
gocryptfs v1.2-35-g0f40afc-dirty without_openssl; go-fuse 0ad840c; 2017-02-11 go1.7.4
github.com/jacobsa/crypto/common
github.com/hanwen/go-fuse/fuse
github.com/rfjakob/gocryptfs/internal/stupidgcm
golang.org/x/crypto/ssh/terminal
github.com/rfjakob/eme
golang.org/x/crypto/pbkdf2
github.com/rfjakob/gocryptfs/internal/syscallcompat
github.com/jacobsa/crypto/cmac
golang.org/x/crypto/scrypt
github.com/jacobsa/crypto/siv
github.com/rfjakob/gocryptfs/internal/siv_aead
github.com/rfjakob/gocryptfs/internal/tlog
github.com/rfjakob/gocryptfs/internal/ctlsock
github.com/rfjakob/gocryptfs/internal/readpassword
github.com/hanwen/go-fuse/fuse/nodefs
github.com/hanwen/go-fuse/fuse/pathfs
github.com/rfjakob/gocryptfs/internal/cryptocore
github.com/rfjakob/gocryptfs/internal/prefer_openssl
github.com/rfjakob/gocryptfs/internal/contentenc
github.com/rfjakob/gocryptfs/internal/nametransform
github.com/rfjakob/gocryptfs/internal/configfile
github.com/rfjakob/gocryptfs/internal/fusefrontend
github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse
github.com/rfjakob/gocryptfs
github.com/jacobsa/crypto/common
golang.org/x/crypto/ssh/terminal
github.com/rfjakob/gocryptfs/internal/stupidgcm
github.com/jacobsa/crypto/cmac
github.com/jacobsa/crypto/siv
github.com/rfjakob/gocryptfs/internal/siv_aead
github.com/rfjakob/gocryptfs/internal/tlog
github.com/rfjakob/gocryptfs/internal/cryptocore
github.com/rfjakob/gocryptfs/internal/contentenc
github.com/rfjakob/gocryptfs/gocryptfs-xray
gocryptfs v1.2-35-g0f40afc-dirty; go-fuse 0ad840c; 2017-02-11 go1.7.4
# github.com/rfjakob/gocryptfs/tests/matrix
tests/matrix/matrix_test.go:728: st.Atim undefined (type syscall.Stat_t has no field or method Atim)
tests/matrix/matrix_test.go:728: st.Mtim undefined (type syscall.Stat_t has no field or method Mtim)
=== RUN   TestPrefixOArgs
--- PASS: TestPrefixOArgs (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs  0.010s
?     github.com/rfjakob/gocryptfs/gocryptfs-xray [no test files]
=== RUN   TestLoadV1
Unsupported on-disk format 1--- PASS: TestLoadV1 (0.00s)
=== RUN   TestLoadV2
--- PASS: TestLoadV2 (0.58s)
=== RUN   TestLoadV2PwdError
failed to unlock master key: cipher: message authentication failed
--- PASS: TestLoadV2PwdError (0.51s)
=== RUN   TestLoadV2Feature
--- PASS: TestLoadV2Feature (0.00s)
=== RUN   TestLoadV2StrangeFeature
Unsupported feature flag "StrangeFeatureFlag"--- PASS: TestLoadV2StrangeFeature (0.01s)
=== RUN   TestCreateConfFile
--- PASS: TestCreateConfFile (0.01s)
=== RUN   TestCreateConfFileAESSIV
--- PASS: TestCreateConfFileAESSIV (0.01s)
=== RUN   TestCreateConfFileRaw64
--- PASS: TestCreateConfFileRaw64 (0.01s)
=== RUN   TestIsFeatureFlagKnown
--- PASS: TestIsFeatureFlagKnown (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/configfile  1.146s
=== RUN   TestSplitRange
--- PASS: TestSplitRange (0.00s)
=== RUN   TestCiphertextRange
--- PASS: TestCiphertextRange (0.00s)
=== RUN   TestBlockNo
--- PASS: TestBlockNo (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/contentenc  0.010s
=== RUN   TestCryptoCoreNewGo15
--- PASS: TestCryptoCoreNewGo15 (0.00s)
=== RUN   TestCryptoCoreNew
--- PASS: TestCryptoCoreNew (0.00s)
=== RUN   TestNewPanic
2017/02/10 23:57:00 Unsupported key length 16
--- PASS: TestNewPanic (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/cryptocore  0.011s
=== RUN   TestSanitizePath
--- PASS: TestSanitizePath (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/ctlsock 0.009s
?     github.com/rfjakob/gocryptfs/internal/fusefrontend  [no test files]
?     github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse  [no test files]
=== RUN   TestIsLongName
--- PASS: TestIsLongName (0.00s)
=== RUN   TestPad16
--- PASS: TestPad16 (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/nametransform 0.013s
=== RUN   TestCurrentCPU
--- PASS: TestCurrentCPU (0.00s)
  prefer_test.go:8: PreferOpenSSL=false
=== RUN   TestXeonE312xx
--- PASS: TestXeonE312xx (0.00s)
=== RUN   TestPentiumG630
--- PASS: TestPentiumG630 (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/prefer_openssl  0.012s
=== RUN   TestExtpass
--- PASS: TestExtpass (0.00s)
=== RUN   TestOnceExtpass
--- PASS: TestOnceExtpass (0.00s)
=== RUN   TestTwiceExtpass
--- PASS: TestTwiceExtpass (0.00s)
=== RUN   TestExtpassEmpty
--- PASS: TestExtpassEmpty (0.01s)
=== RUN   TestStdin
--- PASS: TestStdin (0.01s)
=== RUN   TestStdinEof
--- PASS: TestStdinEof (0.01s)
=== RUN   TestStdinEmpty
--- PASS: TestStdinEmpty (0.01s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/readpassword  0.045s
=== RUN   TestKeyLens
--- PASS: TestKeyLens (0.00s)
=== RUN   TestK32
--- PASS: TestK32 (0.00s)
=== RUN   TestK64
--- PASS: TestK64 (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/siv_aead  0.010s
=== RUN   TestEncryptDecrypt
--- PASS: TestEncryptDecrypt (0.13s)
=== RUN   TestCorruption
--- PASS: TestCorruption (0.00s)
PASS
ok    github.com/rfjakob/gocryptfs/internal/stupidgcm 0.152s
?     github.com/rfjakob/gocryptfs/internal/syscallcompat [no test files]
?     github.com/rfjakob/gocryptfs/internal/tlog  [no test files]
=== RUN   TestInit
--- PASS: TestInit (0.03s)
=== RUN   TestInitAessiv
--- PASS: TestInitAessiv (0.03s)
=== RUN   TestInitReverse
--- PASS: TestInitReverse (0.03s)
=== RUN   TestPasswd
--- PASS: TestPasswd (0.30s)
=== RUN   TestPasswdMasterkey
--- PASS: TestPasswdMasterkey (0.22s)
=== RUN   TestPasswdReverse
--- PASS: TestPasswdReverse (0.08s)
=== RUN   TestInitConfig
--- PASS: TestInitConfig (0.05s)
=== RUN   TestRo
--- PASS: TestRo (0.08s)
=== RUN   TestNonempty
--- PASS: TestNonempty (0.08s)
=== RUN   TestShadows
--- PASS: TestShadows (0.10s)
PASS
ok    github.com/rfjakob/gocryptfs/tests/cli  0.995s

Tests hang at this point. ^C^C^C has no effect. killing the ../../gocryptfs -fg -notifypid=22251 -q -wpanic -nosyslog -zerokey /tmp/gocryptfs-test-parent/827638449/default-cipher /tmp/gocryptfs-test-parent/827638449/default-plain process outputs:

=== RUN   Test1980Tar
--- PASS: Test1980Tar (0.02s)
=== RUN   TestCtlSock
--- PASS: TestCtlSock (0.11s)
=== RUN   TestOpenTruncateRead
--- PASS: TestOpenTruncateRead (0.01s)
PASS
umount(/private/tmp/gocryptfs-test-parent/827638449/default-plain): Resource busy -- try 'diskutil unmount'
exit status 1
panic: exit status 1

goroutine 1 [running]:
panic(0x41acd00, 0xc4200b2260)
  /opt/local/lib/go/src/runtime/panic.go:500 +0x1a1
github.com/rfjakob/gocryptfs/tests/test_helpers.UnmountPanic(0xc4200147c0, 0x32)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/tests/test_helpers/helpers.go:172 +0xe0
github.com/rfjakob/gocryptfs/tests/defaults.TestMain(0xc420053ee8)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/tests/defaults/main_test.go:19 +0xba
main.main()
  github.com/rfjakob/gocryptfs/tests/defaults/_test/_testmain.go:70 +0xc6
FAIL  github.com/rfjakob/gocryptfs/tests/defaults 56.836s

and then continues to hang. Killing the ../../gocryptfs -fg -notifypid=22471 -q -wpanic -nosyslog -aessiv -masterkey e7fb8f0d-2a81df9e-26611e4b-5540b218-e48aa458-c2a623af-d0c82637-1466b5f2 -openssl=true /tmp/gocryptfs-test-parent/787312488/v1.1-reverse-plaintextnames.B /tmp/gocryptfs-test-parent/787312488/v1.1-reverse-plaintextnames.C process outputs:

example_filesystems: testing with "-openssl=true"
=== RUN   TestExampleFSv04
--- PASS: TestExampleFSv04 (0.03s)
=== RUN   TestExampleFSv05
--- PASS: TestExampleFSv05 (0.03s)
=== RUN   TestExampleFSv06
--- PASS: TestExampleFSv06 (0.03s)
=== RUN   TestExampleFSv06PlaintextNames
--- PASS: TestExampleFSv06PlaintextNames (0.03s)
=== RUN   TestExampleFSv07
--- PASS: TestExampleFSv07 (0.20s)
=== RUN   TestExampleFSv07PlaintextNames
--- PASS: TestExampleFSv07PlaintextNames (0.17s)
=== RUN   TestExampleFSv09
--- PASS: TestExampleFSv09 (0.18s)
=== RUN   TestExampleFSv11
--- PASS: TestExampleFSv11 (0.13s)
=== RUN   TestExampleFSv11reverse
decryptPath: tried to decrypt "gocryptfs.longname.3b-SW_DxXiLVpverS0DDoO8RIR2D0SwoEKdqyB4zX_E=.name"!? Returning EINVAL.
-wpanic turns this warning into a panic: decryptPath: tried to decrypt "gocryptfs.longname.3b-SW_DxXiLVpverS0DDoO8RIR2D0SwoEKdqyB4zX_E=.name"!? Returning EINVAL.
panic: -wpanic turns this warning into a panic: decryptPath: tried to decrypt "gocryptfs.longname.3b-SW_DxXiLVpverS0DDoO8RIR2D0SwoEKdqyB4zX_E=.name"!? Returning EINVAL.

goroutine 60 [running]:
panic(0x41c0f60, 0xc4202320f0)
  /opt/local/lib/go/src/runtime/panic.go:500 +0x1a1
log.(*Logger).Panic(0xc420012370, 0xc420040bc0, 0x1, 0x1)
  /opt/local/lib/go/src/log/log.go:206 +0xc7
github.com/rfjakob/gocryptfs/internal/tlog.(*toggledLogger).Printf(0xc42001a6c0, 0x4214f1c, 0x34, 0xc420040c80, 0x1, 0x1)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/tlog/log.go:64 +0x245
github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse.(*ReverseFS).rDecryptName(0xc42001e480, 0xc4202800a0, 0x44, 0xc420248280, 0x10, 0x20, 0x0, 0x0, 0x45, 0x44, ...)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse/rpath.go:75 +0x44d
github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse.(*ReverseFS).decryptPath(0xc42001e480, 0xc4202800a0, 0x44, 0x0, 0x0, 0x0, 0x0)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse/rpath.go:90 +0x5bc
github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse.(*ReverseFS).Access(0xc42001e480, 0xc4202800a0, 0x44, 0xc400000000, 0xc420216648, 0xc4202341c0)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse/rfs.go:237 +0x88
github.com/hanwen/go-fuse/fuse/pathfs.(*pathInode).Access(0xc4201ea090, 0x0, 0xc420216648, 0x0)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/pathfs/pathfs.go:389 +0x68
github.com/hanwen/go-fuse/fuse/nodefs.(*rawBridge).Access(0xc42001b440, 0xc420216630, 0x0)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/nodefs/fsops.go:361 +0x65
github.com/hanwen/go-fuse/fuse.doAccess(0xc42009e620, 0xc420216480)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/opcode.go:357 +0x42
github.com/hanwen/go-fuse/fuse.(*Server).handleRequest(0xc42009e620, 0xc420216480, 0xc420078660)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/server.go:398 +0xd3
created by github.com/hanwen/go-fuse/fuse.(*Server).loop
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/server.go:375 +0x162
Cannot open config file: open /tmp/gocryptfs-test-parent/787312488/v1.1-reverse.B/gocryptfs.conf: device not configured
--- FAIL: TestExampleFSv11reverse (0.07s)
  example_filesystems_test.go:177: /tmp/gocryptfs-test-parent/787312488/v1.1-reverse.B/gocryptfs.conf missing
  helpers.go:163: mount failed: exit status 8
=== RUN   TestExampleFSv11reversePlaintextnames
decryptPath: tried to decrypt "gocryptfs.longname._9a7xcJ1f9d8WBQ3AffeY3UD7PtukytrkQpRXVOtbdU=.name"!? Returning EINVAL.
-wpanic turns this warning into a panic: decryptPath: tried to decrypt "gocryptfs.longname._9a7xcJ1f9d8WBQ3AffeY3UD7PtukytrkQpRXVOtbdU=.name"!? Returning EINVAL.
panic: -wpanic turns this warning into a panic: decryptPath: tried to decrypt "gocryptfs.longname._9a7xcJ1f9d8WBQ3AffeY3UD7PtukytrkQpRXVOtbdU=.name"!? Returning EINVAL.

goroutine 46 [running]:
panic(0x41c0f60, 0xc420142520)
  /opt/local/lib/go/src/runtime/panic.go:500 +0x1a1
log.(*Logger).Panic(0xc420012370, 0xc420132bc0, 0x1, 0x1)
  /opt/local/lib/go/src/log/log.go:206 +0xc7
github.com/rfjakob/gocryptfs/internal/tlog.(*toggledLogger).Printf(0xc42001a6c0, 0x4214f1c, 0x34, 0xc420132c80, 0x1, 0x1)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/tlog/log.go:64 +0x245
github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse.(*ReverseFS).rDecryptName(0xc42001e280, 0xc4201402d0, 0x44, 0xc4201461e0, 0x10, 0x20, 0x0, 0x0, 0x45, 0x44, ...)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse/rpath.go:75 +0x44d
github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse.(*ReverseFS).decryptPath(0xc42001e280, 0xc4201402d0, 0x44, 0x0, 0x0, 0x0, 0x0)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse/rpath.go:90 +0x5bc
github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse.(*ReverseFS).Access(0xc42001e280, 0xc4201402d0, 0x44, 0xc400000000, 0xc420112408, 0xc42013ee70)
  /Users/bits/go/src/github.com/rfjakob/gocryptfs/internal/fusefrontend_reverse/rfs.go:237 +0x88
github.com/hanwen/go-fuse/fuse/pathfs.(*pathInode).Access(0xc4201561b0, 0x0, 0xc420112408, 0x0)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/pathfs/pathfs.go:389 +0x68
github.com/hanwen/go-fuse/fuse/nodefs.(*rawBridge).Access(0xc42001ae40, 0xc4201123f0, 0x0)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/nodefs/fsops.go:361 +0x65
github.com/hanwen/go-fuse/fuse.doAccess(0xc42009e540, 0xc420112240)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/opcode.go:357 +0x42
github.com/hanwen/go-fuse/fuse.(*Server).handleRequest(0xc42009e540, 0xc420112240, 0xc420106000)
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/server.go:398 +0xd3
created by github.com/hanwen/go-fuse/fuse.(*Server).loop
  /Users/bits/go/src/github.com/hanwen/go-fuse/fuse/server.go:375 +0x162
--- FAIL: TestExampleFSv11reversePlaintextnames (0.18s)
  example_filesystems_test.go:221: /tmp/gocryptfs-test-parent/787312488/v1.1-reverse-plaintextnames.B/gocryptfs.conf missing
  example_test_helpers.go:18: open /tmp/gocryptfs-test-parent/787312488/v1.1-reverse-plaintextnames.C/status.txt: device not configured
  example_test_helpers.go:71: open /tmp/gocryptfs-test-parent/787312488/v1.1-reverse-plaintextnames.C/longname_255_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: device not configured
FAIL
FAIL  github.com/rfjakob/gocryptfs/tests/example_filesystems  64.622s

@rfjakob, Can you weigh in on what you think the next steps are?

rfjakob commented 7 years ago

@bits Great work, thanks for the detailed report!

I'll try and merge in as much of the test fixes as possible tomorrow. The hang in the tests may be caused by a hanging unmount? I'll wrap the unmount call with a timeout that prints some status info.

Not yet sure about the decryptPath: tried to decrypt "gocryptfs.longname.XXX... panic. May be a bug in gocryptfs that's hard to hit under Linux.

bits commented 7 years ago

According to the syslimits.h header file distributed in OS X 10.11.6, the maximum filename length of an individual file is 255 bytes, and the maximum length of a path string is 1024 bytes.

$ grep -E '\b(PATH_MAX|NAME_MAX)' /usr/include/sys/syslimits.h

#define NAME_MAX          255   /* max bytes in a file name */
#define PATH_MAX         1024   /* max bytes in pathname */

According to http://stackoverflow.com/a/32408084

From actual testing on Mac OS X Yosemite, the max path length is 1016 characters. 1017 fails.

bits commented 7 years ago

I forgot to mention in the previous notes that I needed to install a flock program since Mac doesn't include one. Cloning, compiling and installing the cross-platform https://github.com/discoteq/flock with cd /tmp && git clone https://github.com/discoteq/flock.git && cd flock && ./autogen.sh && make && sudo make install worked for me.

Upon closer inspection, it seems like only test.bash uses this. Perhaps flock could be avoided entirely with a shell file locking mechanism:

# Set up a race-safe lock file so that only one copy of this script will run at a time
# Use Bash's noclobber mode, which won't redirect to an existing file
SELFNAME=$(basename "$0")
TESTDIR=/tmp/gocryptfs-test-parent
LOCKFILE="$TESTDIR/$SELFNAME.lock"
if (set -o noclobber; echo "$$" > "$LOCKFILE") 2> /dev/null; then  # Lock acquired

   # Set up a trap to delete the lock file when this script exits for any reason
   trap 'rm -f "$LOCKFILE"; exit $?' EXIT

   #
   # Perform lock-protected commands
   #

   # Remove the lock file and clean up the trap for a clean exit code
   rm -f "$LOCKFILE"
   trap - EXIT
else
   echo -n "Failed to acquire lockfile: $LOCKFILE; Held by PID "
   cat "$LOCKFILE"
   exit 1
fi

rfjakob commented 7 years ago

@bits I looked into replacing all fusermount calls with a compatability wrapper, and it turned out to add a lot of boilerplace code to all scripts. What do you think of instead placing this script in your PATH?

bits commented 7 years ago

@rfjakob It might work around the unmounting issue at hand, but it could have unintended consequences for users.

There might be some confusion when they apparently have a fusermount command but it fails to mount anything. Difficulty could also arise when another program checks for, and finds a fusermount, relies on it, and then inexplicably fails. Also, it's an additional step that raises the bar that many new users would have to overcome to get going with gocryptfs on a Mac, taking away from the beautiful simplicity of placing a single binary in the PATH.

Fundamentally, fusermount is a Linux-only command. macOS, FreeBSD, OpenBSD, Solaris, etc… all use umount to dismount FUSE filesystems. Perhaps it would be better to use the OS-native command rather than create a partial stub for an OS-specific one, especially since mounting/unmounting a path is a central concept with this project.

Maybe the scripts could include:

fuse-umount() { test "$(uname)" = "Linux" && fusermount -u -z $@ || umount $@; }

Replacing the fusermount -u -z calls with fuse-umount would get that aspect of the scripts running on several more platforms without too much boilerplate.

bits commented 7 years ago

@rfjakob Have you considered implementing gocryptfs -u MOUNTPOINT to unmount?

rfjakob commented 7 years ago

Good points about fusermount. I'll add an internal wrapper that checks $OSTYPE to do the right thing.

About gocryptfs -u, no, I think I don't want to pull OS functionality into gocryptfs.

rfjakob commented 7 years ago

fusermount issues should be fixed by https://github.com/rfjakob/gocryptfs/commit/ce2e610428c940c2bd5ca1790e7375117b1f6015 .

rfjakob commented 7 years ago

@bits I think I have adapted/fixed everything for OSX except

flock. Since flock it is available through homebrew and the bash replacement is ginormous I want to keep it as-is
build with openssl. I think if you really want it fixing the path is reasonable.
The test hang. This needs more debugging (see below)

When the tests hang again, please send a kill -ABRT to the first go process. The best way (IMO) to find the right process is to note the PID of the shell you are running the tests from (echo $$), and then, once it hangs, run pstree -ap PID to see the process tree the below the shell. At receiving the ABRT, the process should dump a backtrace to the terminal.

Example:

$ pstree -ap 3385
bash,3385
  └─test.bash,24344 ./test.bash
      └─test.bash,24348 ./test.bash
          └─go,24704 test ./...
              ├─{go},24705
              ├─{go},24706
              ├─{go},24707
              ├─{go},24708
              ├─{go},24719
              ├─{go},24724
              ├─{go},24776
              ├─{go},24777
              ├─{go},24781
              └─{go},24783
$ kill -ABRT 24704

spaghetti2514 commented 7 years ago

@rfjakob

I installed the third party flock implementation linked by @bits, but it does not seem to support the --nonblock option that test.bash uses. So I'm running test.bash with the flock block commented out, and with the include paths added to stupidgcm's source. Everything else should be vanilla.

Here is the backtrace resulting from killing the top go process when the tests hang (with the usual test results before it omitted):

ok      github.com/rfjakob/gocryptfs/tests/cli  2.550s
SIGABRT: abort
PC=0xd1ad3 m=0

goroutine 0 [idle]:
runtime.mach_semaphore_timedwait(0x24400000f03, 0xc43a3c80aa, 0x7fff5fbff554, 0x244, 0x3a3c80aa2001c000, 0x632ba0, 0x7fff5fbff588, 0xc8403, 0x8744ee28aa, 0xffffffff, ...)
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/sys_darwin_amd64.s:428 +0x13
runtime.semasleep1(0x8744ee28aa, 0xffffffff)
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/os_darwin.go:424 +0xda
runtime.semasleep.func1()
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/os_darwin.go:451 +0x33
runtime.systemstack(0x632b00)
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/asm_amd64.s:298 +0x79
runtime.mstart()
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/proc.go:1079

goroutine 40 [syscall]:
runtime.notetsleepg(0x6321d8, 0x8bb274da60, 0x1)
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/lock_sema.go:257 +0x4b fp=0xc420027f40 sp=0xc420027f00
runtime.timerproc()
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/time.go:209 +0x2eb fp=0xc420027fc0 sp=0xc420027f40
runtime.goexit()
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420027fc8 sp=0xc420027fc0
created by runtime.addtimerLocked
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/time.go:116 +0xee

goroutine 1 [semacquire, 1 minutes]:
sync.runtime_Semacquire(0xc42021bebc)
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc42021beb0)
    /usr/local/Cellar/go/1.7.5/libexec/src/sync/waitgroup.go:131 +0x97
main.(*builder).do(0xc420054b60, 0xc4202fa5b0)
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/build.go:1330 +0x4e1
main.runTest(0x62c0a0, 0xc42000c2a0, 0x2, 0x2)
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/test.go:618 +0x1328
main.main()
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/main.go:181 +0x624

goroutine 17 [syscall, 1 minutes, locked to thread]:
runtime.goexit()
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/asm_amd64.s:2086 +0x1

goroutine 5 [syscall, 1 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
    /usr/local/Cellar/go/1.7.5/libexec/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
    /usr/local/Cellar/go/1.7.5/libexec/src/os/signal/signal_unix.go:28 +0x41

goroutine 41 [select, locked to thread]:
runtime.gopark(0x494e08, 0x0, 0x44b969, 0x6, 0x18, 0x2)
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/proc.go:259 +0x13a
runtime.selectgoImpl(0xc420026730, 0x0, 0x18)
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/select.go:423 +0x1235
runtime.selectgo(0xc420026730)
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/signal1_unix.go:304 +0x2d1
runtime.goexit()
    /usr/local/Cellar/go/1.7.5/libexec/src/runtime/asm_amd64.s:2086 +0x1

goroutine 8 [select]:
main.(*builder).runTest(0xc420054b60, 0xc42038c4e0, 0x3, 0xc420491e70)
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/test.go:1145 +0x113a
main.(*builder).do.func1(0xc42038c4e0)
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/build.go:1264 +0x85
main.(*builder).do.func2(0xc42021beb0, 0xc420054b60, 0xc4201dd1e0)
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/build.go:1321 +0x147
created by main.(*builder).do
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/build.go:1327 +0x4b8

goroutine 7 [select]:
main.(*builder).runTest(0xc420054b60, 0xc4202869c0, 0x5, 0xc420495e70)
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/test.go:1145 +0x113a
main.(*builder).do.func1(0xc4202869c0)
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/build.go:1264 +0x85
main.(*builder).do.func2(0xc42021beb0, 0xc420054b60, 0xc4201dd1e0)
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/build.go:1321 +0x147
created by main.(*builder).do
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/build.go:1327 +0x4b8

goroutine 42 [chan receive]:
main.processSignals.func1(0xc420361500)
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/signal.go:21 +0x40
created by main.processSignals
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/signal.go:23 +0x93

goroutine 114 [syscall]:
syscall.Syscall(0x3, 0x4, 0xc420047b55, 0x2ab, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.7.5/libexec/src/syscall/asm_darwin_amd64.s:16 +0x5
syscall.read(0x4, 0xc420047b55, 0x2ab, 0x2ab, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.7.5/libexec/src/syscall/zsyscall_darwin_amd64.go:973 +0x55
syscall.Read(0x4, 0xc420047b55, 0x2ab, 0x2ab, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.7.5/libexec/src/syscall/syscall_unix.go:161 +0x49
os.(*File).read(0xc42012c888, 0xc420047b55, 0x2ab, 0x2ab, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.7.5/libexec/src/os/file_unix.go:228 +0x6a
os.(*File).Read(0xc42012c888, 0xc420047b55, 0x2ab, 0x2ab, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.7.5/libexec/src/os/file.go:101 +0x59
bytes.(*Buffer).ReadFrom(0xc4200a2d20, 0x60aba0, 0xc42012c888, 0xc420354688, 0xc420344201, 0x86866)
    /usr/local/Cellar/go/1.7.5/libexec/src/bytes/buffer.go:176 +0x155
io.copyBuffer(0x609b20, 0xc4200a2d20, 0x60aba0, 0xc42012c888, 0x0, 0x0, 0x0, 0xc420344298, 0x0, 0x0)
    /usr/local/Cellar/go/1.7.5/libexec/src/io/io.go:384 +0x323
io.Copy(0x609b20, 0xc4200a2d20, 0x60aba0, 0xc42012c888, 0x3bf2a0, 0xc420344240, 0xc420354790)
    /usr/local/Cellar/go/1.7.5/libexec/src/io/io.go:360 +0x68
os/exec.(*Cmd).writerDescriptor.func1(0x3bf2a0, 0xc420344240)
    /usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:253 +0x4d
os/exec.(*Cmd).Start.func1(0xc4202b78c0, 0xc420220880)
    /usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:370 +0x27
created by os/exec.(*Cmd).Start
    /usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:371 +0x4db

goroutine 115 [chan receive]:
os/exec.(*Cmd).Wait(0xc4202b78c0, 0xc4202e6540, 0xc420355790)
    /usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:442 +0x124
main.(*builder).runTest.func1(0xc420428fc0, 0xc4202b78c0)
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/test.go:1142 +0x2b
created by main.(*builder).runTest
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/test.go:1143 +0xc76

goroutine 134 [chan receive]:
os/exec.(*Cmd).Wait(0xc4201f0f20, 0xc420429260, 0xc420355f90)
    /usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:442 +0x124
main.(*builder).runTest.func1(0xc4203449c0, 0xc4201f0f20)
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/test.go:1142 +0x2b
created by main.(*builder).runTest
    /usr/local/Cellar/go/1.7.5/libexec/src/cmd/go/test.go:1143 +0xc76

goroutine 133 [syscall]:
syscall.Syscall(0x3, 0x5, 0xc4205528e8, 0x518, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.7.5/libexec/src/syscall/asm_darwin_amd64.s:16 +0x5
syscall.read(0x5, 0xc4205528e8, 0x518, 0x518, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.7.5/libexec/src/syscall/zsyscall_darwin_amd64.go:973 +0x55
syscall.Read(0x5, 0xc4205528e8, 0x518, 0x518, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.7.5/libexec/src/syscall/syscall_unix.go:161 +0x49
os.(*File).read(0xc420028a90, 0xc4205528e8, 0x518, 0x518, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.7.5/libexec/src/os/file_unix.go:228 +0x6a
os.(*File).Read(0xc420028a90, 0xc4205528e8, 0x518, 0x518, 0x1, 0x0, 0x0)
    /usr/local/Cellar/go/1.7.5/libexec/src/os/file.go:101 +0x59
bytes.(*Buffer).ReadFrom(0xc4201d4460, 0x60aba0, 0xc420028a90, 0xc42034ee88, 0xc420429601, 0x86866)
    /usr/local/Cellar/go/1.7.5/libexec/src/bytes/buffer.go:176 +0x155
io.copyBuffer(0x609b20, 0xc4201d4460, 0x60aba0, 0xc420028a90, 0x0, 0x0, 0x0, 0xc420429678, 0x0, 0x0)
    /usr/local/Cellar/go/1.7.5/libexec/src/io/io.go:384 +0x323
io.Copy(0x609b20, 0xc4201d4460, 0x60aba0, 0xc420028a90, 0x3bf2a0, 0xc420429620, 0xc42034ef90)
    /usr/local/Cellar/go/1.7.5/libexec/src/io/io.go:360 +0x68
os/exec.(*Cmd).writerDescriptor.func1(0x3bf2a0, 0xc420429620)
    /usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:253 +0x4d
os/exec.(*Cmd).Start.func1(0xc4201f0f20, 0xc4202915e0)
    /usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:370 +0x27
created by os/exec.(*Cmd).Start
    /usr/local/Cellar/go/1.7.5/libexec/src/os/exec/exec.go:371 +0x4db

rax    0xe
rbx    0x3a3c80aa
rcx    0x7fff5fbff528
rdx    0x3a3c80aa
rdi    0xf03
rsi    0x244
rbp    0x7fff5fbff560
rsp    0x7fff5fbff528
r8     0xffffffffffffffff
r9     0xc552811
r10    0x19a3b5582f0f
r11    0x202
r12    0x23933ae9deead
r13    0x239339e1f6fc8
r14    0x14a3e0cc4b14cc00
r15    0x0
rip    0xd1ad3
rflags 0x202
cs     0x7
fs     0x0
gs     0x0

junkblocker commented 7 years ago

@bits I think I have adapted/fixed everything for OSX except

@rfjakob , the fuse mount issue as in https://github.com/rfjakob/gocryptfs/issues/15#issuecomment-264280442 is not fixed.

rfjakob commented 7 years ago

True, it's that I'm not sure about this. I also don't call "modprobe fuse" on Linux if the kernel module is not loaded. That the path to the loader changes across osxfuse versions is also pretty scary.

spaghetti2514 commented 7 years ago

skip openssl build

gocryptfs v1.2-59-g7fbe69b without_openssl; go-fuse 0ad840c; 2017-02-17 go1.8
# github.com/rfjakob/gocryptfs/internal/stupidgcm
internal/stupidgcm/locking.go:11:10: fatal error: 'openssl/evp.h' file not found
#include <openssl/evp.h>
         ^
1 error generated.
FAIL    github.com/rfjakob/gocryptfs [build failed]
# github.com/rfjakob/gocryptfs/internal/stupidgcm
internal/stupidgcm/locking.go:11:10: fatal error: 'openssl/evp.h' file not found
#include <openssl/evp.h>
         ^
1 error generated.

I'm not entirely sure why this is happening when the source build.bash line isn't running.

aglyzov commented 7 years ago

Here is how I install gocryptfs on OSX Sierra 10.12.3 + homebrew:

CGO_CFLAGS="-I/usr/local/opt/openssl/include" CGO_LDFLAGS="-L/usr/local/opt/openssl/lib" go get -v github.com/rfjakob/gocryptfs

Of course I also did brew install openssl beforehand. Just my 2 cents. Perhaps someone's gonna use it.

aglyzov commented 7 years ago

And here is an auto-unmounting shell script with which I use it on OSX:

https://gist.github.com/aglyzov/181f079630dcb5a525497c39149e589b

Basically it's a copy of encfssh script with encfs replaced by gocryptfs.

rfjakob commented 7 years ago

@spaghetti2514 The without_openssl build tag was not propagated to the compilation of the tests. I have added the test-without-openssl.bash script that does that. Note that you will get several

gocryptfs has been compiled without openssl support but you are still trying to use openssl

because some tests try to explicitely enable openssl, but at least it allows you to run the test suite at all.

raff commented 7 years ago

@aglyzov building using "go get" gives some warning and suggests to use build.bash so a better option to build on MacOS may be:

CGO_CFLAGS="-I/usr/local/opt/openssl/include"` CGO_LDFLAGS="-L/usr/local/opt/openssl/lib" ./build.bash

or even:

export CGO_CFLAGS="-I/usr/local/opt/openssl/include" CGO_LDFLAGS="-L/usr/local/opt/openssl/lib"
./build.bash
./test.bash

Also, MacOS doesn't have flock used by test.bash but you can install:

brew install discoteq/discoteq/flock

erlandh commented 7 years ago

Just wanted to make sure folks were aware I'd posted a bounty on Bountysource for this issue. It's token ($15) at this stage, because I'm not wealthy, but I'm hoping others will chip in too! Would love to see simple, stable, production-ready mac implementation. Thanks so much for all your work folks!

https://www.bountysource.com/issues/29566952-mac-os-x-support

rfjakob commented 7 years ago

I think Mac OS X support has reached a stage where things mostly work and this generic ticket can be closed. I have updated the status at https://github.com/rfjakob/gocryptfs#platforms to state "Beta-quality" instead of "Experimental". I'd ask Mac OS X users to create separate tickets for issues you hit so we can deal with them one by one.

@erlandh thanks for the bounty, much appreciated. I'll link to the bounty in future Mac OS X bug reports.

fulldecent commented 7 years ago

Please also update documentation at https://nuetzlich.net/gocryptfs/comparison/

Currently OSX in progress [7] can be macOS in beta

rfjakob commented 7 years ago

Done, I've just listed MacOS after Linux

teras commented 6 years ago

Are there any documentation on how to install gocryptfs on OSX? Thank you

alexanderharm commented 6 years ago

I created a formula for brew, see https://github.com/Homebrew/homebrew-core/pull/21786.

I still have an issue with the Homebrew sandbox preventing the test to run successfully. I will comment here once they are solved and the formula is usable.

alexanderharm commented 6 years ago

@teras: I created a Homebrew formula which can only be merged if the load_osxfuse issue is solved upstream.

In order for the installation to work you need to tick the option Allow apps downloaded from: App Store and identified developers under the Security & Privacy-tab in System Preferences!

First install OSXFuse from the website (https://osxfuse.github.io/) or via brew:

brew cask install osxfuse

Afterwards you need to reboot your machine.

Install gocryptfs from a local tap for now:

brew install alexanderharm/homebrew-gocryptfs/gocryptfs

I tested this successfully on my machine (macOS Sierra) and it also seems to run fine on @teras' machine running macOS High Sierra.

@rfjakob: maybe this can be added to the docs once it is merged into homebrew-core?

Edit: Updated installation procedure based on the feedback of @teras.

teras commented 6 years ago

@alexanderharm I got this message when I use the application: 2017-12-19 16-32-50

alexanderharm commented 6 years ago

@teras: When did you get this message? I believe this is related to the installation of osxfuse since Benjamin Fleischer is one of the authors: https://osxfuse.github.io/.

If you feel more comfortable you can also download and install osxfuse from this website and then install only gocryptfs via brew.

teras commented 6 years ago

Everything went smooth up to the time I typed gocryptfs cipher plain (from the demo instructions on the site).

The second time I typed them, no message was displayed, only failing: fuse.NewServer failed: exit status 1 Maybe you should run: /Library/Filesystems/osxfuse.fs/Contents/Resources/load_osxfuse

Now, I tried what you said. Indeed the installer asked me to open Settings - I did so but there was no option to allow anything. Since I am using OSX High Sierra, maybe they changed something there?

alexanderharm commented 6 years ago

@teras: Normally it should work. Did you brew cask uninstall osxfuse before? Do you have the FUSE preference pane?

alexanderharm commented 6 years ago

@teras: Under Security & Privacyyou need to tick the option Allow apps downloaded from: App Store and identified developers. It sounds like you are using App Store only.

teras commented 6 years ago

@alexanderharm Indeed this is the case. It seems to work now. Thank you

alexanderharm commented 6 years ago

As of today there is a Homebrew formula for gocryptfs.

Install brew and type brew install gocryptfs.

rfjakob / gocryptfs

Mac OS X support #15

Build and run gocryptfs on Mac

Build gocryptfs "without openssl" on OS X / mac OS

Initialize an encrypted directory

Unlock and mount encrypted directory and write to it

Unmount and lock

Unlock and remount

Adapt tests and benchmarks for Mac

fusermount → umount

time

stat

cat /proc/mount → mount

syscall.PathMax → 5000

Eliminate variable warnings

Benchmark

Build gocryptfs "with openssl" on OS X / mac OS

Run tests