search

Azure / azure-storage-fuse

A virtual file system adapter for Azure Blob storage
Other
660 stars 206 forks source link

panic creating temp dir for logs when running blobfuse2 non-interactively as a user without a $HOME #1036

Closed iMHLv2 closed 1 year ago

iMHLv2 commented 1 year ago

Which version of blobfuse was used?

# apt-cache policy blobfuse2
blobfuse2:
  Installed: 2.0.1
  Candidate: 2.0.1
  Version table:
 *** 2.0.1 500
        500 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 Packages
        100 /var/lib/dpkg/status

Which OS distribution and version are you using?

# cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

If relevant, please share your mount command.

Not relevant (yet). Ultimately we want to mount things, but just getting the command to run with --help in this scenario would suffice.

What was the issue encountered?

For some use cases, we'd like to use blobfuse2 non-interactively - for example from a systemd service running under a service account. In some cases, the service account may not have a home directory. Steps to reproduce:

# groupadd test
# useradd -g test --no-create-home -s /bin/false --home-dir /does-not-exist test

The next command merely simulates the problem, but in a slightly different way. In the systemd case, $HOME will be set to /does-not-exist. In the example case, it will be set to /root since I'm invoking via sudo while logged in as root. Either way, the folder either does not exist or it exists and isn't writable by the test user. 

# sudo -u test blobfuse2 --help
panic: Unable to create blobfuse2 temp directory for logs

goroutine 1 [running]:
github.com/Azure/azure-storage-fuse/v2/common/log.init.0()
    /mnt/vss/_work/1/s/azure-storage-fuse/common/log/logger.go:233 +0x1ab

That all makes sense so far. The confusing part occurs when trying to redirect the directory. The REAMDE.md mentions:

--default-working-dir: The default working directory to store log files and other blobfuse2 related information.

That didn't work:

# sudo -u test blobfuse2 --default-working-dir /tmp
panic: Unable to create blobfuse2 temp directory for logs

goroutine 1 [running]:
github.com/Azure/azure-storage-fuse/v2/common/log.init.0()
    /mnt/vss/_work/1/s/azure-storage-fuse/common/log/logger.go:233 +0x1ab

The --default-working-dir also doesn't seem to be a recognized option when run as root:

# sudo blobfuse2 --default-working-dir /tmp
unknown flag: --default-working-dir

There is a similar looking option called --log-file-path which looks like this:

# sudo blobfuse2 mount --help | grep log-file
      --log-file-path string         Configures the path for log files. Default is $HOME/.blobfuse2/blobfuse2.log (default "$HOME/.blobfuse2/blobfuse2.log")

However that doesn't work either:

# sudo -u test blobfuse2 --log-file-path /tmp/log.txt
panic: Unable to create blobfuse2 temp directory for logs

goroutine 1 [running]:
github.com/Azure/azure-storage-fuse/v2/common/log.init.0()
    /mnt/vss/_work/1/s/azure-storage-fuse/common/log/logger.go:233 +0x1ab

The tool seems to still be using the default of $HOME/.blobfuse2 which is /root/.blobfuse2 in the example. 

# strace -f sudo -u test blobfuse2 --log-file-path /tmp/log.txt
...
[pid  1089] newfstatat(AT_FDCWD, "/root/.blobfuse2",  <unfinished ...>
[pid  1090] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid  1089] <... newfstatat resumed> 0xc0000a5218, 0) = -1 EACCES (Permission denied)
[pid  1089] newfstatat(AT_FDCWD, "/root", {st_mode=S_IFDIR|0700, st_size=4096, ...}, 0) = 0
[pid  1090] <... nanosleep resumed> NULL) = 0
[pid  1089] mkdirat(AT_FDCWD, "/root/.blobfuse2", 0777 <unfinished ...>
[pid  1090] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid  1089] <... mkdirat resumed> )     = -1 EACCES (Permission denied)
[pid  1089] newfstatat(AT_FDCWD, "/root/.blobfuse2", 0xc0000a53b8, AT_SYMLINK_NOFOLLOW) = -1 EACCES (Permission denied)
...

Have you found a mitigation/solution?

We can obviously create a home directory for the service account with the proper permissions, but that wasn't necessary for blobfuse v1. If possible, we'd like to avoid creating the directory for this tool's use exclusively.

Please share logs if available.

See above, thank you!

vibhansa-msft commented 1 year ago

$HOME/.blobfuse2 is a directory that blobfuse2 creates to store of the house keeping data. Default log path in case of base logging is also the same. Base issue here is user is not having any home directory set which is leading to failure in creation of. blobfuse2 directory under home. For now, having a home directory is the workaround, till we fix this issue from our end. Amy particular reason you have to not create home directory for the given user.

vibhansa-msft commented 1 year ago

We looked back at the codebase and there are other house-keeping things put in $HOME/.blobfuse2 directory. This also includes a pid based lock file and few initial logs. As this thing is initialized even before the cli parsing begins, default-working-dir that you are trying to set is of no-use. If the user mounting blobfuse2 has access to the $HOME path blobfuse2 will be able to create the directory and proceed. In case when $HOME is not set, $HOME returns back "/" which means user shall have permission to create a directory /.blobfuse2 in your case to run successfully. I see the easier way out here is to have a home directory for the user who is going to mount blobfuse2. Changing the code-flow in this regard will be a breaking change from our side (as existing behavior will change) which we do not intend to do unless there is a reasonable use-case affecting multiple users.

iMHLv2 commented 1 year ago

@vibhansa-msft Thanks for the reply, and I agree with your diagnosis (non-existent or non-writable $HOME). To explain a little more about our use case, we have a set of REST APIs for users to request mounting or unmounting blobs. The client is a web browser or an automated platform such as a SOAR or EDR. The APIs communicate to a systemd service (via database) which performs the operations by invoking the blobfuse tool. For security "defense-in-depth" the APIs and systemd service run as a non-root, neutered user account (i.e., no password set, no shell, no home directory).

As previously mentioned, this interaction worked fine with blobfuse v1 and it is also compatible with aws s3 and GCP, etc. In several years, we haven't run into any other actions/tools used or invoked by the APIs or systemd service that required $HOME to exist for this service account, so there has never been a reason to create the directory.

Here are a few arguments (the friendly kind!) to consider regarding making a change or not:

At the very least, I'd recommend updating the documentation to say that $HOME must exist and be writable for blobfuse v2 to work. It might also be good to modify the --help output for --default-working-dir and --log-file-path to mention that those are less defaults (i.e., "Default is $HOME/.blobfuse2/blobfuse2.log") and more static and unconditional.

Thanks for considering the change!

vibhansa-msft commented 1 year ago

Thanks for your suggestions and inputs. I will look into it and see if we can fix it up in next release. If change has some risks, we will go ahead and update the document saying $HOME is must with v2.

vibhansa-msft commented 1 year ago

I have a potential fix for the problem. Will it be possible for you to check-out the branch and build locally?

brosahay commented 1 year ago

I have a potential fix for the problem. Will it be possible for you to check-out the branch and build locally?

I can help

vibhansa-msft commented 1 year ago

@brosahay : you can try on this branch: https://github.com/Azure/azure-storage-fuse/tree/vibhansa/v2/default_work_dir_fix

dannyfritz3 commented 1 year ago

@vibhansa-msft Sorry for the late response. I was able to build your branch locally and it worked using the --default-working-dir flag now with the $HOME directory not present.

vibhansa-msft commented 1 year ago

@dannyfritz3 : Great so my fix works well it seems for your use case. Will try to get this change in for the next release.

dannyfritz3 commented 1 year ago

@vibhansa-msft Thank you for the swift assistance!

scottporter commented 1 year ago

We have been use blobfuse to actually bring in a home directory that we were storing on a blob storage container. This requirement of blobfuse2 to store files in the home directory obviously breaks what we were doing, keeping us from upgrading. Looking forward to examining the fix.

vibhansa-msft commented 1 year ago

@scottporter : there is still time before this gets officially released. Change is merged into main branch. Can you checkout main branch build locally and try this change out.

scottporter commented 1 year ago

@vibhansa-msft I tried building, but got this error:

build github.com/Azure/azure-storage-fuse/v2: cannot load io/fs: malformed module path "io/fs": missing dot in first path element

scottporter commented 1 year ago

Is there a binary with the fix I could check? I wasn't able to build on our infrastructure, but still willing to test the fix if there is an easy way to do that now.

vibhansa-msft commented 1 year ago

There is no need to build now, latest 2.0.2 release has the fix. Kindly upgrade and try again.