Mounting an SMB share inside a sysbox container

[arp-mbender]

I'm tasked with giving our DevOps agents (which use sysbox so they can do "Docker IN Docker") access to our Windows network share, and I've got some issues.

a) I cannot mount the share from inside the container, as Docker itself doesn't permit that (would need to add --privileged but that seems like it defeats the purpose of using sysbox in the first place) b) I cannot use smbclient because I'd probably need to expose the domain passwords for some kind of user to the container c) I've tried mounting the share to the Docker host and then sharing the mounted folder, only to discover that won't work either (note that sysbox has been declared as the default runtime):

docker run -v /mnt/storage:/storage --rm -it --entrypoint bash my_devops_image
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:425: starting container process caused: process_linux.go:607: container init caused: process_linux.go:578: handleReqOp caused: rootfs_init_linux.go:322: ID-map mount on storage caused: rootfs_init_linux.go:316: setting up ID-mapped mount on path /var/lib/sysbox/rootfs/660dc228a6a90bd401263902db0e1ff7da7a2cd106325d895d947bab97b6e9ff/top/merged/storage (likely means idmapped mounts are not supported on the filesystem at this path (unknown fs)) caused: Failed to set mount attr: invalid argument: unknown.

Googling this suggests that sysbox simply cannot handle such a volume. Are there any other options open to me?

[ctalledo]

Hi @arp-mbender, thanks for trying Sysbox.

So that I can repro locally, how did you create the /mnt/storage network share? Can you do a findmnt at host-level so I can see the mountpoint info?

From the error you posted, I can see that sysbox is trying to ID-map-mount on top of /mnt/storage, but the operation is failing (likely because the underlying filesystem does not yet support ID-map-mounts).

[arp-mbender]

It's a CIFS mount, which I'm guessing is unsupported (created via mount -t cifs). Honestly, I don't think the bug tag is appropriate here - it's more of a question, i.e. "is there some way to do what I'm trying to do?"...

[ctalledo]

Hi @arp-mbender, there may be way to make this work. Check this prior Sysbox issue related to CIFS mounts.

Although CIFS does not yet support ID-mapped-mounts (and the alternative shiftfs does not work on them either), I think you could work around by passing the uid=165536,gid=165536 options to the mount -t cifs command, with the assumption is that Sysbox assigned that same uid:gid to the container. You can get the assigned uid by doing cat /proc/self/uid_map inside the sysbox container.

By passing that option to the mount -t cifs command, I believe Sysbox won't mount ID-mapped-mounts or shiftfs on it, and things should work.

Hope that helps.

P.S. You are correct, I'll change the label for this issue from bug to question.

[arp-mbender]

Brilliant! It works. Thank you very much. May I suggest adding this to the documentation (if it's not already there; if it WAS already there then I apologize sincerely for not having found it myself)? ;)

[ctalledo]

Hi @arp-mbender, glad it worked. We didn't document it because it's a work-around that relies on the user knowing the uid:gid that Sysbox will assign to the container. Ideally ID-mapped-mounts would work with CIFS such that the work-around is not required. That should happen in the near future as different filesystems in the Linux kernel add support for ID-mapped-mounts.

Let's have this issue serve as documentation (together with issue #186).

Closing this one, please re-open if you feel things are not working. Thanks!

[arp-mbender]

@ctalledo I spoke too soon. I've initially mounted the share as you suggested, but I did a poor job of testing this connection - I've merely opened a container with the /mtn/storage, and did ls on it which produced some satisfactory results (i.e. list of shared folders).

However, actually navigating to any of the folders on the network share produces the following result:

root@fefd81d48b0c:/storage# cd Deployment/
bash: cd: Deployment/: Invalid argument

I've previously failed to actually do this simple navigation check. :( For the record the host is able to navigate the share correctly.

Any idea what might be going on?

On a side note; I've noticed that the Docker host, when running containers with sysbox and then rebooted seems to have some issues: Are you aware of this potential problem? This definitely happens only with sysbox containers. Normal containers don't seem to have this problem.

[ctalledo]

Hi @arp-mbender, strange that you that basic failure. Back in issue #186, I recall (vaguely) that things worked well inside the sysbox container with the cifs mount (after using the uid/gid cifs mount option).

I'll work on reproducing this during the week. I need to refresh my mind on how to setup a cifs mount.

Regarding the "clearing orphaned inode" messages, I don't recall seeing them before. It's a separate issue, but can you do a findmnt inside the sysbox container to see if I can spot any issues? Thanks.

(Re-opening issue per prior comment).

[arp-mbender]

Perhaps worth mentioning is that I was using sysbox on Ubuntu Server (22.04.1 LTS) which, AFAIK, has some issues with shiftfs, so I've changed the config to have shiftfs disabled. Could have an impact on either the network share access (definite priority for us) or the orphaned inodes (not a problem ATM; just thought I'd ask if you're aware of such an issue)?

[ctalledo]

Perhaps worth mentioning is that I was using sysbox on Ubuntu Server (22.04.1 LTS) which, AFAIK, has some issues with shiftfs, so I've changed the config to have shiftfs disabled. Could have an impact on either the network share access (definite priority for us) or the orphaned inodes (not a problem ATM; just thought I'd ask if you're aware of such an issue)?

Thanks for the additional context. I was thinking along the same lines, but later realized shiftfs wouldn't affect the CIFS mount because the CIFS mount would not interact with shiftfs in any way (Linux would route all filesystem accesses on the CIFS mount to the CIFS filesystem, they would never go through shiftfs).

Can you do a findmnt inside the container that has the CIFS mount please, and post the output here? Thanks.

[arp-mbender]

Here's the section related to the network share from findmnt:

|-/storage                                                   //arp.local/Storage$                                                                                                                       cifs     rw,relatime,vers=3.1.1,cache=strict,username=<REDACTED_USERNAME>,uid=165536,noforceuid,gid=165536,noforcegid,addr=10.150.1.17,file_mode=0755,dir_mode=0755,soft,nounix,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1

[ctalledo]

Thanks, don't see anything wrong with the findmnt output; I'll try to repro on my side to see what's going on.

Question: does this occur when mounting the cifs mount to a regular (non-Sysbox) Docker container? In that case you would mount the cifs mount with uid=0,gid=0.

[arp-mbender]

Regular containers don't have this issue and work as expected.

[arp-mbender]

@ctalledo Do you want me to try any more things on my end? I've tried the mount with uid=0,gid=0. When I did that sysbox would fail to launch a container with the mounted storage (initial error described in this issue). A non-sysbox container would function as expected.

[ctalledo]

Hi @arp-mbender,

Do you want me to try any more things on my end?

I can't think of anything else to try on your end right now.

Ball is on my court to try to repro locally, since back when I was working on issue #186, I got CIFS mounts to work fairly well.

I dedicated it a few cycles during the weekend, but got stuck at the more basic step of trying to setup a CIFS mount from my linux dev host to a windows dev host, so that I could then mount it into a sysbox container (on my linux dev host). Any hints on how to do that easily would be welcome.

[arp-mbender]

What exactly did you get stuck on?

A basic shared folder on Windows (assuming non-domain workstation, but the process would likely be the same for a domain-joined system):

• create a new folder which will act as the shared folder; possibly add some files and folders for better testing later on
• go to folder properties -> Sharing tab -> Advanced Sharing -> tick the "Share this folder" + go to Permissions and grant the current user write permissions (otherwise the share will be read-only)
• verify the share is working by going to the path \\<computername>\<sharename> on the Windows machine

While not particularly "safe" I also recommend disabling the Windows firewall (since ports and connectivity shouldn't be part of the issue).

Now the share should be accessible from a Linux box via CIFS tools (using cifs-utils -> mount -t cifs). Note it MAY be required to pass the username parameter as .\<username>, to indicate a local Windows username (rather than a domain one).

Other than the above I'm not sure how I can help - I'd need a more specific question related to setting up the shares.

[ctalledo]

Hi @arp-mbender, apologies for the delay in responding, didn't get any cycles last week to look further into this issue.

Thanks for the CIFS mount config steps. I got stuck last week in the last step where I create the CIFS mount on linux and it should connect to the Windows share. For some reason it was not able to find it. Will try again today or tomorrow to get it working so I can repro the issue we are after.