Feature to opt-in into a case sensitive file-system (osxfs) on a volume mount

itskingori commented 7 years ago

Expected behavior

Mounting a folder from OSX in a non-OSX system shouldn't inherit the case insensitivity. For example ... in an Ubuntu container it's not expected behaviour for it to be case insensitive.

I'm wondering if there's a way to provide a case sensitive mount to the container.

Actual behavior (with steps to reproduce)

Host

On the host (Mac OS X), Foo, FOO, and foo are all the same file because it's case insensitive, which is expected behaviour.

~/Desktop/docker-case-test Kingori$ echo '1 is foo' > foo
~/Desktop/docker-case-test Kingori$ echo '2 is Foo' > Foo
~/Desktop/docker-case-test Kingori$ echo '3 is FOO' > FOO

~/Desktop/docker-case-test Kingori$ ls -la
total 0
drwxr-xr-x   3 Kingori  staff  102 17 Aug 11:33 .
drwx------+ 17 Kingori  staff  578 17 Aug 11:31 ..
-rw-r--r--   1 Kingori  staff    0 17 Aug 11:33 foo

docker-case-test Kingori$ cat Foo foo FOO
3 is Foo
3 is Foo
3 is Foo

Container

Inside the container which is Ubuntu, the mounted volume exhibits the case insensitivy (because it is type "osxfs").

~/Desktop/docker-case-test Kingori$ docker run -v ~/Desktop/docker-case-test/:/root/test-code/ -it ubuntu:12.04.5 /bin/bash

root@d99b92c89706:~/test-code# ls -la Foo foo FOO
-rw-r--r-- 1 root root 9 Aug 17 10:05 FOO
-rw-r--r-- 1 root root 9 Aug 17 10:05 Foo
-rw-r--r-- 1 root root 9 Aug 17 10:05 foo

root@d99b92c89706:~/test-code# ls -la
total 5
drwxr-xr-x 3 root root  102 Aug 17 10:05 .
drwx------ 3 root root 4096 Aug 17 09:47 ..
-rw-r--r-- 1 root root    9 Aug 17 10:05 foo

root@d99b92c89706:~/test-code# cat Foo foo FOO
3 is Foo
3 is Foo
3 is Foo

The problem here is that in the container we would like a filesystem that is case sensitive but osxfs seems to be case-preserving and case-insensitive by default.

Information

Diagnostic ID: 49960DDC-8923-487A-B081-8A1DAE0A010A
Docker for Mac: 1.12.0-beta22 (Build 11222)
macOS: Version 10.11.6 (Build 15G31)
[OK]     docker-cli
[OK]     dns
[OK]     moby-syslog
[OK]     app
[OK]     disk
[OK]     system
[OK]     virtualization
[OK]     osxfs
[OK]     menubar
[OK]     db
[OK]     slirp
[OK]     logs
[OK]     moby-console
[OK]     vmnetd
[OK]     env
[OK]     moby
[OK]     driver.amd64-linux

The container filesystem looks something like this ...

root@d99b92c89706:~/test-code# df -h
Filesystem      Size  Used Avail Use% Mounted on
none             19G  8.5G  8.8G  50% /
tmpfs          1001M     0 1001M   0% /dev
tmpfs          1001M     0 1001M   0% /sys/fs/cgroup
osxfs            59T   45T   14T  77% /root/test-code
/dev/vda1        19G  8.5G  8.8G  50% /etc/resolv.conf
/dev/vda1        19G  8.5G  8.8G  50% /etc/hostname
/dev/vda1        19G  8.5G  8.8G  50% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs          1001M     0 1001M   0% /proc/kcore
tmpfs          1001M     0 1001M   0% /proc/sched_debug

dsheets commented 7 years ago

This is because the file system you are mounting from OS X is case-insensitive -- OS X defaults to HFS+ case-insensitive and Docker for Mac shares exactly this file system into your container. If you'd like case sensitivity, you could reinstall and reformat your OS X root volume to be HFS+ case-sensitive but I do not recommend this because there exists Mac software which, insanely, expects case-insensitive behavior. Instead, if you need a case-sensitive file system (which, e.g., Linux's git tree requires even on OS X), I recommend either creating a ramdisk and formatting it as case-sensitive or using a free partition or external disk that is formatted as HFS+ case-sensitive.

I'll add a note to the docs about this issue as other users have recently raised similar questions.

itskingori commented 7 years ago

Interesting link there. We experienced this problem first hand with python imports too.

Instead, if you need a case-sensitive file system (which, e.g., Linux's git tree requires even on OS X), I recommend either creating a ramdisk and formatting it as case-sensitive or using a free partition or external disk that is formatted as HFS+ case-sensitive.

Yes, I understand the problem. Just not sure we have that option. At our scale, asking every developer to repartition isn't tenable.

Feel free to close.

dsheets commented 7 years ago

We've considered shipping tools to help users set up case-sensitive disk image volumes. Perhaps a script which creates a sparse bundle disk image, formats it as HFS+ case-sensitive, and mounts it somewhere would work for you?

itskingori commented 7 years ago

Perhaps a script which creates a sparse bundle disk image, formats it as HFS+ case-sensitive, and mounts it somewhere would work for you?

Yes, I believe any solution that provides a way to have case sensitive file system in the container is a solution for us. Without of course, taking away from how Docker for Mac currently works because it works great as it is.

We've considered shipping tools to help users set up case-sensitive disk image volumes.

That would be great and I think it's much needed as it is a problem that will probably recur.

Just to expand on the use-case a little more:

The reality is a lot of the images people run on Docker are Linux based images ... and development machines (at least for Docker for Mac) are running OS X. The default exception (I surmise) would be that the Linux container will behave Linux-y. In that regard, osxfs behaviour is unexpected (and probably not what will be on production).
In projects that add their files into the image during the build, this isn't a problem. This is the case with apps being packaged to run in production by being copied into the image.
Conversely (to no. 2), we are trying to use Docker for Mac for local development. Volume mounts are an integral component to getting this working right ... i.e. allows us to mount our code on the host into the container and edit files in the dev's favourite editor ... then have changes reflect in the container realtime.

itskingori commented 7 years ago

@dsheets I've investigated the sparse bundle solution. While this is a plausible workaround it introduces complexity to the setup that nullify any gains of using Docker for Mac (at least in our case). One would have to move to a sparse bundle any folder that they want to volume-mount.

As mentioned in my previous comment, we are trying to use Docker for Mac for local development. A developer would therefore have to create the bundle, mount it and move all their existing code/repositories into the mounted volume (assuming that the developer has their code in one place) ... possibly change folder path in the Docker command or on Docker Compose ... etc etc. 😰 ... to take advantage of the fact that osxfs inherits the behaviour of the underlying file-system.

If I understand this correctly ...

osxfs is a new shared file system solution, exclusive to Docker for Mac. osxfs provides a close-to-native user experience for bind mounting OS X file system trees into Docker containers.

... it seems osxfs is an abstraction of the host filesystem. Maybe it's defaults can't be changed? I believe that osxfs default should be to inherit the case-sensitivity behaviour of the container OS and not the host OS. But admittedly, this might be easier said than done. 😁

I appreciate the time you've taken to explain what the problem is and suggesting a possible workaround.

dsheets commented 7 years ago

Docker for Mac attempts to make it appear as though Docker is running directly on OS X. This is, in general, not possible. osxfs is the component of Docker for Mac that is supposed to make OS X directories appear inside of containers. There is no way to provide both normal OS X and Linux views of the same file system hierarchy with the OS X file system retaining its case insensitivity -- how would foo and FOO be stored separately?

I think the best solution is to package a utility that helps users manage their case-sensitive FS volumes if necessary. Yes, this would require changing the local FS structure but, hopefully, that is not onerous because development repositories use relative paths internally. This would be a one-time setup cost that would then continue to operate the same in both OS X and Linux (case-sensitively).

I've updated the osxfs docs and I'll increase the priority of shipping a case-sensitive helper utility. I'm curious about your use case. How do you use case-sensitivity in a way that makes development difficult? I can imagine not getting compile/build errors due to case mistakes which would be unfortunate. Are there other, worse consequences of case-insensitivity that you see?

itskingori commented 7 years ago

There is no way to provide both normal OS X and Linux views of the same file system hierarchy with the OS X file system retaining its case insensitivity -- how would foo and FOO be stored separately?

Well, that is true!!! 🤔 That said, I like the update to the docs.

How do you use case-sensitivity in a way that makes development difficult?

The only issue I've experienced is with the Python import system. See this link for details on how Python handles file imports in case-insensitive systems.

Despite that the filesystem is case-insensitive, Python insists on a case-sensitive match. But not in the way the upper left box works: if you have two files, FiLe.py and file.py on sys.path, and do
  import file
then if Python finds FiLe.py first, it raises a NameError. It does not go on to find file.py; indeed, it's impossible to import any but the first case-insensitive match on sys.path, and then only if case matches exactly in the first case-insensitive match.
An ugly exception: if the first case-insensitive match on sys.path is for a file whose name is entirely in upper case (FILE.PY or FILE.PYC or FILE.PYO), then the import silently grabs that, no matter what mixture of case was used in the import statement. This is apparently to cater to miserable old filesystems that really fit in the lower right box. But this exception is unique to Windows, for reasons that may or may not exist.

The TL;DR there is that Python adapts it's behaviour depending on the OS within which it is running.

In our case, with the osxfs mount, Python's import system adapted to case sensitivity mode because it's in a Linux container but tried to access files on a file system that didn't care about case sensitivity. This resulted in a circular dependency in a certain library which had a package and an import statement that had the same name but different case. So it would try access file.py, then FILE.py which are really the same files (but it doesn't think so) ... and these files had imports to FILE package.

For now I've solved it by changing the library's module name. And nothing else seems to break so far. Which is good.

Documenting the behaviour as you've done is probably the best step here. Then going forward whoever experiences this issue can at least understand the limitations (at least for Python) and adapt if possible.

I can't tell right now where else it might break. For now everything's working well ... including the mounting and updating files in the mount in an editor X. We'll just see how it goes when I get the entire team using Docker for Mac for local development as opposed to just packaging for deployment (which worked).

akougblenou commented 7 years ago

Hello I am facing a similar issue on Mac. I have a docker image running a debian machine on my mac with a project that is siymlinked from the image to some mac repositories with a Python project. @itskingori how did you end up solving the issue with Python ?

itskingori commented 7 years ago

how did you end up solving the issue with Python ?

@AKFourSeven renamed our python code to avoid conflicts.

akougblenou commented 7 years ago

Oh I see, one of the few things I cannot do...

dsheets commented 7 years ago

You can do something like:

hdiutil create -type SPARSE -fs hfsx -size 4g -volname linux linux.hfsx.dmg.sparseimage
hdiutil attach -nobrowse -mountpoint linux linux.hfsx.dmg.sparseimage

and then later hdiutil eject linux.

I use a case sensitive mount like this when I need to browse or manipulate the Linux source tree on my Mac. Suggestions for how best to incorporate this macOS functionality into Docker for Mac are very welcome.

hughsw commented 7 years ago

We have the same need. For development work we are running MariaDB in a Docker container on the Mac, with osxfs mounts for the database file store. We have *.sql files that we need to load that were dumped from a case sensitive system, and they actually have some separate, case-sensitive names for tables that they create, e.g. Projects and projects (yes, MariaDB supports case-sensitive table names; field names are case-insensitive). When the .sql file is loaded, the later table-creating query for projects wipes out the table created earlier for Projects...":

DROP TABLE IF EXISTS `projects`;
CREATE TABLE `projects`   (  ...  );

docker-robott commented 6 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

mph2labs commented 6 years ago

same need here... wasted a lot of time as a result of this...

YRM64 commented 6 years ago

Many thanks go out to dsheets for his comments on creating a ramdisk and reformatting it as case sensitive, or using a free partition or external disk that is reformatted as HFS and case sensitive. I even like the idea of packaging a utility that helps manage case-sensitive FS volumes.

I am also grateful for itskingori's comments on shipping tools to help users set up case sensitive disk image volumes, and changing a library's module name. AK47, thank you for your comments on renaming python code to avoid conflicts. All food for thought. Fine job gentlemen.

rmg commented 5 years ago

Since my laptop is using APFS, I went slightly fancier than a simple disk image and created a volume instead.

In case anyone else finds it useful:

rm -rf $HOME/.docker/Volumes
mkdir $HOME/.docker/Volumes
sudo diskutil apfs addVolume disk1 "Case-sensitive APFS" docker-volumes -mountpoint $HOME/.docker/Volumes -reserve 5g
sudo chown -R $(id -u):$(id -g) $HOME/.docker/Volumes

The ~/.docker/Volumes path is where kubernetes puts PVCs, and that's what was giving me all the trouble.

thaJeztah commented 4 years ago

/cc @djs55 @nebuk89

I think ideally we should switch the default to be "case-sensitive" (for bind-mounts it would obviously not be fully case-sensitive unless the macOS side is case-sensitive), and make "case insensitive" opt-in. When developing for a Linux container, the container should match the expectation of a Linux filesystem being case-sensitive.

Having a case-insensitive bind-mount can easily lead to bugs introduced in ones codebase that don't reproduce in the "dev" situation ("it worked on my machine"), but do kick in once deployed to production / Linux environment with a case-sensitive filesystem.

stephen-turner commented 4 years ago

I'm still not sure what @thaJeztah's proposal means if it's backed by a case insensitive file system on the host. What if you create two files whose names differ only in case, which would be normal on Linux, but can't be represented in the underlying file system?

thaJeztah commented 4 years ago

@stephen-turner sure, fully case-sensitive won't be possible, so it would not be possible to have two files or directories that only differ in case. I think that situation (same name, different casing) is a corner case. However, we should try to mimic the Linux / case-sensitive situation as much as possible; perhaps "case-matching" would be a better description?

An example to illustrate;

mkdir testdir
echo "hello-world" > ./testdir/some_file

Currently, this does not produce an error, even though there's no file named SOME_FILE (all caps):

docker run --rm -v $(pwd)/testdir:/testdir busybox cat /testdir/SOME_FILE
hello-world

What I'd expect to see instead:

docker run --rm -v $(pwd)/testdir:/testdir busybox cat /testdir/SOME_FILE
cat: can't open '/testdir/SOME_FILE': No such file or directory

stephen-turner commented 4 years ago

Well, all the difficulty is in your phrase "as much as possible". Somewhere you have to draw the line, and wherever you draw it there are confusing behaviours and corner cases. For example, remember that people are generally using the same files on the host side too (otherwise they would probably have just used the container's file system not a mounted volume), and having cat /testdir/SOME_FILE have different behaviour depending whether you're in the Mac shell or the container's shell is also weird in a different way. Unfortunately there is really no way to hide the complexity.

thaJeztah commented 4 years ago

having cat /testdir/SOME_FILE have different behaviour depending whether you're in the Mac shell or the container's shell is also weird in a different way.

Well, that's exactly the situation to address: when developing a Linux container, the container should behave as a Linux platform; macOS very likely is not the platform that the container / image will be deployed on.

A very common scenario for development is:

during development, bind-mount source into the container (especially for scripted languages (php / JavaScript / html); doing so allows you to "live" code in your local IDE, without having to re-build an image on every change
the developer can verify changes in their browser
after changes are complete; those changes are committed to source control, and a new version of the image (including the source code in the image) is built and deployed

That situation is problematic currently, because the development scenario does not match the production situation;

If a mistake was made in referencing a file with the wrong case (reference "image.jpeg" instead of "image.JPEG"), such bugs are not reproducible in the developer situation. Such bugs very easily go unnoticed until it's deployed.

Add to that, that most frameworks (at least php, but likely other languages) are designed for Linux deployment, and won't take case-insensitive situations into account.

I think Docker Desktop should (as a default, but at least as an option) allow these scenarios, as they are very common, and assist in prevention of hard to discover bugs.

djs55 commented 4 years ago

@thaJeztah I think we now have behaviour on Windows which matches your desired behaviour in the comment above:

PS C:\Users\djs> notepad some_file.txt

PS C:\Users\djs> docker run --rm -v C:\Users\djs:/testdir busybox cat /testdir/some_file.txt
hello world

PS C:\Users\djs> docker run --rm -v C:\Users\djs:/testdir busybox cat /testdir/SOME_FILE.txt
cat: can't open '/testdir/SOME_FILE.txt': No such file or directory

In the file lookup code on Windows we insist that the user has used the canonical file case, and return ENOENT otherwise. So we guarantee that if it works in the developer environment it will work in production, however

not everything that would work in production will work in development (e.g. I still can't clone the Linux kernel because of case issues)
there is now a mismatch between the native Windows powershell behaviour (where cat SOME_FILE.txt would work) and a Linux container

Perhaps we could aim to have similar semantics on Mac in future.

thaJeztah commented 4 years ago

In the file lookup code on Windows we insist that the user has used the canonical file case, and return ENOENT otherwise.

Is it (technically) possible to return a custom error message with that? (Thinking out loud) so if we had a match for some_file.txt that we could append "did you mean filename_with_correct_case.txt ?

frob commented 11 months ago

This is also an issue on Mac.

markedwards commented 3 months ago

This seems to have two main side effects:

Cannot have "foo.txt" and "Foo.txt" in the same directory.
Cannot rename "foo.txt" to "Foo.txt", (without first renaming it to "foo_temp.txt").

The first limitation seems reasonable, but the second is really silly and annoying, especially since the error raised is incomprehensible until you find this thread.

lucianthorr commented 2 months ago

I'd like to add onto this request. It seems that docker-in-docker is case sensitive even if hosted on Mac so I have a situation where local development builds fine but CI, which runs as docker-in-docker fails on casing issues. If we could enforce this case-sensitivity at the local level, it would be a great way to catch this.

docker / for-mac