search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.05k stars 961 forks source link

"Case insensitive filesystem can't manage this" #1557

Closed JPGygax68 closed 2 years ago

JPGygax68 commented 7 years ago

I tried to install OpenCV3 as a dependency and found OpenCV/3.2.0@xmar/stable, which I added to my requires section.

There was no package available for my platform (Windows 10, Visual C++ 2015 32-bit), so I went with --build missing. The build seems to have run to completion, but right at the end, I got:

ERROR: Requested 'OpenCV/3.2.0@xmar/stable' but found case incompatible 'opencv'
Case insensitive filesystem can't manage this

(I tried other OpenCV packages, but they all had problems of one sort or another. I think this one is the most promising, if I can find a solution to the above problem.)

memsharded commented 7 years ago

I have just tried a conan install for that dependency, with --build=missing in Windows 10, VS14, and it worked.

Is it possible that you had tried another package recipe, that was still in your local cache? It might happen that you installed some opencv/xxx@user/channel, and that difference in case will cause trouble in Windows, that is why the system might be warning.

Did it fail at the very end? Can you please copy the output lines before that? What is the output of conan search?

Try clearing your cache with conan remove "opencv*", and repeat.

JPGygax68 commented 7 years ago

Strange. I applied your suggestion and now I'm getting a quite different error:

ERROR: OpenCV/3.2.0@xmar/stable: Error in package_info() method, line 217
        libs = libs_opencv_win + libs_3rdparty_win + libs_win
        TypeError: unsupported operand type(s) for +: 'map' and 'map'
memsharded commented 7 years ago

Which conan are you running (installed with pip, the Win installer..)? Pip installer much recommended.

In which python version?

Twoflower2 commented 7 years ago

Any way the find the Conan version from cmd line ie. conan ver? (In know the version.txt in the Conan cache has to version.)

memsharded commented 7 years ago

conan --version

JPGygax68 commented 7 years ago

Sorry for the late reply. My python is version 3.5.3, conan is version 0.25.1 EDIT: I installed conan with pip.

memsharded commented 7 years ago

Ok, I see. It is a issue in the package recipe, which is not compatible with python3. I have proposed a fix as a PR to the original recipe in https://github.com/xmar/conan-opencv/pull/1, but it is simple if you want to fork it and apply it yourself:

libs_opencv_win = [n + self.opencv_version_suffix + debug_suffix for n in libs_opencv]
libs_3rdparty_win = [n + debug_suffix for n in libs_3rdparty]

As this is not a conan issue, but a package recipe issue, I think we can close it here, and keep conversation in https://github.com/xmar/conan-opencv/pull/1 if necessary.

JPGygax68 commented 7 years ago

Thank you!

On Tue, Aug 1, 2017 at 10:41 AM, James notifications@github.com wrote:

Ok, I see. It is a issue in the package recipe, which is not compatible with python3. I have proposed a fix as a PR to the original recipe in xmar/conan-opencv#1 https://github.com/xmar/conan-opencv/pull/1, but it is simple if you want to fork it and apply it yourself:

libs_opencv_win = [n + self.opencv_version_suffix + debug_suffix for n in libs_opencv] libs_3rdparty_win = [n + debug_suffix for n in libs_3rdparty]

As this is not a conan issue, but a package recipe issue, I think we can close it here, and keep conversation in xmar/conan-opencv#1 https://github.com/xmar/conan-opencv/pull/1 if necessary.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/conan-io/conan/issues/1557#issuecomment-319307083, or mute the thread https://github.com/notifications/unsubscribe-auth/AAwiiOnWL2bsI83rdhrLxhg79dPlxuidks5sTuSlgaJpZM4Ons0k .

memsharded commented 7 years ago

Good, closing it now, thanks

RobertHabrich commented 6 years ago

Not sure if it is appropriate to resurrect this issue, but I'm having the exact same error.

We have a company internal package of openssl ("openssl/1.0.2m@user/channel") that I have in my local cache. Now i want to use "Qt/5.11@bincrafters/stable" with option openssl enabled. This adds a self.requires("OpenSSL/1.0.2l@conan/stable") in the recipe of the Qt package. Since the Windows filesystem is not case-sensitive, the "OpenSSL/1.0.2l@conan/stable" package goes into my existing "~.conan/data/openssl" folder and appears as "openssl/1.0.2l@conan/stable" in conan search. The problem is that the next time a recipe requires "OpenSSL/1.0.2l@conan/stable", I get the error from above:

"ERROR: Requested 'OpenSSL/1.0.2l@conan/stable' but found case incompatible 'openssl' Case insensitive filesystem can't manage this"

From a user perspective, this does not seem to be appropriate behavior, because I cannot really control the naming decisions of different package authors. Wouldn't it make sense to ignore the case on case insensitive filesystems?

memsharded commented 6 years ago

Hi @Blubbz0r

The problem with ignoring the case in Windows, is that it would be a totally assimetric behavior than in other OSs. Other OS that are case sensitive (and conan package references are case sensitive too) will have a dependency graph with 2 different dependencies, one will be the "openssl/1.0..." package and the other the "OpenSSL/1.0...." package. They are different nodes in the graph, with different libraries, etc. For sure this can be problematic, is something that typically is not wanted. But still, for some extreme cases, it is possible. Personally, I think maybe it could have been better, but at the time it wasn't an issue, and now it can be changed without breaking. It is something we will consider to change for the next 2.0 release, but that will take some time.

In the meantime, what conan does is to error in Windows, is the best that could be done. I would say that the best workaround right now would be to rename your local OpenSSL package to match the case.

Thanks very much for the feedback!

torfinnberset commented 5 years ago

@memsharded, I just had a similar issue, but on macOS. I had the Catch2 package installed from bincrafters, but they deprecated it in favor of the catchorg remote. However, the two remotes used different capitalization of Catch: catch2 vs Catch2, and then I got the same issue.

How to reproduce:

$ conan install catch2/2.3.0@bincrafters/stable
$ conan install Catch2/2.5.0@catchorg/stable

Full output:

conan install catch2/2.3.0@bincrafters/stable
Configuration:
[settings]
arch=x86_64
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=10.0
os=Macos
[options]
[build_requires]
[env]

catch2/2.3.0@bincrafters/stable: Not found in local cache, looking in remotes...
catch2/2.3.0@bincrafters/stable: Trying with 'conan'...
catch2/2.3.0@bincrafters/stable: Trying with 'conan-community'...
catch2/2.3.0@bincrafters/stable: Trying with 'bincrafters'...
Downloading conanmanifest.txt
[==================================================] 166B/166B        
Downloading conanfile.py
[==================================================] 2.0KB/2.0KB      
Downloading conan_export.tgz
[==================================================] 758B/758B        
Decompressing conan_export.tgz: 100%|██████████| 758/758 [00:00<00:00, 363kB/s]
catch2/2.3.0@bincrafters/stable: Installing package
Requirements
    catch2/2.3.0@bincrafters/stable from 'bincrafters' - Downloaded
Packages
    catch2/2.3.0@bincrafters/stable:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 - Download

catch2/2.3.0@bincrafters/stable: Retrieving package 5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9 from remote 'bincrafters' 
Downloading conanmanifest.txt
[==================================================] 604B/604B        
Downloading conaninfo.txt
[==================================================] 150B/150B        
Downloading conan_package.tgz
[==================================================] 97.6KB/97.6KB    
Decompressing conan_package.tgz: 100%|██████████| 95.3k/95.3k [00:00<00:00, 10.1MB/s]
catch2/2.3.0@bincrafters/stable: Package installed 5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
MacBook-Pro-5:miniz torfinn$ conan install Catch2/2.5.0@catchorg/stable
Configuration:
[settings]
arch=x86_64
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=10.0
os=Macos
[options]
[build_requires]
[env]

Catch2/2.5.0@catchorg/stable: Not found in local cache, looking in remotes...
Catch2/2.5.0@catchorg/stable: Trying with 'conan'...
Catch2/2.5.0@catchorg/stable: Trying with 'conan-community'...
Catch2/2.5.0@catchorg/stable: Trying with 'bincrafters'...
Catch2/2.5.0@catchorg/stable: Trying with 'catchorg'...
Downloading conanmanifest.txt
[==================================================] 1.4KB/1.4KB      
Downloading conanfile.py
[==================================================] 889B/889B        
Downloading conan_export.tgz
[==================================================] 891B/891B        
Decompressing conan_export.tgz: 100%|██████████| 891/891 [00:00<00:00, 494kB/s]
ERROR: Failed requirement 'Catch2/2.5.0@catchorg/stable' from 'PROJECT'
ERROR: Requested 'Catch2/2.5.0@catchorg/stable' but found case incompatible 'catch2'
Case insensitive filesystem can't manage this

I initially reported the issue at https://github.com/catchorg/Catch2/issues/1501

monsdar commented 5 years ago

Having a similar issue with Boost:

>conan download boost/1.69.0@conan/stable -r conan-center

Downloading conanmanifest.txt
[==================================================] 125B/125B
Downloading conanfile.py
[==================================================] 31.6KB/31.6KB
Downloading conan_export.tgz
[==================================================] 602B/602B
Decompressing conan_export.tgz: 610B [00:00, 13.0kB/s]
ERROR: Requested 'boost/1.69.0@conan/stable' but found case incompatible 'Boost'
Case insensitive filesystem can't manage this

When using uppercase I get this:

>conan download Boost/1.69.0@conan/stable -r conan-center
ERROR: 'Boost/1.69.0@conan/stable' not found in remote 'conan-center'

I'm on the following system:

As this seems to be a package issue I'll reference this to the corresponding package as well. Just added this here because it seems to be a reoccuring issue that perhaps needs to be solved (or at least handled) within conan directly. Perhaps it's possible to notify the package maintainer before uploading that cases are mixed...

Update:

Update2: Ok, understood a bit better what's going on. I had a directory Boost in my local conan cache, but tried installing a package which was named boost. Of course this causes issues on Windows when things have the same name all of a sudden. Not sure how to prevent this from Conan, it's not even an issue with the package itself, but with the user using multiple packages spelled differently. Long story short: Removing the Boost directory from my local conan cache solved the issue.

RobertHabrich commented 5 years ago

@memsharded are there any news on this topic? This is currently the number one paint point with using conan for us.

We are working on Windows only (DEV workstations as well as CI). Some time ago we decided to start using packages from conan and bincrafters to avoid having to maintain our own packages (which was another big pain point back then). For example, we started using the Qt package from bincrafters. However, they changed the name of the package to all lowercase recently. For us this basically means that we cannot upgrade anymore if we want to a) keep working on different projects on DEV workstations and b) be able to build older versions.

The only workarounds seem to be a) maintain projects on our own again or b) find ways to cleanly separate each project and their dependencies on all DEV and CI machines. None of these seem to be reasonable alternatives to force on each user.

ahauan4 commented 5 years ago

@memsharded We have the same problem with OpenCV + CMake. Recently we used our company internal modules only. Now we want to use packages from conan-center too but are unable to do so because of different uppercase/lowercase writings. Will this issue be fixed in the near future? Until then we are unable to use above mentioned packages from conan-center.

monsdar commented 5 years ago

I understood that this is an issue because of backwards compatibility. It's essentially a user problem to use mixed case and installations for others could be broken when Conan implements a fix that e.g. switches every package name to lower case internally.

That being said: couldn't this be solved with an opt-in config value? use_lowercase_package_names could be False by default. When switching to True it will make Conan internally convert all package names to lower before using it.

Another idea: check if uppercase is used, if that's the case then convert the package name to lower and suffix it with _from_upper or something similar. So CMake turns into cmake_from_upper. When there is a second package already called cmake this approach will not cause conflicts.

memsharded commented 5 years ago

I agree this is still an open pain, more challenging than it seems, but lets at least re-open it, and assign it to Conan 2.0 for further discussion of what could be done.

Aalmann commented 5 years ago

There are some other points to think about.

variation points

At first of all IMHO a Conan reference has enough variation points with "name", "version", "user" and "channel". Having an additional variation point by providing the ability to use different writings (upper, lower, camel case) is not really what a "normal" user expects and is very error prone.

Additionally some other aspects contradicts it.

Server

Using the Conan Python server on Unix one can easily have different packages for upper, lower, camel case. On Windows it is not possible (not tested, but I assume it concerning the used conan cache layout) BUT: The user itself might not know on which OS the server is installed. The Conan server supports case-insensitive conan search. In Artifactory the case-insensitive flag for conan search does not work (at least in our environment). I didn't test it, but I think Artifactory does not support same package references with different writings.

CMake

Besides the server issues described above there is another more important point. All generated CMake files contains the package name in upper case. So it is completely impossible (even on Unix) to have different packages with the same name but different writing in one graph.

integration of dependencies from different teams

Team A uses "Boost/1.2.3@abc/stable" and Team B uses "boost/1.2.3@abc/stable" as requirements for their products. Team C has to integrate both products. This will end up in an error.

Now two different situations may happen:

  1. both packages are completely equal (same compile flags used etc.)
  2. both packages are different (different content, different compile flags, etc.)

This can be easily detected by checksum of the binary package. So

  1. just use one of them
  2. dependency conflict which has to be solved in a different / special way --> e.g. by enforcement in product C. But the question in this case is: may it really happen that there are two different packages with the same name, version, user and channel but with different content? I don't think so and for the rest (e.g. different user) there are already existing implementations which will handle that.

conclusion

Internal lower case handling of the references would solve the most of the above described problems. The migration of existing packages in cache (only required on Unix) might be easy by simply rename the directories. The conan search command would produce a better output for the user --> "conan search Boost -r all" and "conan search boost -r all" may always deliver references like "boost/1.2.3@abc/stable" or "Boost/1.2.3@abc/stable".

memsharded commented 5 years ago

Thanks @Aalmann for the analysis. It is completely correct.

We will be working on this to be fixed, the alternatives would be:

I am not sure if you are suggesting in your conclussion the first one or the second one, as in your search output you actually output upper cased Boost package.

memsharded commented 5 years ago

Other related issues: https://github.com/conan-io/conan/issues/3315 https://github.com/conan-io/conan/issues/1479 https://github.com/conan-io/conan/issues/3463

Aalmann commented 5 years ago

@memsharded Glad to here that you will working on it.

I think it is more the second one of your alternatives. But I don't understand what you mean with internally different user-defined casing. Can you please provide an example to me? But even if there is such a use-case, I would always take the first one. A better approach would be to have the same behavior like dependency override functionality (error if not explicitly overridden etc.)

memsharded commented 5 years ago

In one computer

$ conan create . MyLib/1.0@user/channel
$ conan upload * -r=remote1

This would have to store in the conan cache, in the metadata somewhere, name=MyLib, as the cache path storage will be forced to: mylib/1.0/user/channel. That is the internally stored user-defined name

In another computer

$ conan create . mylib/1.0@user/channel

Now, it will have name=mylib stored in the cache metadata. Now if it is uploaded, there would be a "conflict" of cased name, two different computers competing for a different one. Even if the first one is respected, that would generate confusing situations to the users, for example, the second one doing a conan search -r=remote and getting MyLib/1.0@user/channel, and thinking why, if I clearly created mylib. Should the metadata of the second client be updated? A local conan search in the cache will definitely return mylib, which will be again confusion, but it won't be updated either, as also different remotes can have different cased representations for the same package. Users creating their own wrappers might easily forget doing case-insensitive comparisons too.

For these reasons, right now I tend to balance towards the first one, keep it as simple as possible, forcing everything to lowercase. That is just a very preliminary position, might change when we actually consider all the implications and how to migrate towards such solution.

Aalmann commented 5 years ago

Ok understand what you mean.

Not sure if it is really a problem, because same confusing situation would happen if both users create a package with exact the same name. And if everything stays as is right now, they are getting another confusing error with no solution available to fix this problem (as described above). If the cache metadata would be lowercase for the dependencies, there might be no issue. Again, there are currently 4 variation points in the reference. If two or more users are creating and uploading packages with the same (exact or different writing) reference but different content, there is an issue with the development process not with the dependency management system.

ThomasKes commented 4 years ago

As @memsharded already indicated, there are basically two different solutions. I'll try to formulate them from a user's point of view:

Alternative A

Conan recipe names are case-sensitive, i.e. openssl/1.1.1d and OpenSSL/1.1.1d may coexist and are considered to be independent from each other. This is is the status quo.

This means, that the Conan cache implementation on Windows is broken, as it cannot store both recipes at the same time.

Possible fixes:

Alternative B

Make Conan recipe names case-insensitive. I.e. openssl/1.1.1d and OpenSSL/1.1.1d will represent the same package. This is a breaking change. As various comments in this and related issues stated, this raises questions on how to display the name to the user and how to handle search queries.

Possible implementation: Convert names to lowercase. Adapt dependency resolution accordingly. Uppercase directories in cache are either migrated or ignored.

Summary

I think this issue is quite critical, since conan-center has transformed all recipe names to lowercase recently. On Windows, a Conan cache cannot be shared anymore between two projects where one uses an old version of a recipe from conan-center (mixed case) and the other uses a recent version (lowercase). This is a huge problem in CI. Therefore many projects are stuck with old versions from conan-center until this issue is fixed. In my opinion, alternative A is the way to go since only the cache implementation must be changed and - most importantly - it's a backwards-compatible change and easy to understand. I also think that any approach, that tries to block uploads/exports (like suggested here) or errors on consumer side is no real solution in a distributed system.

ghost commented 4 years ago

What about storing package name followed by hash? So OpenSSL becomes OpenSSL_6ddfbd8bfad97f29d81ee5d257013f3b0e71d29759085ddcc714efd64de5acdd. This is friendly to both humans and computers.

memsharded commented 4 years ago

What about storing package name followed by hash? So OpenSSL becomes OpenSSL_6ddfbd8bfad97f29d81ee5d257013f3b0e71d29759085ddcc714efd64de5acdd. This is friendly to both humans and computers.

This might be very problematic in Windows with the path length limits.

What we are thinking now would be a very pragmatic approach. As true case-insensitivity in all contexts is extremely challenging, specially for decentralized systems, and not only challenging, but could be quite confusing for users, what if we enforce that conan package names are always, everywhere lowercase? (I am talking Conan 2.0, of course)

ThomasKes commented 4 years ago

What we are thinking now would be a very pragmatic approach. As true case-insensitivity in all contexts is extremely challenging, specially for decentralized systems, and not only challenging, but could be quite confusing for users, what if we enforce that conan package names are always, everywhere lowercase? (I am talking Conan 2.0, of course)

What @Lyberta and I were talking about is true case-sensitivity, not case-insensitivity. I agree, that case-insensitivity is challenging. True case-sensitivity should be quite simple, though. And it's the only way to fix the currently broken situation (see comment above) in Conan < 2.0.

After having a short look into the source code, I think the solution could be to exchange os.path.normpath() in line 44 of package_cache_layout.py by a function which directly maps to os.path.normpath() on Unix, but translates the base path into a case-independent representation on Windows (e.g. OpenSSL <-> $open$s$s$l). The packages_ids() method would have to do the reverse translation. @memsharded what do you think?

memsharded commented 4 years ago

After having a short look into the source code, I think the solution could be to exchange os.path.normpath() in line 44 of package_cache_layout.py by a function which directly maps to os.path.normpath() on Unix, but translates the base path into a case-independent representation on Windows (e.g. OpenSSL <-> $open$s$s$l). The packages_ids() method would have to do the reverse translation. @memsharded what do you think?

The client side is not easy, encoding and reversing the naming will be required in different places, also the requires(), the generated files... will be a very complex feature. But the problem is more in the servers side. Reversing or making the servers understand this case-insensitivity is really challenging, and in any case, it can't do backwards, which renders almost everything we can do in the client mostly useless, as upgrading the servers follow a wildly different flow and pace.

So still seems a huge effort (implementation and future maintenance, future bugs, documentation efforts to explain the behavior...) for the value, this is something that is "simply" fixed if the packages are always using lowercase.

ThomasKes commented 4 years ago

I think we have repeatedly misunderstood each other :-) I do not propose case-insensitivity. I do not propose any change on server side, nor any change in the client when running on Linux or any change in documentation. In my opinion, the only thing that should be changed is the client's cache implementation on Windows, because it's obviously broken.

It's currently perfectly fine to have both OpenSSL/1.1.1b and openssl/1.1.1d on the same server and this used to be the case on conan-center until a few days or weeks ago. Changing the package names is not a fix for the problem, it is the root of it. In an environment with many different projects, many branches, and third-party dependencies, it's virtually impossible to change all packages names to lowercase all at once.

I consider it to be a severe bug of the Conan client on Windows, that you cannot have two projects on the same machine of which one uses OpenSSL/1.1.1b (possibly through a third-party dependency) and the other uses a more recent openssl/1.1.1d. Especially because this isn't a corner case but affects nearly everyone who uses packages from conan-center on Windows.

memsharded commented 4 years ago

It's currently perfectly fine to have both OpenSSL/1.1.1b and openssl/1.1.1d on the same server and this used to be the case on conan-center until a few days or weeks ago.

No, it is not. Actually, it is a known security problem called typo-squatting. Unlikely to happen in ConanCenter because we review contributions, but still not nice that a server can host two or more packages that just differ on the letter casing. That only adds confusion and potential problems, without adding any value.

I consider it to be a severe bug of the Conan client on Windows, that you cannot have two projects on the same machine of which one uses OpenSSL/1.1.1b (possibly through a third-party dependency) and the other uses a more recent openssl/1.1.1d. Especially because this isn't a corner case but affects nearly everyone who uses packages from conan-center on Windows.

ConanCenter is transitioning to packages created in the conan-center-index repo. Once this happens, the old packages with upper case will no longer be accessible.

In an environment with many different projects, many branches, and third-party dependencies, it's virtually impossible to change all packages names to lowercase all at once.

Of course it is not easy. But if we all start using lowercase for the package names, then the problem will vanish relatively soon without any effort.

I am not saying that it is not possible, I am saying that these kind of solutions might seem simple on the surface, but they are not. These changes will be problematic, they will break users, many users (for examples, there are users that are accessing the cache directly), and it will imply changes and conversions in many other places, not just the package_layout. This is a huge effort, it is not something simple, and it can easily drown us in a flood of support tickets, regressions and maintenance nightmare. Users not understanding where the problems come from, and why it is not working. For me, this is a huge risk, that we cannot take if we want to be able to focus on many other higher priority issues. It is not worth it, specially because the more pragmatic approach is definitely possible: just use lowercase. As always, it is a tradeoff.

nioncode commented 4 years ago

Just wanted to chime in here from a user's point of view (i.e. only concerned about the client and not the server). I just updated from libcurl/7.64.1@bincrafters/stable to libcurl/7.70.0, which now pulls in openssl/1.1.1g, whereas it previously used OpenSSL/1.1.1c@conan/stable.

All my CI builds are now failing, since the lowercase openssl conflicts with the OpenSSL that is already cached on Windows. At the moment, I can not do anything to fix this since access to the CI environment is limited, so my only option is to revert the version change and go back to the old version.

So while I strongly support moving to lowercase-only package names, it would be great if something could be done about the cache situation on clients (e.g. by appending either the first few chars of the hash, which should be good enough to avoid collisions while still using fairly short path names on Windows, or simply by adding _1, _2, etc. to the directory name; this would then require the client to look into all of those directories when searching for a match in the cache). At the moment, it is just not possible to have projects that depend on both, old and new OpenSSL versions on the same Windows machine.

sourcedelica commented 4 years ago

hear hear!

memsharded commented 4 years ago

I see, and understand the proposal, but the complexities of being able to store all different variants of case of OpenSSL, openssl, Openssl, OpenSsl, in a way that makes sense would be too much to be developed and maintained in practice. I think the only way is to wait until Conan 2.0, all packages will be lowercase. We are now in a not nice transitory situation with the old packages and the new ones in ConanCenter, that will be probably stabilized before such solution can achieve the necessary level of functionality and stability.

nioncode commented 4 years ago

Sorry if I'm being stupid here, but from my point of view the proposal of storing the package suffixed with the hash of the package name made by @Lyberta above looks trivial to implement on the client: whenever conan wants to read / access the cache, instead of doing $cache/{packagename}, do $cache/{packagename}_sha{packagename}. if you are concerned about the length of the path on windows, simply strip the sha to 6 chars or something, which should still give enough conflict resistence.

This would even be a non-breaking change if conan would first try to access the old path with $cache/{packagename} (to access already, successfully cached) packages and only employs the new cache name if it does not find the existing path – it would still break systems that directly access the conan cache for whatever reason.

I'm not sure what the roadmap for conan 2.0 is as of now, but at the moment conan simply does not work out of the box anymore on Windows, which seems to be a critical issue that should be fixed asap.

For anyone needing a solution for this right now, the only thing I can suggest is to use separate conan caches for all projects (or at least for those that conflict). This can be done by doing something like the following in the project's build scripts on Windows (or by modifying the environment of your CI job):

SET CONAN_USER_HOME=c:\conan-cache\project-name
memsharded commented 4 years ago

We have reached limits of https://www.hyrumslaw.com/

There are users managing the cache directly. Yes, I don't love it, it is not documented, but I bet there are more users doing some kind of tooling around the cache directly, that depends on the cache path names than users being affected by this case-sensitive issue. Introducing this "simple" change will break all those users in production.

Oh, and then the migrations. All users caches in the world of many dozens of thousands of users will suddenly become invalid, need to rename everything. Or you will be constantly running per-package migrations lazily, or creating duplicated packages, having the same package twice, and unable to remove the previous one, because the naming pattern doesn't match the previous one. We have done similar things in the past years. Result in many dozens of small issues, support tickets to respond and to fix, doing patch releases that are very time consuming... Something very painful.

Ok, and now a cache using this naming scheme with latest version gets downgraded to previous Conan version. This is not an imaginary scenario, happens all the time. And then, suddenly, you end with packages in the server that are named {packagename}_sha{packagename}, because there is literally nothing you can do about that.

Trust us, if it was that easy we would have already fixed that 😃

DoDoENT commented 3 years ago

Let me just step in - we have a similar issue as well. Indeed Conan has reached the limits of Hyrum's law.

We started using Conan while it was in version 1.4, way before the decision to enforce all package names in lower case. Actually, every internal package in our company has name in PascalCase. This is OK for internal packages but is problematic for our forks of open source packages. Thus, we have OpenCV, ZLib, etc. This then makes it problematic when someone tries to work with some experimental/small/new projects that want to use original packages from conan-center, as it causes the exact same error.

I agree with @memsharded that using both opencv and OpenCV is not possible in the same project. However, I argue it should be possible in different projects on the same machine, so I propose the following:

The benefit of that feature is that it provides a way to make a safe transition from package names containing capital letters to package names containing only lower case letters - both for conan-center and for many internal packages in various companies, without any hard breaks that would cause a lot of pain. The case insensitive filesystem poses a problem also on macOS, not only on Windows.

So, what do you guys think about this idea? AFAIK, the Artifactory server already supports case-sensitive packages, so the change would only happen on the client. And even more, it does not look like a breaking change (correct me if I'm wrong), so we don't have to wait for Conan v2.0 for that feature - we can have it already in the next release!

memsharded commented 3 years ago

Well, yes, it is possible to introduce changes that could be considered non-breaking if we use configuration opt-ins, and then tell that as you opted-in, you agreed that something was changing.

So, in my opinion, this is massively underestimating the development and maintenance effort of this feature. There are tons of things that can and will go wrong, overwhelming (even more, if that is possible) the maintainers in support, bug fixing, releasing new patches versions...

Just to cite a few things that come to my mind:

Hyrum's law, all of these scenarios will inevitably happen. This is not counting the regressions and bugs that will be introduced while coding this feature.

But another important problem with this huge effort is that it keeps perpetuating something that is not a safe behavior:

So the current proposal for Conan future is to make package names case insensitive, not to allow and manage different casing as different packages. And the most straightforward, robust and understandable way for the users, is to force all package names to be lowercase always. At the moment this is the only possibility I see that will keep the sanity of both users and maintainers.

DoDoENT commented 3 years ago

I agree with everything you said and would never agree to allow case sensitivity by default.

However, by insisting on all packages to always be lower-case will make it impossible to upgrade lots of internal packages across various companies. For example, we have over 200 conan packages in PascalCase name and it's completely unfeasable for them to be renamed into snake_case or kebab-case. However, all those names are internal for the company and none of them clash with open source packages. The difference is only for internal packages of open-source software, like OpenCV, ZLib and OpenSSL, which were made before those ended up in conan-center. Those are the only that are problematic.

So, one solution would be for conan to add the opt-in behaviour describe above, with a disclaimer that this is a very dangerous option and may break badly and that should only be used internally in companies during transition period to lowercase package naming. Open source software and packages that are submitted to conan-center must be validated to be lowercase only.

Another solution would be for conan client to treat OpenCV and opencv always as opencv. In practice, that should work pretty much OK, as most of the package forks are in different user/channel than the original one, so there shouldn't be any conflicts. However, in theory, it's possible that there are OpenCV/4.4.0@company/stable and opencv/4.4.0@company/stable and those would conflict, but I would bet that such thing would never happen in practice.

Also, to be completely sure, we can add second solution as opt-in/beta for conan v1.30 (or some other v1.x release) and ask users for feedback and if it works OK then turn that by default on in conan v2.0. What do you think about that?

memsharded commented 3 years ago

But that will be another, bigger problem. If companies cannot convert their packages to lowercase for Conan 2.0, how will they upgrade to Conan 2.0, that might require bigger changes in their recipes (to use toolchain() approach, changes in the graph model, deprecation of exports_sources in favor of the export_source() method, complete removal of some generators, etc, etc)?

Conan 2.0 is a great opportunity for cleaning the house, getting rid of technical debt and bad defaults. But if full backwards compatibility is required, then there is little we could do, let's continue doing incremental Conan 1.X releases, and lets keep adding debt and slowing development. If changing the package names to lowercase is "unfeasible", then is it really worth kicking off Conan 2.0, that will require bigger changes than that?

To summarize:

This decision, as many times, is a trade-off. I believe that going all lowercase will indeed be very beneficial for users, despite the added migration inconvenience. Just my opinion, will probably ask for feedback in the Conan 2.0 Tribe.

DoDoENT commented 3 years ago

If companies cannot convert their packages to lowercase for Conan 2.0, how will they upgrade to Conan 2.0 ...?

They won't unless there will be a possibility for co-existence of Conan v1.x and Conan v2.x projects. No company will migrate all their packages to lowercase and to other conan v2.0 features all at once. The transition will need to happen gradually, one project at a time, during a certain period. This requires a period where single conan cache has the support for both old and new packages (i.e. OpenCV and opencv, etc.).

Conan 2.0 is a great opportunity for cleaning the house, getting rid of technical debt and bad defaults.

I agree with that. But keep in mind that lot's of houses are "too big" to be cleaned at once. You need to start with one room, then move on to the next and so on... And during that period you will be in a state where some rooms are cleaned and others are not.

But if full backwards compatibility is required, then there is little we could do, let's continue doing incremental Conan 1.X releases, and lets keep adding debt and slowing development. If changing the package names to lowercase is "unfeasible", then is it really worth kicking off Conan 2.0, that will require bigger changes than that?

I wouldn't say that full backwards compatibility is required. Such a thing would definitely make Conan v2.0 pointless. However, co-existance of v1.x and v2.0 packages for different projects on the same machine is required. Without this, the transition to v2.0 would be "all or nothing", and most companies will choose the "nothing" option. Just remember what happened with pyhon 2 -> python3 upgrade - more than 10 years have passed and some projects are still locked to python 2. It would be shame to let happen the same to Conan.

Let me clarify - I don't expect for conan v1.x project to be able to consume conan v2.0 packages or vice versa. I'm quite OK with breaking this. But, in order to migrate 200+ packages from conan v1 to conan v2 I'll have to start one package at the time, updating dependencies. Thus, the "incompatibility border line" will spread from a single package at first to the majority of the packages in the future. But during that, the developers on the most downstream packages should be able to work on them and make product releases, while having a separate development branch on their machine where they could work on migration to conan v2.0 and using v2.x-compatible dependencies. This means that, if I today rename my internal OpenCV package to opencv, it will take approximately 2 months to propagate it to the most downstream users that are building final products. In those 2 months, our CI servers and machines used by senior developers and project managers that work on both low-level ("most upstream") and high-level ("most downstream") projects will need to handle caches with both conan v1 and conan v2 packages.

Of course, one possibility would be to maintain two local caches on each machine - but this looks to me like tool's job, not a developer's job (maybe conan v2.0 can use different default locations for cache than v1.0? That would immediately make v1 packages "invisible" to v2 projects and vice-versa).

This decision, as many times, is a trade-off. I believe that going all lowercase will indeed be very beneficial for users, despite the added migration inconvenience.

I agree. But we need to find a trade-off to minimize the migration inconvenience, not create another pyhon2 -> python3 fiasco.

Just my opinion, will probably ask for feedback in the Conan 2.0 Tribe.

About that, has the mailing started? I didn't get any notification about any message.

keysight-daryl commented 3 years ago

We need an option to enable case insensitivity in conan 1.x as we are hitting the wall on our heads right now with conan-center recent premature name changes. they should never have started doing that until this future of the platform was decided within conan. While I am sure anyone can postulate a thousand 'potential' issues with adding an option here and now about it the truth is the option is needed here and now. keep it simple and just lowercase all names until conan 2.0 arrives. please let's not let minutia details of hypothetical problems obscure common sense.

keysight-daryl commented 3 years ago

Is it possible someone in the know please do whatever it takes to release this as a hot-fix from 1.30.1? please. pretty please even. i want to love conan, i really do, honestly

memsharded commented 3 years ago

Hi @keysight-daryl

I am afraid this is not possible. I would really love a "keep it simple" strategy, some hotfix that could be done, but this is a massive, structural issue, that cannot be solved that easily, without a huge breaking of users. The decentralization, including the fact that other third party servers beside Artifactory and conan_server implement the protocol makes it even harder, if possible at all.

Just making all package names internally lowercase will not fix it, it would break every user that they have a "OpenCV/2.4" package in their server, for example. Unless I am missing something, but I have been thinking about this for a while, and I honestly see no way a hotfix can be implemented. I honestly think this would be the one of the most challenging things to implement right now in Conan, and delivering it robustly could take more than 6 months, in the best case scenario.

Is it possible someone in the know please do whatever it takes to release this as a hot-fix from 1.30.1? please. pretty please even. i want to love conan, i really do, honestly

This is emotional blackmail :) :) :) If we could do something to alleviate this pain, we would certainly do it. We always try our best to help and support.

I think the best we could do now is to propose some mitigation strategies:

Please let us know about which are the things that are biting you more regarding the different casing of packages, give more details about the broken bits, and we will try to prioritize that as much as possible.

keysight-daryl commented 3 years ago

We have forked copies of older recipes etc internally but it seems even not using the newer package is causing conan errors for user of the older packages with the non lowercase names now that a package with lowercase has been put to our art factory. OpenSSL is the big breaker currently now that it is 'OpenSSL' and 'openssl'. our teams pull from our art factory across a bunch of remotes in it and lot of those other 3rd party packages these teams still use from before this was a thing and they have no interest in updating them all.

keysight-daryl commented 3 years ago

maybe the right soultion is to go backwards with the conan-center strategy and rename all packages back to how they were until conan 2.0 is out? It seems their renaming is pre-mature change that is causing the systemic breakage across the board for us... ie they should wait for conan 2.0 to start using the new conventions.

keysight-daryl commented 3 years ago

I was under the impression that as long as your project didn't use any package revisions with the mixed naming as dependencies it wouldn't cause a problem but it appears to error out in conan just trying to find the old package because of the newer named one is in the remote even when they aren't using the newer one.

ant-caichu commented 3 years ago

any settings can skip this error?

CAMOBAP commented 3 years ago

I faced with a similar problem:

macOS, Case insensitive FS

conanfile.txt:

...
opencv/4.5.0
...

[generators]
cmake_find_package
....

And it produces FindOpenCV.cmake but with opencv_LIBRARIES and opencv_INCLUDE_DIRS definitions.

As result find_package(OpenCV REQUIRED) failed

I managed to fix this with this macro

Is there a better option to fix this problem?

xahon commented 3 years ago

I can't beat this issue There is 2 package in my requirements One of them requires "OpenSSL" and the other "openssl". I add either one or another (also with "override") and conan just prefers to ignore me and tries to download both despite the "override" specifier

Ext3h commented 3 years ago

Can we perhaps address this issue in the opposite direction?

With fsutil.exe file setCaseSensitiveInfo $CONAN_USER_HOME\data enable, Windows clients with the WSL subsystem (not WSL2!) installed (Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux ), do support case sensitivity on Windows on NTFS, too.

This is enough to get conan to install both e.g. CMake and cmake on Windows in 2 distinct directories in the cache.

The only catch, for now you have to create these 2 directories by hand side-by-side, as conan now incorrectly complains about the file system being case-insensitive while the relevant part no longer is.

I don't know how this is accessible from Python, but via the CreaterFile API there is also FILE_FLAG_POSIX_SEMANTICS to set this flag programmatically when creating the cache directory structure.