Closed atussiot closed 7 years ago
You are very right. Symbolic links right now are only followed for files, not directories. Not sure if it is a bug, or it has a reason, let's have a look and see if it makes sense to fix it. Annotating it as bug at the moment. Lets try for next release 0.23.
Thanks very much for your detailed report, really useful!
Hi @atussiot , I have started to have a look at this for next release. I would like to know more about where the symbolic link is defined, and when it is lost. I thought that it was during the export stage, but now not so sure.
Several possibilities:
exports
or exports_sources
to store the source folder into the recipebuild
, and it points to something in the build folder, but when packages are created, just the links are copied into the package
folder. This is why it works locally, but later when uploaded it breaks.
You might be able to inspect the actual files inside your <userhome>/.conan/data
folder.Thanks for your feedback!
Hi, thanks for taking care of it!
I am not sure these possibilities apply here:
exports
, but only for a trivial patch.make
.I tried removing everything under ~/.conan/data
, except the package
and it still works fine locally. To me it really looks like it is something happening during the upload.
Let me know if there is something else I can test/check/do to help!
I am working on this, and my initial tests are confusing. It is true that symlinks are not maintained while packaging, from the "build" to the "package" folder, with the package()
method.
When symlinks are defined in the build between folders, such folders and files are actually duplicated in the package folder, so everything keep workings, the only problem is that files are duplicated wasting extra disk space.
So I would like to know a couple of things:
package()
method.
Thank you very much!Hmm interesting. I am actually not using the package()
method at all. I simply use the make install
command that copies everything directly into the package
folder from the build()
method (hence also duplicating). Something like this:
# Tell the configure script to install directly in the package folder
self.run('cd {} && ./configure -prefix {}'.format(self.project_root, self.package_folder))
# Frameworks are created here (with the symlinks within themselves)...
self.run('cd {} && make -j {}'.format(self.project_root, cpu_count()))
# ... and copied here from build to package folder.
self.run('cd {} && make install'.format(self.project_root))
The layout I showed above was in the package
folder, and it is the same in the build
one.
In the duplication you describe, you mention both folders and files. I guess the problem I'm having is different as the links pointing to files are preserved. Wouldn't they be lost too if it would be a matter of where the files/folders are located?
Good to know. Then, I think the issue has two different parts: one would be to preserve folder links in the package()
method, because eventually someone else will suffer this problem, even if you are fine now with make install
, and then also check the compression and decompression of the package .tgz, to check what is happening.
It is ok that symlinks to files are preserved, because they are explicitely handled. The duplication only happens if the symlink is done between folders (because symlinks=True, so directory walk will traverse it, but then files will not be symlinks and be treated as different files).
I will have another look, thanks for the info!
Agreed, I guess it is important to look at both. In fact, I am facing other issues with folder because the copy()
method doesn't let me copy them. I'm still not sure how to properly handle it in client conan files. It would be handy to be able to do something like self.copy('*.framework', 'lib', 'lib', symlinks=True)
. But I guess that should be reported as a separate issue?
But regarding my current issue, I believe checking the compression and decompression of the package is the way to go. Let me know if I can help with that too!
Out of curiosity, would you know maybe if there are examples of anybody attempting to handle frameworks like this before?
Thanks again for handling it!
I would like some others feedback on this issue. The problem that I am facing to implement this is that it might break existing packages, because the generated tgz will be different, zipping folders symlink that previously were not being packaged.
So I would like to understand the real goal of such symlinks. They seem to make sense to me (from my ignorance of OSX frameworks) when there are different versions, so the symlink between "latest"->"versions/2" is useful. But conan already provides the solution for handling different versions, without symlinks. Packages will work fine just by packaging from the folder you want.
So please, I would like to know:
I keep working on it, trying to find a way. Thanks for the feedback!
I have submitted the PR anyway, in case someone wants to check it. https://github.com/conan-io/conan/pull/1289
The PR has been merged, so it will be in next release :)
Great news, thank you very much! Looking forward to testing it!
Thanks! If you haven't yet, you probably want to subscribe to the release announcement mailing list (in conan.io footer). Exclusively for announcing new releases. We will be announcing the release candidate and release there.
Hi @atussiot , this is already released in 0.23, could you please upgrade and try it? Thanks!
Hi @memsharded!
Sorry for the late reply, I will give it a try today and let you know how it goes!
As far as it concerns the reported issue, it now works as expected, thank you very much again!
However I run into a new problem, in the conanfile.py
of the test_package
. The only way I figured out so far to copy the folders is:
def imports(self):
self.copy("lib*", root_package="Qt5")
But apparently this one is not preserving links (not even for binaries this time), leading to a crash. Is that expected or am I doing something wrong?
self.copy()
takes a links
argument. Set that to true and that should work.
Also, there is an alternative approach you might be interested in having a look. Instead of copying the shared libraries (imports is mainly for that) from the Qt package folder, to the test folder, you can try making the Qt package to add itself, or its bin
or lib
subfolder to the path, or to LD_LIBRARY_PATH, depending on the system. Something like (not tested):
def package_info(self):
if self.settings.os == "Windows":
self.env_info.PATH.append("bin")
elif self.settings.os == "Linux":
self.env.info.LD_LIBRARY_PATH.append("lib")
else: ...
Hello,
I'm trying to make a recipe to build Qt5 for macOS and build the modules as frameworks. Everything works perfectly when they are built and used locally. However, if I upload the package to the server, the package is not usable on any machine that will pull this package.
My feeling is that the export of the package is not preserving the symbolic links to directories (which are fundamental in frameworks).
For instance, under
QtCore.framework
, on the build machine I have:while in the same package downloaded from the server I get (only the link to the binary survives):
This causes CMake to fail in our project:
I saw that the PR #947 addressed a similar issue. But somehow it doesn't seem to apply to links to directory?
I also checked the
conanmanifest.txt
and the missing links are not referenced there, perhaps the root cause is that their hashes cannot be computed? Or am I missing something?Thank you!