memsharded
commented
3 years ago Hi @prietomj
The truth is that the Conan cache has never guaranteed concurrency. While it tried to provide some protections for concurrent usage for some specific use cases, it was never really concurrent in the general case. From Conan 1.7 the cache design and functionality has evolved a lot, to support many important new features (for example, the .tgz files are not removed anymore, so for re-uploading it is order of magnitude more efficient). Also now the Conan cache supports revisions, which is an important feature to support immutable packages and improve traceability, which didn't in Conan 1.7.
In the general case, parallel CI jobs for example, should use their own caches, to avoid concurrency issues. Using a fast server like Artifactory close to the CI agents, or maybe using the new download cache can help keeping fast builds even if the cache is not shared among different jobs. We will try to give it another try at Conan 2.0 new cache design, but concurrency is extremely challenging, so we cannot guarantee that it will be possible to achieve it soon.
prietomj
Hello, some information about our system OS Windows 10 Conan version 1.32.1 Python version 3.8 We use Jenkins for automatizing builds
After updating conan from 1.7.0 to 1.32.1 version I am getting sometimes a build error like this in Windows
Traceback (most recent call last): File "c:\python38\lib\site-packages\conans\client\command.py", line 2115, in run method(args[0][1:]) File "c:\python38\lib\site-packages\conans\client\command.py", line 495, in install info = self._conan.install(path=args.path_or_reference, File "c:\python38\lib\site-packages\conans\client\conan_api.py", line 94, in wrapper return f(api, *args, **kwargs) File "c:\python38\lib\site-packages\conans\client\conan_api.py", line 598, in install deps_install(app=self.app, File "c:\python38\lib\site-packages\conans\client\manager.py", line 55, in deps_install deps_graph = graph_manager.load_graph(ref_or_path, create_reference, graph_info, build_modes, File "c:\python38\lib\site-packages\conans\client\graph\graph_manager.py", line 114, in load_graph deps_graph = self._resolve_graph(root_node, graph_info, build_mode, check_updates, update, File "c:\python38\lib\site-packages\conans\client\graph\graph_manager.py", line 238, in _resolve_graph deps_graph = self._load_graph(root_node, check_updates, update, File "c:\python38\lib\site-packages\conans\client\graph\graph_manager.py", line 360, in _load_graph self._recurse_build_requires(graph, builder, check_updates, update, build_mode, File "c:\python38\lib\site-packages\conans\client\graph\graph_manager.py", line 284, in _recurse_build_requires self._binary_analyzer.evaluate_graph(graph, build_mode, update, remotes, nodes_subset, root) File "c:\python38\lib\site-packages\conans\client\graph\graph_binaries.py", line 384, in evaluate_graph self._evaluate_node(node, build_mode, update, remotes) File "c:\python38\lib\site-packages\conans\client\graph\graph_binaries.py", line 176, in _evaluate_node self._process_node(node, pref, build_mode, update, remotes) File "c:\python38\lib\site-packages\conans\client\graph\graph_binaries.py", line 242, in _process_node self._evaluate_cache_pkg(node, package_layout, pref, metadata, remote, remotes, update) File "c:\python38\lib\site-packages\conans\client\graph\graph_binaries.py", line 84, in _evaluate_cache_pkg if self._check_update(upstream_manifest, package_folder, output): File "c:\python38\lib\site-packages\conans\client\graph\graph_binaries.py", line 26, in _check_update read_manifest = FileTreeManifest.load(package_folder) File "c:\python38\lib\site-packages\conans\model\manifest.py", line 91, in load text = load(os.path.join(folder, CONAN_MANIFEST)) File "c:\python38\lib\site-packages\conans\util\files.py", line 232, in load with open(path, 'rb') as handle: FileNotFoundError: [Errno 2] No such file or directory: 'C:\\ConanFolder\\.conan\\data\\Library\\1.0.0\\builduser\\snapshot\\package\\6a4a582aa8fcd14dd5cfc5414d5dfa4e4ffe41d0\\conanmanifest.txt'This happens when Library is being built around the same time that this program is being build that uses Library. In conan 1.7.0 when that happened normally there was a lockfile that could be removed in order to the program build to continue, but now it directly throws an error which is not the behavior we want. Is there any workaround for this problem?
I tried enabling and disabling shortpaths as I thought it was related but that did not changed the problem (only the pathname).
Thank you very much.