[bug] lru code broke with 2.0.14

gjasny commented 10 months ago

Environment details

Operating System+version: MacOS 14, Windows 10
Compiler+version: various
Conan version: 2.0.14
Python version: various

Steps to reproduce

I can only reproduce it in CI. There Conan 2.0.13 might have ran before on the machine. I'm executing conan install with a custom profile.

The following backtrace can be found in the output:

[2023-11-20T11:09:44.773Z]    actual-err> ======== Computing dependency graph ========
[2023-11-20T11:09:44.773Z]    actual-err> ERROR: Traceback (most recent call last):
[2023-11-20T11:09:44.773Z]    actual-err>   File "conan\cli\cli.py", line 272, in main
[2023-11-20T11:09:44.773Z]    actual-err>   File "conan\cli\cli.py", line 172, in run
[2023-11-20T11:09:44.773Z]    actual-err>   File "conan\cli\command.py", line 125, in run
[2023-11-20T11:09:44.773Z]    actual-err>   File "conan\cli\commands\install.py", line 58, in install
[2023-11-20T11:09:44.773Z]    actual-err>   File "conan\api\subapi\graph.py", line 148, in load_graph_consumer
[2023-11-20T11:09:44.773Z]    actual-err>   File "conan\api\subapi\graph.py", line 180, in load_graph
[2023-11-20T11:09:44.773Z]    actual-err>   File "conans\client\graph\graph_builder.py", line 51, in load_graph
[2023-11-20T11:09:44.773Z]    actual-err>   File "conans\client\graph\graph_builder.py", line 90, in _expand_require
[2023-11-20T11:09:44.773Z]    actual-err>   File "conans\client\graph\graph_builder.py", line 262, in _create_new_node
[2023-11-20T11:09:44.773Z]    actual-err>   File "conans\client\graph\graph_builder.py", line 210, in _resolve_recipe
[2023-11-20T11:09:44.773Z]    actual-err>   File "conans\client\graph\proxy.py", line 23, in get_recipe
[2023-11-20T11:09:44.773Z]    actual-err>   File "conans\client\graph\proxy.py", line 37, in _get_recipe
[2023-11-20T11:09:44.773Z]    actual-err>   File "conans\client\cache\cache.py", line 60, in recipe_layout
[2023-11-20T11:09:44.773Z]    actual-err>   File "conan\internal\cache\cache.py", line 99, in get_recipe_layout
[2023-11-20T11:09:44.773Z]    actual-err>   File "conan\internal\cache\db\cache_database.py", line 73, in get_recipe
[2023-11-20T11:09:44.773Z]    actual-err>   File "conan\internal\cache\db\recipes_table.py", line 105, in get_recipe
[2023-11-20T11:09:44.773Z]    actual-err> TypeError: _.__new__() missing 1 required positional argument: 'lru'

The code in conan\internal\cache\db\recipes_table.py was last touched with: 469fb2cf45b2cecbbc58933630622f0e7d1bcf38 (by @memsharded).

Logs

No response

gjasny commented 10 months ago

Maybe this is related to #15129. We definitely set core.cache:storage_path.

memsharded commented 10 months ago

Hi @gjasny

Thanks for your report. This might be the effect of a bad Python package update from Conan 2.0.13 to 2.0.14, this would be a reason why you are only seeing it in the CI? Could you please try to do a fresh installation of Conan 2.0.14 in that machine?

Maybe this is related to https://github.com/conan-io/conan/issues/15129. We definitely set core.cache:storage_path.

That only applies to whether the migration runs or not, but it will result in errors in the migration, much earlier in the execution. This seems like a problem in the code, more like a failed package update.

gjasny commented 10 months ago

I now have a local and reproducible testcase. It boils down to re-using a storage_path that was pre-populated with Conan 2.0.13. If I run a conan list */ against that I see the following error:

Local Cache
  ERROR: no such column: lru

But the conan exit code still signals success by returning with 0.

If I do a conan install I do not see any messages hinting at any database conversion. Instead I see the aforementioned stacktrace.

My testcase:

#/usr/bin/env bash

PROJECT_ROOT=$(cd $(dirname "${BASH_SOURCE[0]}") && /bin/pwd -P)

export CONAN_HOME="${PROJECT_ROOT}/_c"
export CONAN="/Users/gregorj/tmp/conan/conan" # this is 2.0.14 from github

rm -rf "${CONAN_HOME}" "${PROJECT_ROOT}/_conan"
mkdir -p "${CONAN_HOME}"
cp "${PROJECT_ROOT}/global.conf" "${CONAN_HOME}/"

#"${CONAN}" list '*/'

"${CONAN}" profile detect
#"${CONAN}" remote add dvp ...
#"${CONAN}" remote add --force conancenter ...
"${CONAN}" install "${PROJECT_ROOT}" --output-folder "${PROJECT_ROOT}/_conan" --build missing

The global.conf content:

core.cache:storage_path=/path/to/my/conan-storage

#core.net.http:proxies=...
#core.net.http:no_proxy_match=...

gjasny commented 10 months ago

conan-storage/cache.sqlite3 dump:

PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE recipes (reference text NOT NULL, rrev text NOT NULL, path text NOT NULL UNIQUE, timestamp real NOT NULL , UNIQUE(reference, rrev));
INSERT INTO recipes VALUES('gtest/1.13.0@dvp/master','e2299ac406263a0ffe82665aaa765da6','gteste8ec463151c56',1683795591.4350000163);
CREATE TABLE packages (reference text NOT NULL, rrev text NOT NULL, pkgid text, prev text, path text NOT NULL UNIQUE, timestamp real NOT NULL, build_id text , UNIQUE(reference, rrev, pkgid, prev));
INSERT INTO packages VALUES('gtest/1.13.0@dvp/master','e2299ac406263a0ffe82665aaa765da6','33aaa63df96c29714f7ab9e22e0b4944ef6c46b1','8d64835647d34c7edc828053ca403d97','b/gtest92248918f2688',1691689037.9328539673,NULL);
INSERT INTO packages VALUES('gtest/1.13.0@dvp/master','e2299ac406263a0ffe82665aaa765da6','fc73756a4cb27f495378d17ccd60849a56b5f0c2','c6169055e53b1603d78adf431be3bfb9','b/gtest35e2888c1ab74',1695795676.167481325,NULL);
COMMIT;

gjasny commented 10 months ago

Maybe our usage pattern of setting CONAN_HOME to a temporary dir and re-using conan-storage (and conan-download) across versions has not been covered yet.

If conan-storage could be independently set of CONAN_HOME, shouldn't the database schema version be stored within or right next to the database?

memsharded commented 10 months ago

Maybe our usage pattern of setting CONAN_HOME to a temporary dir and re-using conan-storage (and conan-download) across versions has not been covered yet.

Yes, I am afraid that this is the case. The Conan cache is not designed to be used by multiple different Conan versions. And for the most part, this migration could only be implemented assuming a 1:1 relationship between the Conan home and the storage path. This is not a problem of this specific migration, but of the migrations framework, that has this limitation. We are very sorry for the inconvenience of overlooking this.

If conan-storage could be independently set of CONAN_HOME, shouldn't the database schema version be stored within or right next to the database?

Yes, there is already some work by contributors and we will try to move it forward to implement a migration framework that migrates the storage folder independently of the home. I think it will still only support 1 Conan version, it doesn't seem feasible that the same storage could satisfy 2 different Conan versions that need different storage schema. But it will still be an improvement over the current migrations framework. Still, it cannot apply retroactively, and 2.0.14 already introduced this migration, so this a bit unlucky.

It is also true that migrations over the storage DB are very unlikely, we hope that this doesn't happen easily again in the future.

memsharded commented 2 months ago

Closing this as outdated, the 2.0.14 migration issues were mostly solved or outdated, thanks!

conan-io / conan