search

iterative / dvclive

📈 Log and track ML metrics, parameters, models with Git and/or DVC
https://dvc.org/doc/dvclive
Apache License 2.0
161 stars 35 forks source link

Error RuntimeError: SQLite version is too old, please upgrade to >= (3, 8, 3) #777

Closed loveis98 closed 7 months ago

loveis98 commented 7 months ago

I try to use DVC extention for VSCode, and I have run the example from started page:

from dvclive import Live
import random
import sys

with Live(save_dvc_exp=True, ) as live:
    epochs = 6
    live.log_param("epochs", epochs)
    for epoch in range(epochs):
        live.log_metric("train/accuracy", epoch + random.random())
        live.log_metric("train/loss", epochs - epoch - random.random())
        live.log_metric("val/accuracy",epoch + random.random() )
        live.log_metric("val/loss", epochs - epoch - random.random())
        live.next_step()

But see the error:

RuntimeError                              Traceback (most recent call last)
Cell In[1], line 7
      4 import random
      5 import sys
----> 7 with Live(save_dvc_exp=True, ) as live:
      8     epochs = 6
      9     live.log_param("epochs", epochs)

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvclive/live.py:626](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvclive/live.py:626), in Live.__exit__(self, exc_type, exc_val, exc_tb)
    624 def __exit__(self, exc_type, exc_val, exc_tb):
    625     self._inside_with = False
--> 626     self.end()

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvclive/live.py:603](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvclive/live.py:603), in Live.end(self)
    598     if self._dvcyaml:
    599         catch_and_warn(DvcException, logger)(self._dvc_repo.scm.add)(
    600             self.dvc_file
    601         )
--> 603 self.save_dvc_exp()
    605 # Mark experiment as done
    606 self.post_to_studio("done")

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvclive/utils.py:182](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvclive/utils.py:182), in catch_and_warn..decorator..wrapper(*args, **kwargs)
    180 def wrapper(*args, **kwargs):
    181     try:
--> 182         return func(*args, **kwargs)
    183     except exception as e:
    184         logger.warning(f"Error in {func.__name__}: {e}")

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvclive/live.py:633](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvclive/live.py:633), in Live.save_dvc_exp(self)
    631 if self._dvcyaml:
    632     self._include_untracked.append(self.dvc_file)
--> 633 self._experiment_rev = self._dvc_repo.experiments.save(
    634     name=self._exp_name,
    635     include_untracked=self._include_untracked,
    636     force=True,
    637     message=self._exp_message,
    638 )

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/experiments/__init__.py:359](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/experiments/__init__.py:359), in Experiments.save(self, *args, **kwargs)
    356 def save(self, *args, **kwargs):
    357     from dvc.repo.experiments.save import save
--> 359     return save(self.repo, *args, **kwargs)

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/experiments/save.py:36](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/experiments/save.py:36), in save(repo, targets, name, recursive, force, include_untracked, message)
     33 executor = queue.init_executor(repo.experiments, entry)
     35 try:
---> 36     save_result = executor.save(
     37         executor.info,
     38         targets=targets,
     39         recursive=recursive,
     40         force=force,
     41         include_untracked=include_untracked,
     42         message=message,
     43     )
     44     result = queue.collect_executor(repo.experiments, executor, save_result)
     45 finally:

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/experiments/executor/base.py:277](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/experiments/executor/base.py:277), in BaseExecutor.save(cls, info, targets, recursive, force, include_untracked, message)
    274     os.chdir(dvc.root_dir)
    276 include_untracked = include_untracked or []
--> 277 include_untracked.extend(cls._get_top_level_paths(dvc))
    278 # dvc repro automatically stages dvc.lock. Running redundant `git add`
    279 # on it causes an error when exiting the detached head context.
    280 if LOCK_FILE in dvc.scm.untracked_files():

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/experiments/executor/base.py:249](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/experiments/executor/base.py:249), in BaseExecutor._get_top_level_paths(cls, repo)
    243 @classmethod
    244 def _get_top_level_paths(cls, repo: "Repo") -> list["str"]:
    245     return list(
    246         chain(
    247             _collect_top_level_metrics(repo),
    248             _collect_top_level_params(repo),
--> 249             repo.index._plot_sources,
    250         )
    251     )

File [~/dvc_demo/venv/lib/python3.10/site-packages/funcy/objects.py:25](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/funcy/objects.py:25), in cached_property.__get__(self, instance, type)
     23 if instance is None:
     24     return self
---> 25 res = instance.__dict__[self.fget.__name__] = self.fget(instance)
     26 return res

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/index.py:452](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/index.py:452), in Index._plot_sources(self)
    449 from dvc.repo.plots import _collect_pipeline_files
    451 sources: list[str] = []
--> 452 for data in _collect_pipeline_files(self.repo, [], {}).values():
    453     for plot_id, props in data.get("data", {}).items():
    454         if isinstance(props.get("y"), dict):

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/plots/__init__.py:476](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/plots/__init__.py:476), in _collect_pipeline_files(repo, targets, props, onerror)
    473             k, v = next(iter(elem.items()))
    474             dvcfile_defs_dict[k] = v
--> 476     resolved = _resolve_definitions(
    477         repo.dvcfs, targets, props, dvcfile_path, dvcfile_defs_dict, onerror=onerror
    478     )
    479     dpath.merge(result, {dvcfile_path: resolved})
    480 return result

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/plots/__init__.py:451](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/plots/__init__.py:451), in _resolve_definitions(fs, targets, props, config_path, definitions, onerror)
    449     data_path = _normpath(fs.join(config_dir, plot_id))
    450     if _matches(targets, config_path, plot_id):
--> 451         unpacked = unpack_if_dir(
    452             fs, data_path, props={**plot_props, **props}, onerror=onerror
    453         )
    454         dpath.merge(result, unpacked)
    455 elif _matches(targets, config_path, plot_id):

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/plots/__init__.py:509](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/plots/__init__.py:509), in unpack_if_dir(fs, path, props, onerror)
    507 def unpack_if_dir(fs, path, props: dict[str, str], onerror: Optional[Callable] = None):
    508     result: dict[str, dict] = defaultdict(dict)
--> 509     if fs.isdir(path):
    510         unpacked = _unpack_dir_files(fs, path, onerror=onerror)
    511     else:

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvc_objects/fs/base.py:295](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvc_objects/fs/base.py:295), in FileSystem.isdir(self, path)
    294 def isdir(self, path: AnyFSPath) -> bool:
--> 295     return self.fs.isdir(path)

File [~/dvc_demo/venv/lib/python3.10/site-packages/fsspec/spec.py:654](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/fsspec/spec.py:654), in AbstractFileSystem.isdir(self, path)
    652 """Is this entry directory-like?"""
    653 try:
--> 654     return self.info(path)["type"] == "directory"
    655 except IOError:
    656     return False

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvc/fs/dvc.py:421](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvc/fs/dvc.py:421), in _DVCFileSystem.info(self, path, **kwargs)
    419 key = self._get_key_from_relative(path)
    420 ignore_subrepos = kwargs.get("ignore_subrepos", True)
--> 421 return self._info(key, path, ignore_subrepos=ignore_subrepos)

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvc/fs/dvc.py:426](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvc/fs/dvc.py:426), in _DVCFileSystem._info(self, key, path, ignore_subrepos, check_ignored)
    423 def _info(  # noqa: C901, PLR0912
    424     self, key, path, ignore_subrepos=True, check_ignored=True
    425 ):
--> 426     repo, dvc_fs, subkey = self._get_subrepo_info(key)
    428     dvc_info = None
    429     if dvc_fs:

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvc/fs/dvc.py:337](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvc/fs/dvc.py:337), in _DVCFileSystem._get_subrepo_info(self, key)
    334     repo_key = self._get_key(repo.root_dir)
    335     subkey = key[len(repo_key) :]
--> 337 dvc_fs = self._datafss.get(repo_key)
    338 return repo, dvc_fs, subkey

File /usr/local/basement/Python-3.10.8/lib/python3.10/functools.py:981, in cached_property.__get__(self, instance, owner)
    979 val = cache.get(self.attrname, _NOT_FOUND)
    980 if val is _NOT_FOUND:
--> 981     val = self.func(instance)
    982     try:
    983         cache[self.attrname] = val

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvc/fs/dvc.py:263](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvc/fs/dvc.py:263), in _DVCFileSystem._datafss(self)
    261 if hasattr(self.repo, "dvc_dir"):
    262     key = self._get_key(self.repo.root_dir)
--> 263     datafss[key] = DataFileSystem(index=self.repo.index.data["repo"])
    265 return datafss

File [~/dvc_demo/venv/lib/python3.10/site-packages/funcy/objects.py:25](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/funcy/objects.py:25), in cached_property.__get__(self, instance, type)
     23 if instance is None:
     24     return self
---> 25 res = instance.__dict__[self.fget.__name__] = self.fget(instance)
     26 return res

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/index.py:547](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvc/repo/index.py:547), in Index.data(self)
    545 index = self.repo.data_index
    546 prefix = ("tree", self.data_tree.hash_info.value)
--> 547 if index.has_node(prefix):
    548     loaded = True
    550 if not loaded:

File [~/dvc_demo/venv/lib/python3.10/site-packages/dvc_data/index/index.py:738](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/dvc_data/index/index.py:738), in DataIndex.has_node(self, key)
    737 def has_node(self, key: DataIndexKey) -> bool:
--> 738     return self._trie.has_node(key)

File [~/dvc_demo/venv/lib/python3.10/site-packages/sqltrie/serialized.py:101](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/sqltrie/serialized.py:101), in SerializedTrie.has_node(self, key)
    100 def has_node(self, key):
--> 101     return self._trie.has_node(key)

File [~/dvc_demo/venv/lib/python3.10/site-packages/sqltrie/sqlite/sqlite.py:334](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/sqltrie/sqlite/sqlite.py:334), in SQLiteTrie.has_node(self, key)
    332 def has_node(self, key: TrieKey) -> bool:
    333     try:
--> 334         self._get_node(key)
    335         return True
    336     except KeyError:

File [~/dvc_demo/venv/lib/python3.10/site-packages/sqltrie/sqlite/sqlite.py:202](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/sqltrie/sqlite/sqlite.py:202), in SQLiteTrie._get_node(self, key)
    196 if not key:
    197     return self._conn.execute(
    198         "SELECT * FROM nodes WHERE id == ?",
    199         (self._root_id,),
    200     ).fetchone()
--> 202 rows = list(self._traverse(key))
    203 if len(rows) != len(key):
    204     raise KeyError(key)

File [~/dvc_demo/venv/lib/python3.10/site-packages/sqltrie/sqlite/sqlite.py:191](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/sqltrie/sqlite/sqlite.py:191), in SQLiteTrie._traverse(self, key)
    189 def _traverse(self, key):
    190     path = "/".join(key).replace("'", "''")
--> 191     self._conn.executescript(STEPS_SQL.format(path=path, root=self._root_id))
    193     return self._conn.execute(f"SELECT * FROM {STEPS_TABLE}").fetchall()

File [~/dvc_demo/venv/lib/python3.10/site-packages/sqltrie/sqlite/sqlite.py:145](https://vscode-remote+ssh-002dremote-002b7b22686f73744e616d65223a224a485f524241227d.vscode-resource.vscode-cdn.net/dvc_demo/~/dvc_demo/venv/lib/python3.10/site-packages/sqltrie/sqlite/sqlite.py:145), in SQLiteTrie._conn(self)
    142 @property
    143 def _conn(self):  # pylint: disable=method-hidden
    144     if sqlite3.sqlite_version_info < MIN_SQLITE_VER:
--> 145         raise RuntimeError(
    146             f"SQLite version is too old, please upgrade to >= {MIN_SQLITE_VER}"
    147         )
    149     conn = getattr(self._local, "conn", None)
    150     if conn is None:

RuntimeError: SQLite version is too old, please upgrade to >= (3, 8, 3)

Could you help me, please? I can't understand even why and when I call sqlite

dberenbaum commented 7 months ago

sqlite is used by dvc to keep some info so it doesn't need to be recollected from files constantly. As the error suggests, the solution is to upgrade your sqlite version.

If that's not possible, you could try dvc_exp_save=False. Note that with dvc_exp_save=False, it will be up to you to commit the results of each experiment to git.

loveis98 commented 7 months ago

sqlite is used by dvc to keep some info so it doesn't need to be recollected from files constantly. As the error suggests, the solution is to upgrade your sqlite version.

If that's not possible, you could try dvc_exp_save=False. Note that with dvc_exp_save=False, it will be up to you to commit the results of each experiment to git.

But how can I upgrade my sqlite version? How can I do it with pip?

shcheklein commented 7 months ago

@loveis98 could you please run dvc version?

Also:

import sqlite3
sqlite3.sqlite_version

Also, describe please the context. Where do you run it (is it some Amazon AWS image / box, do you have a control on what distribution you are using, etc).

The problem usually is that sqlite comes as part of the Linux distro itself. Depending on the context, either you could upgrade it or change the distro.

Example: https://stackoverflow.com/questions/57214850/trying-to-upgrade-sqlite-on-amazon-ec2

loveis98 commented 7 months ago

import sqlite3 sqlite3.sqlite_version

image I run it locally (Linux), in VSCode

shcheklein commented 7 months ago

@loveis98 could you also please give more context (where do you run it, etc)? Run dvc version, etc?

loveis98 commented 7 months ago

@loveis98 could you also please give more context (where do you run it, etc)? Run dvc version, etc?

dvc version DVC version: 3.43.1 (pip)

Platform: Python 3.10.8 on Linux-3.10.0-1160.99.1.el7.x86_64-x86_64-with-glibc2.17 Subprojects: dvc_data = 3.9.0 dvc_objects = 3.0.6 dvc_render = 1.0.1 dvc_task = 0.3.0 scmrepo = 2.1.1 Supports: hdfs (fsspec = 2022.11.0, pyarrow = 11.0.0), http (aiohttp = 3.9.0, aiohttp-retry = 2.8.3), https (aiohttp = 3.9.0, aiohttp-retry = 2.8.3) Config: Global: /home/user/.config/dvc System: /etc/xdg/dvc Cache types: https://error.dvc.org/no-dvc-cache Caches: local Remotes: None Workspace directory: ext4 on /dev/mapper/vg_data-lv_home Repo: dvc, git Repo.site_cache_dir: /var/tmp/dvc/repo/851fd0d197ac3f88c8a5ca03350330ed

shcheklein commented 7 months ago

So, I might be wrong but it seems that Linux-3.10.0 is quite outdated. Probably release in 2017 and not maintained anymore. You would have to ask your devops / admins to upgrade the distro (can be complicated), or at least find a way to update the sqlite for you on it. I don't think we can do much on our end for this tbh. I would too tedious for us as a small team to keep compatibility with a very outdated distros.