Open marioperezj opened 1 month ago
dberenbaum
commented
1 month ago Maybe something is corrupted from the old repo? Could you try to destroy again and make sure everything is gone? Could you also try destroying the site cache dir (/var/tmp/dvc/repo/80dc6aaf08a6c608dfe4ff9c3c907a02)?
marioperezj
commented
1 month ago Thank you for the response. I think my general issue is that I don't know very well how the cache works in dvc. I undestand the repository cache is stored in .dvc inside the repository, but other cache levels are also active and they have some files that are corrupted.
I do dvc destroy inside the repository. Then I do dvc init. followed by dvc exp show and I still see some experiments from previous commits in the master and workspace. How is it possible I have an experiment instance in the workspace if I don't even have stages defined? How to completely start a clean dvc instance?
For instance, in a newly dvc instance (right after dvc destroy) trying to do dvc gc -A outputs:
PS D:\repo\new_repo> dvc gc -A
WARNING: This will remove all cache except items used in the workspace and all git commits of the current repo.
Are you sure you want to proceed? [y/n]: y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 6d4d0548ad88fd32d3fd9f364db422df.dir
Missing cache for directory '/datasets'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 55a3fdcf02b802bf4eb2d7b0eaf166c1.dir
Missing cache for directory '/models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 407217359088a8c817c69fb4fae7f30f.dir
Missing cache for directory '/evaluations'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 798d782e087169db9e90de53d29686de.dir
Missing cache for directory '/original_models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: fed68f66bd0db0a079d7405a00771906.dir
Missing cache for directory '/datasets'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 3572310ba721c824bdf2647005c2265b.dir
Missing cache for directory '/models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 407217359088a8c817c69fb4fae7f30f.dir
Missing cache for directory '/evaluations'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 1ad87cbce6f391d05b1ce4b40a550a57.dir
Missing cache for directory '/original_models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: fed68f66bd0db0a079d7405a00771906.dir
Missing cache for directory '/datasets'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 3572310ba721c824bdf2647005c2265b.dir
Missing cache for directory '/models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 407217359088a8c817c69fb4fae7f30f.dir
Missing cache for directory '/evaluations'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 1ad87cbce6f391d05b1ce4b40a550a57.dir
Missing cache for directory '/original_models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: fed68f66bd0db0a079d7405a00771906.dir
Missing cache for directory '/datasets'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 3572310ba721c824bdf2647005c2265b.dir
Missing cache for directory '/models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 407217359088a8c817c69fb4fae7f30f.dir
Missing cache for directory '/evaluations'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 1ad87cbce6f391d05b1ce4b40a550a57.dir
Missing cache for directory '/original_models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 6632f42841f6355685e089b62a85c45c.dir
Missing cache for directory '/models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Output 'evaluations'(stage: '..\..\dvc.yaml:evaluation') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 1ad87cbce6f391d05b1ce4b40a550a57.dir
Missing cache for directory '/original_models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 6632f42841f6355685e089b62a85c45c.dir
Missing cache for directory '/models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 1ad87cbce6f391d05b1ce4b40a550a57.dir
Missing cache for directory '/original_models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 6632f42841f6355685e089b62a85c45c.dir
Missing cache for directory '/models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 1ad87cbce6f391d05b1ce4b40a550a57.dir
Missing cache for directory '/original_models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 9726d31307db79567a1f1a863a8b9423.dir
Missing cache for directory '/models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 837f40add133f220032f2d43fd8f91d1.dir
Missing cache for directory '/original_models'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 938afceb0aa2859532678f14ede61b79.dir
Missing cache for directory '/big_dataset_only_remove_empty'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: fc0c116d205a776d1fd78796446df6e2.dir
Missing cache for directory '/custom-tokenizer-from-old'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 5f7cf0e6c325c2eeecc9d4458fdd4d3b.dir
Missing cache for directory '/data'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 17d6c38d7f28838eafb2217e16747aeb.dir
Missing cache for directory '/.\models\distillbert-big-dataset-baseline\training'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: e309eb997ca05aa54fd654e5f76d4c5a.dir
Missing cache for directory '/.\models\distillbert-big-dataset-baseline\evaluation'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: None, md5: 5fc384024f96196f95d2bd0f068da7a3.dir
Missing cache for directory '/raw_data'. Cache for files inside will be lost. Would you like to continue? Use '-f' to force. [y/n] y
ERROR: Failed to collect '2ce912ccc3fbb9b8bdf2f9331d72640b4b859c4a': unable to read: '..\..\dvc.lock', YAML file structure is corrupted: while scanning a simple key
in "<unicode string>", line 31, column 1
could not find expected ':'
in "<unicode string>", line 32, column 10Thanks for the clarifications.
I do dvc destroy inside the repository. Then I do dvc init. followed by dvc exp show and I still see some experiments from previous commits in the master and workspace. How is it possible I have an experiment instance in the workspace if I don't even have stages defined?
Ah, that's happening because these experiments are stored in git, not in dvc itself. Git stores all of its info in .git/, and DVC stores all of its info in .dvc/. dvc destroy will delete .dvc/ along with other DVC-related files, but it won't touch .git/, which is where experiment references are stored.
tldr you can drop those with dvc exp remove -A.
For instance, in a newly dvc instance (right after dvc destroy) trying to do dvc gc -A outputs:
dvc gc -A checks your entire Git history. You have previous commits in you Git history that still have dvc.yaml pipelines or .dvc files, so the warnings are related to those previous commits.
Also, note that dvc gc -A does not delete objects from all commits. It does the opposite, trying to delete only objects that are not associated with any commit (for example, objects generated by experiments). DVC is warning you that objects it expects to be there are not because you already destroyed all of the old cache during dvc destroy.
Hi, I just created a new dvc repo. I destroy my previous repo and init a new one. Then I simply create a new pipeline and run an experiment then when trying to apply the experiment I hit this issue.