Closed ermolaev94 closed 1 month ago
@ermolaev94 is it the same error if you run with dvc push --no-run-cache
? (split dvc exp push
into git push
+ dvc push
as Dave suggested)?
File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/stage/cache.py", line 254, in transfer
for src in from_fs.find(runs):
File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 529, in find
yield from self.fs.find(path)
here it is actually trying to push run cache (that AFAIU you don't need), so it' related to https://github.com/iterative/dvc/issues/10449 most likely
Also, an improvement can be made in PyDrive2 to split the files into batches when it runs a query to check their existence. I'll look into this.
@ermolaev94 could you check also the <google_drive>/<dvc_cache/runs
directory? does it exists? how many subdirectories inside each subdirectory there?
@ermolaev94 also, could try this https://github.com/iterative/dvc/issues/10449#issuecomment-2229242286 please?
@ermolaev94 btw, could you please share some details (can DM ivan @ dvc.ai ) about your use case please - ~5Tb and ~150k files is quite impressive. I never thought it was possible with Google Drive tbh :). It would be great to catch up and lear more. Let me know or share your coordinates so that I can reach out.
@ermolaev94 is it the same error if you run with
dvc push --no-run-cache
? (splitdvc exp push
intogit push
+dvc push
as Dave suggested)?File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc/stage/cache.py", line 254, in transfer for src in from_fs.find(runs): File "/home/ermolaev/projects/radml/venv/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 529, in find yield from self.fs.find(path)
here it is actually trying to push run cache (that AFAIU you don't need), so it' related to iterative/dvc#10449 most likely
Also, an improvement can be made in PyDrive2 to split the files into batches when it runs a query to check their existence. I'll look into this.
Yes, it works! Thank you! You are right, looks like it' related to https://github.com/iterative/dvc/issues/10449
@ermolaev94 could you check also the
<google_drive>/<dvc_cache/runs
directory? does it exists? how many subdirectories inside each subdirectory there?
There are totally 702 files and 942 directories.
Closing a as duplicate for now of that issue. Thanks @ermolaev94 for confirming. We are looking into it.
Closing a as duplicate for now of that issue. Thanks @ermolaev94 for confirming. We are looking into it.
ok, thank you!
btw, I wrote you an email on box, that you've mentioned about our use case
Context
My project is huge, containing several different ML directions. Each direction has several pipilines desccribed in
dvc.yaml
. There are several different datasets, some of them more than 10Tb.Issue has appeared when I tried to push particular experiment data. Command
exp push
tries to cope with all .dvc files in the repo. And I have several hundreds of them. Some of them are stored on one remote, while other stored at all remotes. Logic is different from task to task. Some of the .dvc are missing on all remotes and it's ok for me now, because they are still pushing (25Tb on S3).The main remote is gdrive. It's a folder on team drive. Now its size ~5Tb and ~150k files. So we are not very close to the limits.
Error
Pushing experiment raises the following error:
Interesting that since that I've started getting the same error with even easy commands that used to work ealier:
I've tried to remove
.cache/pydrive2
to reauth. It didn't help.Additional Details
I've asked a question firstly, so other details can be seen here: https://discuss.dvc.org/t/error-in-pushing-experiments/2138/10