Caching mechanics was broken in certain scenarios and affected DVC tremendously. Before we were assuming pre-cached paths for 00, 01 .... ff in DVC. It got broken during DVC 3.0 release, and then got removed completely because of this. Now it's not the case and DVC was kept asking for files, files\md5, etc - all those IDs again and again and again on dvc pull. Pretty much every object could be 3x slower to download.
Among other things, I checked that dvc pull works now even with duplicated root dirs in DVC remote.
Still need to self review this, and next see if we can add tests for all of this. It keeps biting us. Caching is non trivial and we cant be losing the knowledge as we go, every time coming back to fix it after users complain.
Caching mechanics was broken in certain scenarios and affected DVC tremendously. Before we were assuming pre-cached paths for
00
,01
....ff
in DVC. It got broken during DVC 3.0 release, and then got removed completely because of this. Now it's not the case and DVC was kept asking forfiles
,files\md5
, etc - all those IDs again and again and again ondvc pull
. Pretty much every object could be 3x slower to download.Among other things, I checked that
dvc pull
works now even with duplicated root dirs in DVC remote.Still need to self review this, and next see if we can add tests for all of this. It keeps biting us. Caching is non trivial and we cant be losing the knowledge as we go, every time coming back to fix it after users complain.