Copying takes too long - Githubissues

andriygm commented 1 year ago

look into why file.close takes so long
make sure we're asynchronously logging file copy updates

andriygm commented 1 year ago

@dpastoor am using a progress bar package now which seems to have negligible performance impact, but am not sure how to optimize the file sync operation

dpastoor commented 1 year ago

nice - some benchmarking:

previous behavior:

Devins-MacBook-Pro:test1 devin$ time dvs add test_data.csv 
🌀 Devious

1/1   test_data.csv
    Generating hash... ✔
    Writing file... ✔
    Cleaning up... ✔
    Adding gitignore entry... ✔

real    0m28.615s
user    0m1.002s
sys     0m1.107s

compared to the fastest "possible" reference time to raw copy:

Devins-MacBook-Pro:test1 devin$ time cp test_data.csv test_data2.csv

real    0m0.380s
user    0m0.001s
sys     0m0.150s

Time with the fix in fb7ec72 :

Devins-MacBook-Pro:test1 devin$ time dvs add test_data.csv 
🌀 Devious

1/1   test_data.csv
    Generating hash... ✔
    Writing file... ✔
    Cleaning up... ✔
    Adding gitignore entry... ✔

real    0m1.958s
user    0m0.807s
sys     0m0.408s

yep looks like that flushing of the status each time was blocking. Fix looks great!

A2-ai / devious

Copying takes too long #20