Open raveslave opened 4 months ago
The sync
action checks the hash of the files-on-device and compares them to the ones on the computer, only syncing the files with differing hashes. At a high level:
Can you see if this line is taking up the majority of your sync time? If so, then we have 3 options:
hashlib.sha256
available. I haven't looked at the micropython source code, but many microcontrollers have a hardware-accelerated sha256 engine that is typically about an order of magnitude faster than a CPU implementation. In those scenarios, we could use the sha256 hash instead of the FNV1a hash.Another attractive option is to utilize the file modification time and only transfer files with differing timestamps. This way we wouldn't have to read the whole file on-device.
thanks for clarifying around sync
will get back where time is spent / profiling.
about using timestamps instead, should be as safe as a hash afaik, any corner cases? would def. be faster
related question:
when using device subclassing @Device.setup
, is there a similar mechanisem to avoid the full content of the file being transferred each time?
when using device subclassing @Device.setup, is there a similar mechanisem to avoid the full content of the file being transferred each time?
The body of setup
has to be sent everytime. If it is substantially large, you could put the contents in a file, have that sync over as part of your sync
call, and then call it from your setup
.
I just published a fnv1a hash native module that is several times faster than the one used in this repo. So this may provide the speedup we desire with little downside/sharp-corners. @raveslave were you able to profile this in your project?
@raveslave any updates?
this is what takes the most time. we run on rpi5, but same on a fast pc https://github.com/BrianPugh/belay/blob/e76da2198341dd3bcd266708d576d414ca798dd2/belay/sync_preprocess.py#L74
spawning a new thread per file will eat some cycles, an async approach could be better
then it would be nice if the minify step also makes use of a freeze/hashed approach for speed
can you share the files you are syncing? Just so I can fiddle with optimization locally.
Optimization options (as you mentioned):
preprocess_src_file
is really fast and we're wasting a bunch of time multithreading it. I wonder what the time different of this line is compared to just single-threading it.
src_files_and_hashes = [_preprocess_src_file_hash_helper(x) for x in src_files]
We need to profile more to see whats really taking up all the time. It could even be just disk io stuff since we're writing to a temporary folder.
hi, files attached, agree that it is safer with some sort of hash over timestamps, hopefully the preprocess step can be made faster. suggest to test this from a rpi4 o r 5 if you have, the rpi tends to be unreasonable slow sometimes, especially usb transfers
slow-file-sync.zip
testing this particular folder on my rpi setup, which is:
The initial sync took 1.58 seconds, with subsequent sync checks taking around 0.33 seconds. I'll look into seeing if there's a bottleneck, but this certainly seems faster than your initial report. Is there anything else unique about your setup?
investigating a lil, for a sync where the files are already at the destination the breakdown is roughly:
I'll think about how we can optimize it further. I think some form of host-side caching is going to be the solution.
I'll look into seeing if there's a bottleneck, but this certainly seems faster than your initial report. Is there anything else unique about your setup?
a few things
hm ok, this is all good to know. Later this week I'll prepare a few optimization PRs for you to try. If they improve things, we'll merge them in. If not, we'll not (no need to make the code more complicated for no benefit). I naively tried some of the ideas on my machine, but they had no impact. In your setup different elements of optimization may have more significant impact.
sync adds a few seconds each time you run which can be problematic in some environments (host is a rpi so usb tend to be a bit slow)
would be nice if belay could compare and avoid an erase and rewrite each time.
another option would be a freeze command where a checksum file is generated and stored on both the device and host side