sync only if files has changed

[raveslave]

sync adds a few seconds each time you run which can be problematic in some environments (host is a rpi so usb tend to be a bit slow)

would be nice if belay could compare and avoid an erase and rewrite each time.

another option would be a freeze command where a checksum file is generated and stored on both the device and host side

[BrianPugh]

The sync action checks the hash of the files-on-device and compares them to the ones on the computer, only syncing the files with differing hashes. At a high level:

1. The computer asks the microcontroller for the hash of all the files in the destination directory. Its not guarenteed that the microcontroller has a good hash function builtin, so we use the FNV1A hash function because it has good 32-bit performance, and a super simple implementation. Depending on the micropython features available, we send over either a viper, a native, or a vanilla micropython implementation.
2. The computer globs all the files in the provided folder and calculates the hash for all of them in parallel.
3. We compare the hashes, and push over the files that have a different hash.

Can you see if this line is taking up the majority of your sync time? If so, then we have 3 options:

1. We can have the opt-in checksum/hash you mentioned to quickly query the device about the files without having to rehash them. This would have to be opt-in because it could be very easy to have a stale checksum/hash if something external modifies the files. While this could work, I like options 2 & 3 below since they'd have less of a draw-back.
2. Optimize the on-device hashing. I could program up an FNV1a native module that should be able to get faster performance than the viper implementation. I think I'll do this regardless because it would be a fun minimal demo about how to structure a micropython native module project.
3. Query the device if it has hashlib.sha256 available. I haven't looked at the micropython source code, but many microcontrollers have a hardware-accelerated sha256 engine that is typically about an order of magnitude faster than a CPU implementation. In those scenarios, we could use the sha256 hash instead of the FNV1a hash.

[BrianPugh]

Another attractive option is to utilize the file modification time and only transfer files with differing timestamps. This way we wouldn't have to read the whole file on-device.

[raveslave]

thanks for clarifying around sync will get back where time is spent / profiling.

about using timestamps instead, should be as safe as a hash afaik, any corner cases? would def. be faster

[raveslave]

related question:
when using device subclassing @Device.setup, is there a similar mechanisem to avoid the full content of the file being transferred each time?

[BrianPugh]

when using device subclassing @Device.setup, is there a similar mechanisem to avoid the full content of the file being transferred each time?

The body of setup has to be sent everytime. If it is substantially large, you could put the contents in a file, have that sync over as part of your sync call, and then call it from your setup.

[BrianPugh]

I just published a fnv1a hash native module that is several times faster than the one used in this repo. So this may provide the speedup we desire with little downside/sharp-corners. @raveslave were you able to profile this in your project?

[BrianPugh]

@raveslave any updates?

[raveslave]

this is what takes the most time. we run on rpi5, but same on a fast pc https://github.com/BrianPugh/belay/blob/e76da2198341dd3bcd266708d576d414ca798dd2/belay/sync_preprocess.py#L74

spawning a new thread per file will eat some cycles, an async approach could be better

then it would be nice if the minify step also makes use of a freeze/hashed approach for speed

[BrianPugh]

can you share the files you are syncing? Just so I can fiddle with optimization locally.

Optimization options (as you mentioned):

1. Maybe the preprocess_src_file is really fast and we're wasting a bunch of time multithreading it. I wonder what the time different of this line is compared to just single-threading it.

   src_files_and_hashes = [_preprocess_src_file_hash_helper(x) for x in src_files]

2. Add some on-host caching; here we can certainly just check the before-processing-file timestamps (I was hesitant of doing this for on-device files because micropython understandably doesn't have strong guarantees about time/date). For all the ones that have not changed since the previous invocation, skip preprocessing and used a cached copy.

We need to profile more to see whats really taking up all the time. It could even be just disk io stuff since we're writing to a temporary folder.

[raveslave]

hi, files attached, agree that it is safer with some sort of hash over timestamps, hopefully the preprocess step can be made faster. suggest to test this from a rpi4 o r 5 if you have, the rpi tends to be unreasonable slow sometimes, especially usb transfers
slow-file-sync.zip

[BrianPugh]

testing this particular folder on my rpi setup, which is:

• pi400 (rpi4 with 4GB of ram) running raspian. I'm connected via SSH.
• pi pico running micropython 1.23.0

The initial sync took 1.58 seconds, with subsequent sync checks taking around 0.33 seconds. I'll look into seeing if there's a bottleneck, but this certainly seems faster than your initial report. Is there anything else unique about your setup?

[BrianPugh]

investigating a lil, for a sync where the files are already at the destination the breakdown is roughly:

1. ~33% of time spent minifying the code on host (which is subsequently hashed for comparison). This could maybe be faster if we cached minified outputs, but it could get messy.
2. ~33% of the time is spent querying the device about what hashes are on it's device. Swapping to my native implementation of the hash function might speed it up a tiny.
3. ~16% of the time is spent bootstrapping the sync process with sending over all the necessary micropython-side code.

I'll think about how we can optimize it further. I think some form of host-side caching is going to be the solution.

[raveslave]

I'll look into seeing if there's a bottleneck, but this certainly seems faster than your initial report. Is there anything else unique about your setup?

a few things

• we run chrome/chromium do visualise things on screen. latency/timing is much worse on usb in general when chromium is running. haven't dug into this, but indeed strange
• the micropython device is a full-speed mcu (rp2040), but we also have high-speed usb devices connected to the raspberry. this also seems to affect things in a bad way on the usb-host stack, most likely some slowness on the kernel-driver/setting when the rpi is switching speeds / fragmented usb packages
• our setup runs using trio, belay in a thread (can be changed)

[BrianPugh]

hm ok, this is all good to know. Later this week I'll prepare a few optimization PRs for you to try. If they improve things, we'll merge them in. If not, we'll not (no need to make the code more complicated for no benefit). I naively tried some of the ideas on my machine, but they had no impact. In your setup different elements of optimization may have more significant impact.