search

balena-os / leviathan-worker

The worker layer for Leviathan including worker plugins QemuKit & AutoKit
Apache License 2.0
6 stars 4 forks source link

unzip image after it is sent #101

Closed rcooke-warwick closed 5 months ago

rcooke-warwick commented 5 months ago

We have many failures when provisioning devices at the point where we send the image over the public url. Currently to save space and time, we unzip the gzipped image as part of the request - it was part of the pipeline() from the request to the file.

The issue with that is that there seems to be some hidden timeout on the balena cloud public URL - if a file takes longer than some amount of time (close to 5 mins anecdotally) to send, then the connection closes and it fails. Unzipping the image as part of the recieving pipeline seems to keep that pipe open for longer, making a timeout more likely. Empircally tested, seperating the transfer of the file and the unzip seems to improve things.

There is a corresponding leviathan core update that zips the image before sending, rather than during also, to save additional time.

These changes should be backwards compatible both ways. A gzipped file is still sent and recieved no matter what.

Change-type: patch

rcooke-warwick commented 5 months ago

addressed the comments - will now merge to unblock

rcooke-warwick commented 5 months ago

@balena-ci I self-certify!