search

commaai / flash

https://flash.comma.ai
5 stars 14 forks source link

refactor: download and decompress xz images #29

Closed samrum closed 7 months ago

samrum commented 7 months ago

For #5

The manifest tests pass and the worker intializes on page load without error, but didn't test whether flashes actually go through.

Wasn't sure if the issue also meant to combine the download and unpacking steps, but that would be possible.

adeebshihadeh commented 7 months ago

@jnewb1 can you test this and merge?

jnewb1 commented 7 months ago

[fastboot] Unpack error Error unpacking archive: TypeError: (intermediate value).getReader is not a function

jnewb1 commented 7 months ago

flash_log.txt

attached the console log

adeebshihadeh commented 7 months ago

@samrum can you update the test to catch this too?

samrum commented 7 months ago

Ah, oops, I'm missing a body before the getReader call. I'll add an image worker test for unpackImage which would catch that.

samrum commented 7 months ago

Fixed and updated the manifest tests to use the image worker for validating images

jnewb1 commented 7 months ago

goes over 100% now image

edit: it went up to like 500%, then went back to 100% and got stuck image

jnewb1 commented 7 months ago

flash_log_2.txt

jnewb1 commented 7 months ago

blob.js:20 [blob] Download size mismatch {url: 'https://raw.githubusercontent.com/commaai/openpilot/master/system/hardware/tici/agnos.json', expected: 742, actual: 3158}

samrum commented 7 months ago

Sorry for the trouble all, I was avoiding testing the actual flashing because I didn't want to mess up my 3x, but actually went through and properly tested it this time.

For the incorrect progress issue: I pushed a fix for the way progress is calculated when unpacking the xz images since it should be using the actual image size instead of the compressed archive size. ~It won't be fully fixed until https://github.com/commaai/openpilot/pull/31154 is merged and we have the system alt size available to use for that image.~ It's fixed now!

For the system alt image getting stuck after unpacking: It seems like the system alt image's final writable.close() call is dying without throwing an error somehow. Looked into it a bit and not sure why it's happening. It could be related to how long the xz archive takes to decompress, but not sure.

For the agnos.json size mismatch: This seems like a preexisting issue on https://flash.comma.ai. It's comparing the content length header from the compressed GitHub response to the uncompressed blob.

samrum commented 7 months ago

Should be good now. Tested up to the actual flashing step and verified that the images grabbed for flashing match their expected sizes.

adeebshihadeh commented 7 months ago

CI is failing - https://github.com/commaai/flash/actions/runs/7664903685/job/20907213255?pr=29

samrum commented 7 months ago

Seems like the fetch for the image failed for some reason. It's fine when I run them locally. Does it happen if you re-run the job?

adeebshihadeh commented 7 months ago

It works, thanks! I'm planning on putting up some more bounties for this once #7 is done if you'd be interested, such as streaming the system image to require less memory (like openpilot does).