openwrt / asu

An image on demand server for OpenWrt based distributions
https://sysupgrade.openwrt.org
GNU General Public License v2.0
321 stars 80 forks source link

util: improve package file parsing #991

Closed efahl closed 2 days ago

efahl commented 4 days ago

Occasionally, upstream package files are being terminated with extra newlines, resulting in '"null": null' entries in the resulting package index. Strip the input text to avoid this.

Change the splitting method on the text to both reduce memory footprint and speed things up.

I reported this back in https://github.com/openwrt/asu/issues/866, but it was transient and I never got to the bottom of it. Fixed now.

efahl commented 4 days ago

Just discovered that adding headersonly=True to the parsestr call speeds the function up by about 10% overall, so put that in, too.

10 runs:
headersonly  :   1.812119s ( 88.8%)
original     :   2.039979s (100.0%)
codecov[bot] commented 2 days ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 89.43%. Comparing base (5e65dec) to head (db5b9f9). Report is 106 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #991 +/- ## ========================================== + Coverage 80.75% 89.43% +8.68% ========================================== Files 15 14 -1 Lines 977 1032 +55 ========================================== + Hits 789 923 +134 + Misses 188 109 -79 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.