During testing of the suspend/resume feature, I noticed that there is a potential multiplication warp-around in Area::count. The multiplication of width and height values is done as unsigned so that the multiplication can wrap-around and return an incorrect and too small value. The incorrect calculation of Area::count in this case fooled the RAM-quota check in nitpicker.
Note that b939358c36799d1f08a3e965831d8663d76e7043 already fixed the root-cause that triggered this issue (i.e. very large width/height) in the first place.
jschlatow
commented
2 weeks ago
During testing of the suspend/resume feature, I noticed that there is a potential multiplication warp-around in
Area::count. The multiplication of width and height values is done asunsignedso that the multiplication can wrap-around and return an incorrect and too small value. The incorrect calculation ofArea::countin this case fooled the RAM-quota check in nitpicker.Note that b939358c36799d1f08a3e965831d8663d76e7043 already fixed the root-cause that triggered this issue (i.e. very large width/height) in the first place.