since the display of the 3ds is actually 800x240, with rectangular pixels, libtru has a function that makes the top screen render an image of that size, instead of the regular 400x240 with square pixels.
This seems to be a pretty straightforward change that definitely improves the quality of the non-integer scaled modes on the top screen. Put together a quick PR for this
example implementation