Actually, I'm an idiot. If you set your environment to Live, you'll get a proper response.
There's still some outstanding problems however:
If you access a path without an an extension, your .htaccess rule returns a forbidden response before you ever hit PHP, so you get the generic apache error then.
It's not clear why we're not serving a proper error page in DEV or TEST. It's not like "Not Found" on a blank page gives you a lot of information about what went wrong. And it will lead some people to think the error page is broken or very unusable.
Overview
silverstripe/assets
does not use the pretty error pages from https://github.com/silverstripe/silverstripe-errorpageInstead you get some very nondescript error message. This is pretty shitty user experience.