This seems like a good idea but it's not. When you request the raw file like that, you're not directly accessing the file from the filesystem! You're going through layers of application code too, which will definitely slow your site down. Don't do that. Instead, create a gh-pages branch, and load it from there like this:
http://code.lancepollard.com/github-as-a-cdn/
https://pages.github.com/