App: create overrides folder if it doesn't exist

[danirabbit]

And make these keyfile warnings debug only since it seems like these files are being created and modified just fine