Once you upload your first custom favicon, it appears that the original one is gone. It doesn't have a backup (like other uploaded images). You may be able to refresh the favicon back to the original by changing the theme (although I haven't tested this yet), but even so, that shouldn't be necessary to get the old favicon back. This may also be an issue with logos and headers, but I didn't notice it.
Once you upload your first custom favicon, it appears that the original one is gone. It doesn't have a backup (like other uploaded images). You may be able to refresh the favicon back to the original by changing the theme (although I haven't tested this yet), but even so, that shouldn't be necessary to get the old favicon back. This may also be an issue with logos and headers, but I didn't notice it.