Open pcmantinker opened 9 years ago
Yes it works on ps3 but but ps4 is different, it does not use the .esfm file
Sent from my Samsung Galaxy smartphone.-------- Original message -------- From: Cameron Tinker notifications@github.com Date: 22/08/2015 06:03 (GMT+00:00) To: jhewt/gumer-psn gumer-psn@noreply.github.com Subject: [gumer-psn] Hidden trophy support with .ESFM decryption and hash generation (#17)
I've been doing some research regarding getting hidden trophy information. It looks like major trophy sites use the .ESFM file decryption and hash generation method to fill in all trophies for games in conjunction with the PlayStation mobile application/website. I found some sample code on the psdevwiki for downloading trophy related files: http://www.psdevwiki.com/ps3/Trophy_files#Tools
The trick is to find the HMAC key, erk, and iv for running the python script. I haven't been able to locate these just yet, but in theory, one should be able to get these from the metadata header on the np_trophy_util.sprx/prx file from a PS3 firmware extraction. After this is done, files should be able to download. I don't know how to decrypt ESFM files at the moment, but I think that there are tools available that can handle decryption. If anyone has pointers on how to extract the HMAC key, erk, and iv, I think we can make this part of the API to get hidden trophy information. Currently, it appears that larger organizations profit off of the mentioned method and it would be great to open source the method.
— Reply to this email directly or view it on GitHub.
It looks like the PS4 still uses .esfm files for trophies. http://www.ps3devwiki.com/ps4/Trophy00.trp I don't fully understand it, but it might be that the Trophy00.trp file is the container for the .esfm file and trophy images.
Where can I find a np_trophy_util.prx? I want to take a look at it.
I've only got a HMAC key from the PlayStation App and it's for gems
If you can successfully decrypt and unpack the CORE_OS_PACKAGE.pkg from a PS3 firmware update package (PUP), it should contain vshell modules and np_trophy_util.sprx is part of those modules. Also, if you search various places, you can find the already extracted CORE_OS_PACKAGE.pkg files used for building PS3 custom firmwares. I'm having troubles with the decryption of the np_trophy_util.sprx bit as I don't know how to setup the keys for use with ps3tools, scetool or similar though. After it's decrypted, I have a good feeling about being able to generate the hashes for downloading trophy resources. Then it's just a matter of decrypting .ESFM files from Sony. Images should be unencrypted from what I can tell.
I've found a few files already extracted from the page you linked in the first comment. However, this is quite hard. Following the route of "ICON0.PNG" to hash is so damn hard, I'll try to check the Jumptables. I've no idea what I'm doing, lol.
Yeah, IDA Pro is a bit beyond my level of expertise. I think that there should be tools/code available that can read the information we need without having to step through a debugger and disassembly of binaries. https://github.com/wargio/ps3tools Take a look specifically at https://github.com/wargio/ps3tools/blob/master/self.h#L167 and see that there is information in the struct for the hmac key. Also, https://github.com/wargio/ps3tools/blob/master/self.h#L134 has the METADATA_INFO structure defined and I think that is where we can get our erk and iv for AES encryption.
I was able to get this to compile using MinGW on Windows 10 x64, but I couldn't get anything to decrypt as of yet. I believe I have the keys I need to decrypt, but I don't know how to set them up at the moment. If you manage to get the keys setup for decryption, please let me know how. This way we can work on the solution together.
I've been able to unpack the lastest version (4.75) of PS3UPDAT.PUP from http://dus01.ps3.update.playstation.net. I've compiled the ps3tools in my VPS with Ubuntu 14.04 and used the keys published in this repository https://bitbucket.org/c0RoNa/ps3keys. The README have instructions on how to use them with ps3tools.
NOTE: You've to udpate the .sh script with the correct path to the BUILD folder
Then I simply ran:
jose@DevServer:~/ps3tools$ ./dev_flash_extractor_linux.sh ../ps3/PS3UPDAT.PUP
And I've got a folder dev_flash with the modules inside like so (only showing sprx files):
... more stuff vsh ├── etc │ └── print ├── module │ ........... more stuff │ ├── ap_plugin.sprx │ ├── audioplayer_plugin_dummy.sprx │ ├── musicbrowser_plugin.sprx │ ├── nas_plugin.sprx │ ├── netconf_plugin.sprx │ ├── newstore_effect.sprx │ ├── newstore_plugin.sprx │ ├── np_eula_plugin.sprx │ ├── np_matching_plugin.sprx │ ├── np_multisignin_plugin.sprx │ ├── np_oauth.sprx │ ├── npsignin_plugin.sprx │ ├── np_sns_plugin.sprx
│ ├── np_trophy_ingame.sprx │ ├── np_trophy_plugin.sprx │ ├── np_trophy_util.sprx
│ ├── oskfullkeypanel_plugin.sprx │ ├── oskpanel_plugin.sprx │ ├── osk_plugin.sprx │ ├── paf_ext.sprx │ ├── paf_psjs.sprx │ ├── paf_web.sprx │ ├── pesm_plugin.sprx │ ├── photolist_plugin.sprx │ ........... more stuff ... even more stuff
Once I've got np_trophy_util.sprx I've ran the tool readself2
jose@DevServer:~/ps3tools/BUILD$ ./readself2 /home/jose/ps3tools/dev_flash/dev_flash/vsh/module/np_trophy_util.sprx
And got this:
SELF Info header type: SELF SDK type: unknown app type: application arch type: PowerPC64 ELF type: unknown app version: 4.70.0 auth id type: UnknownAuthIdType SELF Header file offset data magic: 0000 = 53434500 header version: 0004 = 00000002 sdk type: 0008 = 001c header type: 000a = 0001 meta offset: 000c = 00000240 header length 0010 = 00000000_00000580 bytes file length: 0018 = 00000000_000707a0 bytes unknown: 0020 = 00000000_00000003 info offset: 0028 = 00000000_00000070 elf #1 offset: 0030 = 00000000_00000090 phdr offset: 0038 = 00000000_000000d0 shdr offset: 0040 = 00000000_00000000 sinfo offset: 0048 = 00000000_00000180 version offset: 0050 = 00000000_000001e0 control offset: 0058 = 00000000_000001f0 control length: 0060 = 00000000_00000070 bytes unknown: 0068 = 00000000_00000000 App Info Header file offset data auth id: 0070 = 10700000_52000001 (UnknownAuthIdType) vendor id: 0078 = 01000002 app type: 007c = 00000004 app version: 0080 = 00040070_00000000 unknown: 0088 = 00000000_00000000 ELF Header file Offset data ident: 0090 = 7f454c46_02020166 00000000_00000000 type: 00a0 = ffa4 (unknown) machine: 00a2 = 0015 (PowerPC64) version: 00a4 = 00000001 entry: 00a8 = 00000000_00000000 phdr offset: 00b0 = 00000000_00000040 shdr offset: 00b8 = 00000000_00000000 flags: 00c0 = 01000000 header size: 00c4 = 0040 bytes pheader size: 00c6 = 0038 bytes pheaders num: 00c8 = 0003 sheader size: 00ca = 0000 bytes sheaders num: 00cc = 0000 sheader str idx: 00ce = 0000 ELF PHDR file offset data pheader 00: type: 00d0 = 00000001 (LOAD) flags: 00d4 = 00400005 PPU:r-x SPE:r-- RSX:--- offset: 00d8 = 00000000_000000f0 vaddr: 00e0 = 00000000_00000000 paddr: 00e8 = 00000000_00052774 filesize: 00f0 = 00000000_00055118 bytes memsize: 00f8 = 00000000_00055118 bytes align: 0100 = 00000000_00000010 pheader 01: type: 0108 = 00000001 (LOAD) flags: 010c = 00600006 PPU:rw- SPE:rw- RSX:--- offset: 0110 = 00000000_00055280 vaddr: 0118 = 00000000_00055180 paddr: 0120 = 00000000_00000000 filesize: 0128 = 00000000_00001ef8 bytes memsize: 0130 = 00000000_00002bb0 bytes align: 0138 = 00000000_00000080 pheader 02: type: 0140 = 700000a4 (?????) flags: 0144 = 00000000 PPU:--- SPE:--- RSX:--- offset: 0148 = 00000000_00057180 vaddr: 0150 = 00000000_00000000 paddr: 0158 = 00000000_00000000 filesize: 0160 = 00000000_00019620 bytes memsize: 0168 = 00000000_00000000 bytes align: 0170 = 00000000_00000010 Section Info file offset data section 00: start offset: 0180 = 00000000_00000670 section size: 0188 = 00000000_00022b50 bytes compression: 0190 = [YES] unknown: 0194 = 00000000 unknown: 0198 = 00000000 encryption: 019c = [YES] section 01: start offset: 01a0 = 00000000_00023240 section size: 01a8 = 00000000_00000e98 bytes compression: 01b0 = [YES] unknown: 01b4 = 00000000 unknown: 01b8 = 00000000 encryption: 01bc = [YES] section 02: start offset: 01c0 = 00000000_000240e0 section size: 01c8 = 00000000_00005153 bytes compression: 01d0 = [YES] unknown: 01d4 = 00000000 unknown: 01d8 = 00000000 encryption: 01dc = [YES] SCE Version Info file offset data unknown: 01e0 = 00000001 unknown: 01e4 = 00000000 unknown: 01e8 = 00000010 unknown: 01ec = 00000000 Control Info file offset data control type: 01f0 = 0001 control length: 01f4 = 0030 unknown: 01f8 = 00000000_00000001 control flags: 0200 = 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 unknown: 0210 = 00000000_00000000 unknown: 0218 = 00000000_00000000 control type: 0220 = 0002 control length: 0224 = 0040 unknown: 0228 = 00000000_00000000 file digest: 0230 = 62 7c b1 80 8a b9 38 e3 2c 8c 09 17 08 72 6a 57 9e 25 86 e4 file digest: 0244 = d5 29 8f f6 eb be bd 5d b9 9a 39 cf 01 fa 0e 67 a8 20 d3 96 unknown: 0258 = 00000000_00000000 Metadata Info file offset data Key: 0260 = 77 81 73 16 1a e6 19 18 ae 41 9e cf 10 fb f0 24 IV : 0280 = 1c 8e 02 39 27 b3 92 7c f1 b7 01 21 1f ad df 5e Metadata Header file offset data Signature end: 02a4 = 00000500 Sections: 02ac = 3 Keys: 02b0 = 24 Metadata Sections file offset Data section 00: Offset: 02c0 = 00000000_00000670 Length: 02c8 = 00000000_00022b50 bytes Key: 02e4 = 0006 IV: 02e8 = 0007 SHA1: 02dc = 0000 Type: 02d0 = 0002 section 01: Offset: 02f0 = 00000000_00023240 Length: 02f8 = 00000000_00000e98 bytes Key: 0314 = 000e IV: 0318 = 000f SHA1: 030c = 0008 Type: 0300 = 0002 section 02: Offset: 0320 = 00000000_000240e0 Length: 0328 = 00000000_00005153 bytes Key: 0344 = 0016 IV: 0348 = 0017 SHA1: 033c = 0010 Type: 0330 = 0002 Metadata Keys file offset Data key idx 0000: 0350 = 55 1f 96 c0 b5 bb 8d 46 b1 14 d5 77 31 12 bb 75 key idx 0001: 0360 = ff 9a f7 e1 00 00 00 00 00 00 00 00 00 00 00 00 key idx 0002: 0370 = 22 4d cc 4c 09 a3 02 45 1e 95 80 ee 16 f5 eb be key idx 0003: 0380 = 3f 88 c9 66 82 5b 4f 8b 7f e6 8a 97 89 12 fd 89 key idx 0004: 0390 = 9d 04 23 da f1 0a c9 77 85 9f 5b 5b 3b 48 72 f0 key idx 0005: 03a0 = 6a 3f e9 25 f6 85 8b 0c d2 93 d9 1a 2d 94 fe 58 key idx 0006: 03b0 = 78 b0 ad 01 c2 cc e2 92 64 27 29 14 54 b4 b9 bb key idx 0007: 03c0 = 08 b9 a2 ff 60 23 ac 8c 0b 3f da f7 00 00 00 00 key idx 0008: 03d0 = 6f b6 fe 95 53 82 ed 79 c7 d4 52 91 1d cc 25 df key idx 0009: 03e0 = 34 b1 04 e7 00 00 00 00 00 00 00 00 00 00 00 00 key idx 000a: 03f0 = 22 4d cc 4c 09 a3 02 45 1e 95 80 ee 16 f5 eb be key idx 000b: 0400 = 3f 88 c9 66 82 5b 4f 8b 7f e6 8a 97 89 12 fd 89 key idx 000c: 0410 = 9d 04 23 da f1 0a c9 77 85 9f 5b 5b 3b 48 72 f0 key idx 000d: 0420 = 6a 3f e9 25 f6 85 8b 0c d2 93 d9 1a 2d 94 fe 58 key idx 000e: 0430 = ae 95 a2 6e 90 1b d9 39 7e a3 06 0f fe 61 6e 56 key idx 000f: 0440 = fe 2b a1 16 f6 fd e9 ac 82 10 f6 6c 00 00 00 00 key idx 0010: 0450 = 09 50 1a cc e7 7b 63 dc 8c 4b 91 94 04 be ff 5f key idx 0011: 0460 = c6 2f ba 12 00 00 00 00 00 00 00 00 00 00 00 00 key idx 0012: 0470 = 22 4d cc 4c 09 a3 02 45 1e 95 80 ee 16 f5 eb be key idx 0013: 0480 = 3f 88 c9 66 82 5b 4f 8b 7f e6 8a 97 89 12 fd 89 key idx 0014: 0490 = 9d 04 23 da f1 0a c9 77 85 9f 5b 5b 3b 48 72 f0 key idx 0015: 04a0 = 6a 3f e9 25 f6 85 8b 0c d2 93 d9 1a 2d 94 fe 58 key idx 0016: 04b0 = 3d ac 48 3b b3 7d 72 12 6c c9 c7 d3 39 81 cf f2 key idx 0017: 04c0 = 38 aa 15 97 c5 07 c0 5a 28 e4 fa e6 00 00 00 00 ELF SHDRs No ELF Section Headers in this file.
Yet none of those are the HMAC key for the trophies, those are for decrypting the file itself.
So, I've ran unself2 to np_trophy_util.sprx to convert it to ELF. And once again, I get the same file in the ZIP linked in ps3wiki in your first comment.
I've been able to seek for the keys already shown in ps3wiki. Still, no luck on finding the hmac key that generates the URL nor the key for decrypting. I've not been able to find at least the hinted hex in the file using IDA either.
Ah, from what you're saying, it seems I misunderstood the metadata header information and the hmac key from selfs. I may just have to byte (pun intended lol) the bullet and dig into IDA Pro at some point to further investigate.
Lol, yeah it sucks. Too bad I do not own a PS3 to dump it's memory when it's loading hidden trophies. There are tools to dump it and debug it right on IDA Pro
If you guys need it still, I have all the necessary keys.
I've been looking at this independently and pretty much hit the same point as above - with the sprx/prx loaded into IDA, but nothing that looks like or works as the HMAC.
@Red-EyeX32, is the offer still on the table for the necessary keys, or alternatively a push in the right direction?
@TJangles Send me your skype and I'll be more that happy to help out.
I just realised I never dropped an update here and I've started looking into this again. Firstly, thanks to @Red-EyeX32 - with their help I managed to get a script working to create valid URLs for game images, trophy images, and the trop.esfm (which are subsequently decrypted and provide the hidden trophy details!).
The next hurdle seems to be figuring out what's changed for newer titles (post-2013?). Taking npcommids for recent titles (ie NPWR07817_00
- a PS4 game) and running them through the same code results in 404's. This suggests either the keys differ or, if that isn't the case, the files reside at a different server path.
Any input would be much appreciated.
I just realised I never dropped an update here and I've started looking into this again. Firstly, thanks to @Red-EyeX32 - with their help I managed to get a script working to create valid URLs for game images, trophy images, and the trop.esfm (which are subsequently decrypted and provide the hidden trophy details!).
The next hurdle seems to be figuring out what's changed for newer titles (post-2013?). Taking npcommids for recent titles (ie
NPWR07817_00
- a PS4 game) and running them through the same code results in 404's. This suggests either the keys differ or, if that isn't the case, the files reside at a different server path.Any input would be much appreciated.
are you still around or is redeye im looking for the update erk and hmac keys for ps3 trophies
Not to necro, but this issue is key for PSN emulation regarding decryption on Trophy .ESFM files, any word on this @TJangles ? I am VERY interested.
I've been doing some research regarding getting hidden trophy information. It looks like major trophy sites use the .ESFM file decryption and hash generation method to fill in all trophies for games in conjunction with the PlayStation mobile application/website. I found some sample code on the psdevwiki for downloading trophy related files: http://www.psdevwiki.com/ps3/Trophy_files#Tools The trick is to find the HMAC key, erk, and iv for running the python script. I haven't been able to locate these just yet, but in theory, one should be able to get these from the metadata header on the np_trophy_util.sprx/prx file from a PS3 firmware extraction. After this is done, files should be able to download. I don't know how to decrypt ESFM files at the moment, but I think that there are tools available that can handle decryption. If anyone has pointers on how to extract the HMAC key, erk, and iv, I think we can make this part of the API to get hidden trophy information. Currently, it appears that larger organizations profit off of the mentioned method and it would be great to open source the method.