Hidden trophy support with .ESFM decryption and hash generation

[pcmantinker]

I've been doing some research regarding getting hidden trophy information. It looks like major trophy sites use the .ESFM file decryption and hash generation method to fill in all trophies for games in conjunction with the PlayStation mobile application/website. I found some sample code on the psdevwiki for downloading trophy related files: http://www.psdevwiki.com/ps3/Trophy_files#Tools The trick is to find the HMAC key, erk, and iv for running the python script. I haven't been able to locate these just yet, but in theory, one should be able to get these from the metadata header on the np_trophy_util.sprx/prx file from a PS3 firmware extraction. After this is done, files should be able to download. I don't know how to decrypt ESFM files at the moment, but I think that there are tools available that can handle decryption. If anyone has pointers on how to extract the HMAC key, erk, and iv, I think we can make this part of the API to get hidden trophy information. Currently, it appears that larger organizations profit off of the mentioned method and it would be great to open source the method.

[xXREDXIIIIXx]

Yes it works on ps3 but but ps4 is different, it does not use the .esfm file

Sent from my Samsung Galaxy smartphone.-------- Original message -------- From: Cameron Tinker notifications@github.com Date: 22/08/2015 06:03 (GMT+00:00) To: jhewt/gumer-psn gumer-psn@noreply.github.com Subject: [gumer-psn] Hidden trophy support with .ESFM decryption and hash generation (#17)

I've been doing some research regarding getting hidden trophy information. It looks like major trophy sites use the .ESFM file decryption and hash generation method to fill in all trophies for games in conjunction with the PlayStation mobile application/website. I found some sample code on the psdevwiki for downloading trophy related files: http://www.psdevwiki.com/ps3/Trophy_files#Tools

The trick is to find the HMAC key, erk, and iv for running the python script. I haven't been able to locate these just yet, but in theory, one should be able to get these from the metadata header on the np_trophy_util.sprx/prx file from a PS3 firmware extraction. After this is done, files should be able to download. I don't know how to decrypt ESFM files at the moment, but I think that there are tools available that can handle decryption. If anyone has pointers on how to extract the HMAC key, erk, and iv, I think we can make this part of the API to get hidden trophy information. Currently, it appears that larger organizations profit off of the mentioned method and it would be great to open source the method.

— Reply to this email directly or view it on GitHub.

[pcmantinker]

It looks like the PS4 still uses .esfm files for trophies. http://www.ps3devwiki.com/ps4/Trophy00.trp I don't fully understand it, but it might be that the Trophy00.trp file is the container for the .esfm file and trophy images.

[jhewt]

Where can I find a np_trophy_util.prx? I want to take a look at it. I've only got a HMAC key from the PlayStation App and it's for gems

[pcmantinker]

If you can successfully decrypt and unpack the CORE_OS_PACKAGE.pkg from a PS3 firmware update package (PUP), it should contain vshell modules and np_trophy_util.sprx is part of those modules. Also, if you search various places, you can find the already extracted CORE_OS_PACKAGE.pkg files used for building PS3 custom firmwares. I'm having troubles with the decryption of the np_trophy_util.sprx bit as I don't know how to setup the keys for use with ps3tools, scetool or similar though. After it's decrypted, I have a good feeling about being able to generate the hashes for downloading trophy resources. Then it's just a matter of decrypting .ESFM files from Sony. Images should be unencrypted from what I can tell.

[jhewt]

I've found a few files already extracted from the page you linked in the first comment. However, this is quite hard. Following the route of "ICON0.PNG" to hash is so damn hard, I'll try to check the Jumptables. I've no idea what I'm doing, lol.

[pcmantinker]

Yeah, IDA Pro is a bit beyond my level of expertise. I think that there should be tools/code available that can read the information we need without having to step through a debugger and disassembly of binaries. https://github.com/wargio/ps3tools Take a look specifically at https://github.com/wargio/ps3tools/blob/master/self.h#L167 and see that there is information in the struct for the hmac key. Also, https://github.com/wargio/ps3tools/blob/master/self.h#L134 has the METADATA_INFO structure defined and I think that is where we can get our erk and iv for AES encryption.

I was able to get this to compile using MinGW on Windows 10 x64, but I couldn't get anything to decrypt as of yet. I believe I have the keys I need to decrypt, but I don't know how to set them up at the moment. If you manage to get the keys setup for decryption, please let me know how. This way we can work on the solution together.

[jhewt]

I've been able to unpack the lastest version (4.75) of PS3UPDAT.PUP from http://dus01.ps3.update.playstation.net. I've compiled the ps3tools in my VPS with Ubuntu 14.04 and used the keys published in this repository https://bitbucket.org/c0RoNa/ps3keys. The README have instructions on how to use them with ps3tools.

NOTE: You've to udpate the .sh script with the correct path to the BUILD folder

Then I simply ran:

jose@DevServer:~/ps3tools$ ./dev_flash_extractor_linux.sh ../ps3/PS3UPDAT.PUP

And I've got a folder dev_flash with the modules inside like so (only showing sprx files):

... more stuff
 vsh
        ├── etc
        │   └── print
        ├── module
        │   ........... more stuff
        │   ├── ap_plugin.sprx
        │   ├── audioplayer_plugin_dummy.sprx
        │   ├── musicbrowser_plugin.sprx
        │   ├── nas_plugin.sprx
        │   ├── netconf_plugin.sprx
        │   ├── newstore_effect.sprx
        │   ├── newstore_plugin.sprx
        │   ├── np_eula_plugin.sprx
        │   ├── np_matching_plugin.sprx
        │   ├── np_multisignin_plugin.sprx
        │   ├── np_oauth.sprx
        │   ├── npsignin_plugin.sprx
        │   ├── np_sns_plugin.sprx

        │   ├── np_trophy_ingame.sprx
        │   ├── np_trophy_plugin.sprx
        │   ├── np_trophy_util.sprx

        │   ├── oskfullkeypanel_plugin.sprx
        │   ├── oskpanel_plugin.sprx
        │   ├── osk_plugin.sprx
        │   ├── paf_ext.sprx
        │   ├── paf_psjs.sprx
        │   ├── paf_web.sprx
        │   ├── pesm_plugin.sprx
        │   ├── photolist_plugin.sprx
        │   ........... more stuff
... even more stuff

Once I've got np_trophy_util.sprx I've ran the tool readself2

jose@DevServer:~/ps3tools/BUILD$ ./readself2 /home/jose/ps3tools/dev_flash/dev_flash/vsh/module/np_trophy_util.sprx

And got this:

SELF Info
  header type:     SELF
  SDK type:        unknown
  app type:        application
  arch type:       PowerPC64
  ELF type:        unknown
  app version:     4.70.0
  auth id type:    UnknownAuthIdType
SELF Header        file
                  offset  data
  magic:           0000 = 53434500
  header version:  0004 = 00000002
  sdk type:        0008 =     001c
  header type:     000a =     0001
  meta offset:     000c = 00000240
  header length    0010 = 00000000_00000580 bytes
  file length:     0018 = 00000000_000707a0 bytes
  unknown:         0020 = 00000000_00000003
  info offset:     0028 = 00000000_00000070
  elf #1 offset:   0030 = 00000000_00000090
  phdr offset:     0038 = 00000000_000000d0
  shdr offset:     0040 = 00000000_00000000
  sinfo offset:    0048 = 00000000_00000180
  version offset:  0050 = 00000000_000001e0
  control offset:  0058 = 00000000_000001f0
  control length:  0060 = 00000000_00000070 bytes
  unknown:         0068 = 00000000_00000000
App Info Header    file
                  offset  data
  auth id:         0070 = 10700000_52000001 (UnknownAuthIdType)
  vendor id:       0078 = 01000002
  app type:        007c = 00000004
  app version:     0080 = 00040070_00000000
  unknown:         0088 = 00000000_00000000
ELF Header         file
                  Offset  data
  ident:           0090 = 7f454c46_02020166 00000000_00000000
  type:            00a0 = ffa4 (unknown)
  machine:         00a2 = 0015 (PowerPC64)
  version:         00a4 = 00000001
  entry:           00a8 = 00000000_00000000
  phdr offset:     00b0 = 00000000_00000040
  shdr offset:     00b8 = 00000000_00000000
  flags:           00c0 = 01000000
  header size:     00c4 = 0040 bytes
  pheader size:    00c6 = 0038 bytes
  pheaders num:    00c8 = 0003
  sheader size:    00ca = 0000 bytes
  sheaders num:    00cc = 0000
  sheader str idx: 00ce = 0000
ELF PHDR           file
                  offset  data
  pheader 00:
    type:          00d0 = 00000001 (LOAD)
    flags:         00d4 = 00400005  PPU:r-x  SPE:r--  RSX:---
    offset:        00d8 = 00000000_000000f0
    vaddr:         00e0 = 00000000_00000000
    paddr:         00e8 = 00000000_00052774
    filesize:      00f0 = 00000000_00055118 bytes
    memsize:       00f8 = 00000000_00055118 bytes
    align:         0100 = 00000000_00000010
  pheader 01:
    type:          0108 = 00000001 (LOAD)
    flags:         010c = 00600006  PPU:rw-  SPE:rw-  RSX:---
    offset:        0110 = 00000000_00055280
    vaddr:         0118 = 00000000_00055180
    paddr:         0120 = 00000000_00000000
    filesize:      0128 = 00000000_00001ef8 bytes
    memsize:       0130 = 00000000_00002bb0 bytes
    align:         0138 = 00000000_00000080
  pheader 02:
    type:          0140 = 700000a4 (?????)
    flags:         0144 = 00000000  PPU:---  SPE:---  RSX:---
    offset:        0148 = 00000000_00057180
    vaddr:         0150 = 00000000_00000000
    paddr:         0158 = 00000000_00000000
    filesize:      0160 = 00000000_00019620 bytes
    memsize:       0168 = 00000000_00000000 bytes
    align:         0170 = 00000000_00000010
Section Info       file
                  offset  data
  section 00:
    start offset:  0180 = 00000000_00000670
    section size:  0188 = 00000000_00022b50 bytes
    compression:   0190 = [YES]
    unknown:       0194 = 00000000
    unknown:       0198 = 00000000
    encryption:    019c = [YES]
  section 01:
    start offset:  01a0 = 00000000_00023240
    section size:  01a8 = 00000000_00000e98 bytes
    compression:   01b0 = [YES]
    unknown:       01b4 = 00000000
    unknown:       01b8 = 00000000
    encryption:    01bc = [YES]
  section 02:
    start offset:  01c0 = 00000000_000240e0
    section size:  01c8 = 00000000_00005153 bytes
    compression:   01d0 = [YES]
    unknown:       01d4 = 00000000
    unknown:       01d8 = 00000000
    encryption:    01dc = [YES]
SCE Version Info   file
                  offset  data
  unknown:         01e0 = 00000001
  unknown:         01e4 = 00000000
  unknown:         01e8 = 00000010
  unknown:         01ec = 00000000
Control Info       file
                  offset  data
  control type:    01f0 = 0001
  control length:  01f4 = 0030
  unknown:         01f8 = 00000000_00000001
  control flags:   0200 = 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  unknown:         0210 = 00000000_00000000
  unknown:         0218 = 00000000_00000000
  control type:    0220 = 0002
  control length:  0224 = 0040
  unknown:         0228 = 00000000_00000000
  file digest:     0230 = 62 7c b1 80 8a b9 38 e3 2c 8c 09 17 08 72 6a 57 9e 25 86 e4
  file digest:     0244 = d5 29 8f f6 eb be bd 5d b9 9a 39 cf 01 fa 0e 67 a8 20 d3 96
  unknown:         0258 = 00000000_00000000
Metadata Info      file
                  offset  data
  Key:             0260 = 77 81 73 16 1a e6 19 18 ae 41 9e cf 10 fb f0 24
  IV :             0280 = 1c 8e 02 39 27 b3 92 7c f1 b7 01 21 1f ad df 5e
Metadata Header    file
                  offset  data
  Signature end:   02a4 = 00000500
  Sections:        02ac = 3
  Keys:            02b0 = 24
  Metadata Sections  file
                    offset  Data
    section 00:
      Offset:      02c0 = 00000000_00000670
      Length:      02c8 = 00000000_00022b50 bytes
      Key:         02e4 = 0006
      IV:          02e8 = 0007
      SHA1:        02dc = 0000
      Type:        02d0 = 0002
    section 01:
      Offset:      02f0 = 00000000_00023240
      Length:      02f8 = 00000000_00000e98 bytes
      Key:         0314 = 000e
      IV:          0318 = 000f
      SHA1:        030c = 0008
      Type:        0300 = 0002
    section 02:
      Offset:      0320 = 00000000_000240e0
      Length:      0328 = 00000000_00005153 bytes
      Key:         0344 = 0016
      IV:          0348 = 0017
      SHA1:        033c = 0010
      Type:        0330 = 0002
  Metadata Keys    file
                  offset  Data
    key idx 0000:  0350 = 55 1f 96 c0 b5 bb 8d 46 b1 14 d5 77 31 12 bb 75
    key idx 0001:  0360 = ff 9a f7 e1 00 00 00 00 00 00 00 00 00 00 00 00
    key idx 0002:  0370 = 22 4d cc 4c 09 a3 02 45 1e 95 80 ee 16 f5 eb be
    key idx 0003:  0380 = 3f 88 c9 66 82 5b 4f 8b 7f e6 8a 97 89 12 fd 89
    key idx 0004:  0390 = 9d 04 23 da f1 0a c9 77 85 9f 5b 5b 3b 48 72 f0
    key idx 0005:  03a0 = 6a 3f e9 25 f6 85 8b 0c d2 93 d9 1a 2d 94 fe 58
    key idx 0006:  03b0 = 78 b0 ad 01 c2 cc e2 92 64 27 29 14 54 b4 b9 bb
    key idx 0007:  03c0 = 08 b9 a2 ff 60 23 ac 8c 0b 3f da f7 00 00 00 00
    key idx 0008:  03d0 = 6f b6 fe 95 53 82 ed 79 c7 d4 52 91 1d cc 25 df
    key idx 0009:  03e0 = 34 b1 04 e7 00 00 00 00 00 00 00 00 00 00 00 00
    key idx 000a:  03f0 = 22 4d cc 4c 09 a3 02 45 1e 95 80 ee 16 f5 eb be
    key idx 000b:  0400 = 3f 88 c9 66 82 5b 4f 8b 7f e6 8a 97 89 12 fd 89
    key idx 000c:  0410 = 9d 04 23 da f1 0a c9 77 85 9f 5b 5b 3b 48 72 f0
    key idx 000d:  0420 = 6a 3f e9 25 f6 85 8b 0c d2 93 d9 1a 2d 94 fe 58
    key idx 000e:  0430 = ae 95 a2 6e 90 1b d9 39 7e a3 06 0f fe 61 6e 56
    key idx 000f:  0440 = fe 2b a1 16 f6 fd e9 ac 82 10 f6 6c 00 00 00 00
    key idx 0010:  0450 = 09 50 1a cc e7 7b 63 dc 8c 4b 91 94 04 be ff 5f
    key idx 0011:  0460 = c6 2f ba 12 00 00 00 00 00 00 00 00 00 00 00 00
    key idx 0012:  0470 = 22 4d cc 4c 09 a3 02 45 1e 95 80 ee 16 f5 eb be
    key idx 0013:  0480 = 3f 88 c9 66 82 5b 4f 8b 7f e6 8a 97 89 12 fd 89
    key idx 0014:  0490 = 9d 04 23 da f1 0a c9 77 85 9f 5b 5b 3b 48 72 f0
    key idx 0015:  04a0 = 6a 3f e9 25 f6 85 8b 0c d2 93 d9 1a 2d 94 fe 58
    key idx 0016:  04b0 = 3d ac 48 3b b3 7d 72 12 6c c9 c7 d3 39 81 cf f2
    key idx 0017:  04c0 = 38 aa 15 97 c5 07 c0 5a 28 e4 fa e6 00 00 00 00
ELF SHDRs
No ELF Section Headers in this file.

Yet none of those are the HMAC key for the trophies, those are for decrypting the file itself.

So, I've ran unself2 to np_trophy_util.sprx to convert it to ELF. And once again, I get the same file in the ZIP linked in ps3wiki in your first comment.

I've been able to seek for the keys already shown in ps3wiki. Still, no luck on finding the hmac key that generates the URL nor the key for decrypting. I've not been able to find at least the hinted hex in the file using IDA either.

[pcmantinker]

Ah, from what you're saying, it seems I misunderstood the metadata header information and the hmac key from selfs. I may just have to byte (pun intended lol) the bullet and dig into IDA Pro at some point to further investigate.

[jhewt]

Lol, yeah it sucks. Too bad I do not own a PS3 to dump it's memory when it's loading hidden trophies. There are tools to dump it and debug it right on IDA Pro

[Red-EyeX32]

If you guys need it still, I have all the necessary keys.

[TJangles]

I've been looking at this independently and pretty much hit the same point as above - with the sprx/prx loaded into IDA, but nothing that looks like or works as the HMAC.

@Red-EyeX32, is the offer still on the table for the necessary keys, or alternatively a push in the right direction?

[Red-EyeX32]

@TJangles Send me your skype and I'll be more that happy to help out.

[TJangles]

I just realised I never dropped an update here and I've started looking into this again. Firstly, thanks to @Red-EyeX32 - with their help I managed to get a script working to create valid URLs for game images, trophy images, and the trop.esfm (which are subsequently decrypted and provide the hidden trophy details!).

The next hurdle seems to be figuring out what's changed for newer titles (post-2013?). Taking npcommids for recent titles (ie NPWR07817_00 - a PS4 game) and running them through the same code results in 404's. This suggests either the keys differ or, if that isn't the case, the files reside at a different server path.

Any input would be much appreciated.

[shadowtrophy]

I just realised I never dropped an update here and I've started looking into this again. Firstly, thanks to @Red-EyeX32 - with their help I managed to get a script working to create valid URLs for game images, trophy images, and the trop.esfm (which are subsequently decrypted and provide the hidden trophy details!).

The next hurdle seems to be figuring out what's changed for newer titles (post-2013?). Taking npcommids for recent titles (ie NPWR07817_00 - a PS4 game) and running them through the same code results in 404's. This suggests either the keys differ or, if that isn't the case, the files reside at a different server path.

Any input would be much appreciated.

are you still around or is redeye im looking for the update erk and hmac keys for ps3 trophies

[Jump-Suit]

Not to necro, but this issue is key for PSN emulation regarding decryption on Trophy .ESFM files, any word on this @TJangles ? I am VERY interested.