`peek -m` does not show metadata, only peeked output

The peek unit states, in its help output, that the -m, --meta flag does the following:

Only display attached metadata, do not add the peek value.

However, it seems that the peek unit currently does not display the metadata at all if the -m flag is specified; it behaves the same as if -r, --bare is specified, and only shows the peeked data:

$ emit yosemite.png | peek
-------------------------------------------------------------------------------------------------------------------------------------------------------------
peek = 5.441 MB; PNG image data, 2560 x 1440, 8-bit/color RGB, non-interlaced
-------------------------------------------------------------------------------------------------------------------------------------------------------------
000000: 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52 00 00 0A 00 00 00 05 A0 08 02 00 00 00 1D 62 8D 88 00 00 00 09  .PNG........IHDR..............b......
000025: 70 48 59 73 00 00 0B 13 00 00 0B 13 01 00 9A 9C 18 00 00 0A 4F 69 43 43 50 50 68 6F 74 6F 73 68 6F 70 20 49 43  pHYs................OiCCPPhotoshop.IC
00004A: 43 20 70 72 6F 66 69 6C 65 00 00 78 DA 9D 53 67 54 53 E9 16 3D F7 DE F4 42 4B 88 80 94 4B 6F 52 15 08 20 52 42  C.profile..x..SgTS..=...BK...KoR...RB
00006F: 8B 80 14 91 26 2A 21 09 10 4A 88 21 A1 D9 15 51 C1 11 45 45 04 1B C8 A0 88 03 8E 8E 80 8C 15 51 2C 0C 8A 0A D8  ....&*!..J.!...Q..EE...........Q,....
000094: 07 E4 21 A2 8E 83 A3 88 8A CA FB E1 7B A3 6B D6 BC F7 E6 CD FE B5 D7 3E E7 AC F3 9D B3 CF 07 C0 08 0C 96 48 33  ..!.........{.k........>...........H3
0000B9: 51 35 80 0C A9 42 1E 11 E0 83 C7 C4 C6 E1 E4 2E 40 81 0A 24 70 00 10 08 B3 64 21 73 FD 23 01 00 F8 7E 3C 3C 2B  Q5...B..........@..$p....d!s.#...~<<+
0000DE: 22 C0 07 BE 00 01 78 D3 0B 08 00 C0 4D 9B C0 30 1C 87 FF 0F EA 42 99 5C 01 80 84 01 C0 74 91 38 4B 08 80 14 00  ".....x.....M..0.....B.\.....t.8K....
000103: 40 7A 8E 42 A6 00 40 46 01 80 9D 98 26 53 00 A0 04 00 60 CB 63 62 E3 00 50 2D 00 60 27 7F E6 D3 00 80 9D F8 99  @z.B..@F....&S....`.cb..P-.`'........
000128: 7B 01 00 5B 94 21 15 01 A0 91 00 20 13 65 88 44 00 68 3B 00 AC CF 56 8A 45 00 58 30 00 14 66 4B C4 39 00 D8 2D  {..[.!.......e.D.h;...V.E.X0..fK.9..-
00014D: 00 30 49 57 66 48 00 B0 B7 00 C0 CE 10 0B B2 00 08 0C 00 30 51 88 85 29 00 04 7B 00 60 C8 23 23 78 00 84 99 00  .0IWfH.............0Q..)..{.`.##x....
-------------------------------------------------------------------------------------------------------------------------------------------------------------

$ emit yosemite.png | peek --meta
-------------------------------------------------------------------------------------------------------------------------------------------------------------
000000: 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52 00 00 0A 00 00 00 05 A0 08 02 00 00 00 1D 62 8D 88 00 00 00 09  .PNG........IHDR..............b......
000025: 70 48 59 73 00 00 0B 13 00 00 0B 13 01 00 9A 9C 18 00 00 0A 4F 69 43 43 50 50 68 6F 74 6F 73 68 6F 70 20 49 43  pHYs................OiCCPPhotoshop.IC
00004A: 43 20 70 72 6F 66 69 6C 65 00 00 78 DA 9D 53 67 54 53 E9 16 3D F7 DE F4 42 4B 88 80 94 4B 6F 52 15 08 20 52 42  C.profile..x..SgTS..=...BK...KoR...RB
00006F: 8B 80 14 91 26 2A 21 09 10 4A 88 21 A1 D9 15 51 C1 11 45 45 04 1B C8 A0 88 03 8E 8E 80 8C 15 51 2C 0C 8A 0A D8  ....&*!..J.!...Q..EE...........Q,....
000094: 07 E4 21 A2 8E 83 A3 88 8A CA FB E1 7B A3 6B D6 BC F7 E6 CD FE B5 D7 3E E7 AC F3 9D B3 CF 07 C0 08 0C 96 48 33  ..!.........{.k........>...........H3
0000B9: 51 35 80 0C A9 42 1E 11 E0 83 C7 C4 C6 E1 E4 2E 40 81 0A 24 70 00 10 08 B3 64 21 73 FD 23 01 00 F8 7E 3C 3C 2B  Q5...B..........@..$p....d!s.#...~<<+
0000DE: 22 C0 07 BE 00 01 78 D3 0B 08 00 C0 4D 9B C0 30 1C 87 FF 0F EA 42 99 5C 01 80 84 01 C0 74 91 38 4B 08 80 14 00  ".....x.....M..0.....B.\.....t.8K....
000103: 40 7A 8E 42 A6 00 40 46 01 80 9D 98 26 53 00 A0 04 00 60 CB 63 62 E3 00 50 2D 00 60 27 7F E6 D3 00 80 9D F8 99  @z.B..@F....&S....`.cb..P-.`'........
000128: 7B 01 00 5B 94 21 15 01 A0 91 00 20 13 65 88 44 00 68 3B 00 AC CF 56 8A 45 00 58 30 00 14 66 4B C4 39 00 D8 2D  {..[.!.......e.D.h;...V.E.X0..fK.9..-
00014D: 00 30 49 57 66 48 00 B0 B7 00 C0 CE 10 0B B2 00 08 0C 00 30 51 88 85 29 00 04 7B 00 60 C8 23 23 78 00 84 99 00  .0IWfH.............0Q..)..{.`.##x....
-------------------------------------------------------------------------------------------------------------------------------------------------------------

$ emit yosemite.png | peek --bare
-------------------------------------------------------------------------------------------------------------------------------------------------------------
000000: 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52 00 00 0A 00 00 00 05 A0 08 02 00 00 00 1D 62 8D 88 00 00 00 09  .PNG........IHDR..............b......
000025: 70 48 59 73 00 00 0B 13 00 00 0B 13 01 00 9A 9C 18 00 00 0A 4F 69 43 43 50 50 68 6F 74 6F 73 68 6F 70 20 49 43  pHYs................OiCCPPhotoshop.IC
00004A: 43 20 70 72 6F 66 69 6C 65 00 00 78 DA 9D 53 67 54 53 E9 16 3D F7 DE F4 42 4B 88 80 94 4B 6F 52 15 08 20 52 42  C.profile..x..SgTS..=...BK...KoR...RB
00006F: 8B 80 14 91 26 2A 21 09 10 4A 88 21 A1 D9 15 51 C1 11 45 45 04 1B C8 A0 88 03 8E 8E 80 8C 15 51 2C 0C 8A 0A D8  ....&*!..J.!...Q..EE...........Q,....
000094: 07 E4 21 A2 8E 83 A3 88 8A CA FB E1 7B A3 6B D6 BC F7 E6 CD FE B5 D7 3E E7 AC F3 9D B3 CF 07 C0 08 0C 96 48 33  ..!.........{.k........>...........H3
0000B9: 51 35 80 0C A9 42 1E 11 E0 83 C7 C4 C6 E1 E4 2E 40 81 0A 24 70 00 10 08 B3 64 21 73 FD 23 01 00 F8 7E 3C 3C 2B  Q5...B..........@..$p....d!s.#...~<<+
0000DE: 22 C0 07 BE 00 01 78 D3 0B 08 00 C0 4D 9B C0 30 1C 87 FF 0F EA 42 99 5C 01 80 84 01 C0 74 91 38 4B 08 80 14 00  ".....x.....M..0.....B.\.....t.8K....
000103: 40 7A 8E 42 A6 00 40 46 01 80 9D 98 26 53 00 A0 04 00 60 CB 63 62 E3 00 50 2D 00 60 27 7F E6 D3 00 80 9D F8 99  @z.B..@F....&S....`.cb..P-.`'........
000128: 7B 01 00 5B 94 21 15 01 A0 91 00 20 13 65 88 44 00 68 3B 00 AC CF 56 8A 45 00 58 30 00 14 66 4B C4 39 00 D8 2D  {..[.!.......e.D.h;...V.E.X0..fK.9..-
00014D: 00 30 49 57 66 48 00 B0 B7 00 C0 CE 10 0B B2 00 08 0C 00 30 51 88 85 29 00 04 7B 00 60 C8 23 23 78 00 84 99 00  .0IWfH.............0Q..)..{.`.##x....
-------------------------------------------------------------------------------------------------------------------------------------------------------------

Reproduced with binary-refinery version 0.4.30, Python 3.10.4.

This is a case of poor wording: The key attribute is "attached" metadata. Outside a frame (i.e. not receiving output after some unit ended with a [ argument), no chunk has any metadata attached to it at all, it is always just a stream of bytes. Even inside a frame, however, a chunk would only have "attached" metadata if something did actively attach any, or at least that is how the nomenclature is intended here. In the current HEAD, this looks as follows, because I moved the peek summary out of the metadata portion:

$ emit hello [| put key value | peek ]
-----------------------------------------------------------------------------------------------------------------------
0.005 kB; 24.02% entropy; PHP script, ASCII text, with no line terminators
-----------------------------------------------------------------------------------------------------------------------
key = value
-----------------------------------------------------------------------------------------------------------------------
0000: 68 65 6C 6C 6F                                                                       hello
-----------------------------------------------------------------------------------------------------------------------

$ emit hello [| put key value | peek -m ]
-----------------------------------------------------------------------------------------------------------------------
key = value
-----------------------------------------------------------------------------------------------------------------------
0000: 68 65 6C 6C 6F                                                                       hello
-----------------------------------------------------------------------------------------------------------------------

Now, that said, I am happy to change this. It would probably be good to start at: What did you expect peek -m to do?

Ah, I see where I misunderstood now. So if I understand correctly, the metadata referred to here is metadata variables, i.e, the objects from refinery.lib.meta. rather than the file properties such as size, entropy, and filetype, which appear in the peek summary.

The appearance of the latter is what I had expected to be controlled with the --meta and --bare switches, such that I would have expected the output of emit yosemite.png | peek --meta to look something like this:

$ emit yosemite.png | peek --meta
--------------------------------------------------------------------------------------------------------------------
peek = 5.441 MB; PNG image data, 2560 x 1440, 8-bit/color RGB, non-interlaced
--------------------------------------------------------------------------------------------------------------------

I think that the reason I expected this to be the behaviour is due to the way that peek -m is introduced in the Netwalker Dropper tutorial, which introduces the -m switch as a way to display the file summary and verify the type of the carved file:

I will no longer use the -d switch for peek because I don't expect the result to be printable any more: %emit nl.ps1 | carve -s intarray | pack | peek

----------------------------------------------------------------------------------------------------------------------
000: FD EA 20 B0 B3 B0 B0 B0 B4 B0 B0 B0 4F 4F B0 B0 08 B0 B0 B0 B0 B0 B0 B0 F0 B0 B0 B0  ............OO..............
01C: B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0  ............................
038: B0 B0 B0 B0 70 B0 B0 B0 BE AF 0A BE B0 04 B9 7D 91 08 B1 FC 7D 91 E4 D8 D9 C3 90 C0  ....p..........}....}.......
054: C2 DF D7 C2 D1 DD 90 D3 D1 DE DE DF C4 90 D2 D5 90 C2 C5 DE 90 D9 DE 90 F4 FF E3 90  ............................
070: DD DF D4 D5 9E BD BD BA 94 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0  ............................
08C: B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0  ............................
0A8: B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 E0 F5 B0 B0  ............................
0C4: D4 36 B6 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 B0 40 B0 92 90 BB B2 BE A0 B0 C6 B1 B0  .6..............@...........
0E0: B0 E6 B0 B0 B0 B0 B0 B0 E0 9B B1 B0 B0 A0 B0 B0 B0 B0 B0 30 B1 B0 B0 B0 B0 A0 B0 B0  ...................0........
0FC: B0 B2 B0 B0 B6 B0 B0 B0 B0 B0 B0 B0 B5 B0 B0 B0 B0 B0 B0 B0 B0 90 B2 B0 B0 B4 B0 B0  ............................
118: B0 B0 B0 B0 B2 B0 D0 B0 B0 B0 A0 B0 B0 B0 B0 B0 B0 A0 B0 B0 B0 B0 B0 B0 B0 B0 A0 B0  ............................
----------------------------------------------------------------------------------------------------------------------

I pass the -m switch (for metadata) to peek so that it will tell me some basic information, as a sanity check of sorts:

%emit nl.ps1 | carve -s intarray | pack | xor 0xB0 | peek -m
----------------------------------------------------------------------------------------------------------------------
entropy = 80.32%
     ic = 03.70%
  magic = PE32+ executable (DLL) (GUI) x86-64, for MS Windows
   size = 0.119 MB
----------------------------------------------------------------------------------------------------------------------
000: 4D 5A 90 00 03 00 00 00 04 00 00 00 FF FF 00 00 B8 00 00 00 00 00 00 00 40 00 00 00  MZ......................@...
01C: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ............................
038: 00 00 00 00 C0 00 00 00 0E 1F BA 0E 00 B4 09 CD 21 B8 01 4C CD 21 54 68 69 73 20 70  ................!..L.!This.p
054: 72 6F 67 72 61 6D 20 63 61 6E 6E 6F 74 20 62 65 20 72 75 6E 20 69 6E 20 44 4F 53 20  rogram.cannot.be.run.in.DOS.
070: 6D 6F 64 65 2E 0D 0D 0A 24 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  mode....$...................
08C: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ............................
0A8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50 45 00 00  ........................PE..
0C4: 64 86 06 00 00 00 00 00 00 00 00 00 00 00 00 00 F0 00 22 20 0B 02 0E 10 00 76 01 00  d................."......v..
0E0: 00 56 00 00 00 00 00 00 50 2B 01 00 00 10 00 00 00 00 00 80 01 00 00 00 00 10 00 00  .V......P+..................
0FC: 00 02 00 00 06 00 00 00 00 00 00 00 05 00 00 00 00 00 00 00 00 20 02 00 00 04 00 00  ............................
118: 00 00 00 00 02 00 60 00 00 00 10 00 00 00 00 00 00 10 00 00 00 00 00 00 00 00 10 00  ......`.....................
----------------------------------------------------------------------------------------------------------------------

Was the implementation of peek different when the tutorial was written, such that -m was required to print the file summary?

Ah, I see, and I am sorry. And yes, I changed my mind about how I want peek to work a couple of times since the tutorial, and the tutorial has never been updated to match the subsequent changes I made to the code. My problem is that I still haven't found the interface I really like =/. Back then I realized that I would basically always specify the -m switch and it felt like having some metadata output should be the default, so I moved towards displaying one "virtual" piece of metadata called peek which had entropy, file size, and file magic all rolled into one line. Now sometimes, this would bug me and I (stupidly) repurposed the -m switch to give me an option to leave out this virtual metadata and only show the actual metadata. And very recently, I then decided that I don't really want to mix this synthesized piece of metadata with the actual metadata at all and moved it into its own section. I will work on fixing this and document my changes here. I'll also update the tutorial once I am done. If you have deep thoughts on how this should work, let me know.

Thanks for clarifying! I think that updating the tutorial to match the current behaviour is good enough for me.

I currently have no deep thoughts on how this should work; my only comment is that I almost always have peek at the very end of a pipeline, and have never (yet) found myself wanting to programmatically extract the data from the peek summary to use elsewhere in the pipeline. peek is really intended for human consumption and review, so I think it's OK for the synthesized metadata in the peek summary to be its own special thing separate from the actual metadata, as it is now.

I just realized as well that my expected/desired behaviour from peek --meta (i.e. where you only get the file summary, for a quick review) can be achieved via the peek --lines 0 option anyways, so I'm happy:

$ emit folder_full_of_elves/* [| peek --lines 0 ]
----------------------------------------------------------------------------------------------------------------------------------------------------------
peek = 27.288 kB; 37.87% entropy; ELF 64-bit LSB shared object, x86-64, version 1 (SYSV)
----------------------------------------------------------------------------------------------------------------------------------------------------------
peek = 27.344 kB; 52.09% entropy; ELF 64-bit LSB shared object, x86-64, version 1 (SYSV)
----------------------------------------------------------------------------------------------------------------------------------------------------------
peek = 55.792 kB; 68.62% entropy; ELF 64-bit LSB shared object, x86-64, version 1 (SYSV)
----------------------------------------------------------------------------------------------------------------------------------------------------------
peek = 0.467 MB; 78.05% entropy; ELF 64-bit LSB shared object, x86-64, version 1 (SYSV)
----------------------------------------------------------------------------------------------------------------------------------------------------------
peek = 22.800 kB; 41.74% entropy; ELF 64-bit LSB shared object, x86-64, version 1 (SYSV)
----------------------------------------------------------------------------------------------------------------------------------------------------------
peek = 0.863 MB; 69.68% entropy; ELF 64-bit LSB executable, x86-64, version 1 (SYSV)
----------------------------------------------------------------------------------------------------------------------------------------------------------
peek = 1.158 MB; 74.56% entropy; ELF 64-bit LSB executable, x86-64, version 1 (SYSV)
----------------------------------------------------------------------------------------------------------------------------------------------------------
peek = 0.863 MB; 69.68% entropy; ELF 64-bit LSB executable, x86-64, version 1 (SYSV)
----------------------------------------------------------------------------------------------------------------------------------------------------------
peek = 1.904 MB; 75.71% entropy; ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux)
----------------------------------------------------------------------------------------------------------------------------------------------------------

Oh indeed, I should have mentioned that it does that. You might also like:

ef ** [| cfmt {size!r} {sha256} {entropy!r} {magic} ]]

With cfmt you can make some fairly granular choices of what sort of metadata you would like to have displayed. I realize now that there is no really good documentation of what sort of "magic" variables exist, and that will be another TODO I'm putting on my list ;-).

Alright;

The documentation now has a section about magic meta variables.
The peek has been re-designed and the tutorial was updated to match how it works in the most recent release.

With that, I'll close this out.

Awesome! Thanks very much for doing that. The new colours in peek look great as well (:

binref / refinery

`peek -m` does not show metadata, only peeked output #15