Support parsing encoded/binary protobuf data

puffyCid commented 6 days ago

Some log entries (specifically Statedump entries) may contain binary Protobuf data. Even though we do not have the .proto files, it still possible to parse the binary data (with caveats).

It would be cool if this library supported attempts to parse the binary protobuf data. Right now the library makes no attempts to parse this

References:

puffyCid commented 6 days ago

Some example data can be found in the tests.zip file ex: ./system_logs_monterey.logarchive/Special/000000000000000d.tracev3

Using CyberChef the following example:

10, 45, 99, 111, 109, 46, 97, 112, 112, 108, 101, 46, 97, 112, 112, 115, 116, 111, 114, 101, 100, 46, 77, 105, 103, 114, 97, 116, 111, 114, 77, 105, 115, 99, 101, 108, 108, 97, 110, 101, 111, 117, 115, 84, 97, 115, 107, 10, 40, 99, 111, 109, 46, 97, 112, 112, 108, 101, 46, 97, 112, 112, 115, 116, 111, 114, 101, 100, 46, 77, 105, 103, 114, 97, 116, 111, 114, 65, 112, 112, 85, 115, 97, 103, 101, 84, 97, 115, 107, 10, 38, 99, 111, 109, 46, 97, 112, 112, 108, 101, 46, 97, 112, 112, 115, 116, 111, 114, 101, 100, 46, 77, 105, 103, 114, 97, 116, 111, 114, 65, 114, 99, 97, 100, 101, 84, 97, 115, 107

Can be parsed into:

{
    "field #1: L-delim (e.g. string, message)": [
        "com.apple.appstored.MigratorMiscellaneousTask",
        "com.apple.appstored.MigratorAppUsageTask",
        "com.apple.appstored.MigratorArcadeTask"
    ]
}

puffyCid commented 6 days ago

Looks like same binary data may have extra data? Or may have extended the protobuf spec?

48, 129, 230, 2, 1, 8, 49, 115, 48, 19, 12, 9, 79, 83, 86, 101, 114, 115, 105, 111, 110, 12, 6, 50, 49, 69, 50, 53, 56, 48, 23, 12, 9, 77, 111, 100, 101, 108, 78, 97, 109, 101, 12, 10, 77, 97, 99, 32, 83, 116, 117, 100, 105, 111, 48, 27, 12, 22, 77, 101, 115, 115, 97, 103, 101, 80, 114, 111, 116, 111, 99, 111, 108, 86, 101, 114, 115, 105, 111, 110, 2, 1, 0, 48, 38, 12, 12, 67, 111, 109, 112, 117, 116, 101, 114, 78, 97, 109, 101, 12, 22, 65, 110, 100, 114, 111, 105, 100, 226, 128, 153, 115, 32, 77, 97, 99, 32, 83, 116, 117, 100, 105, 111, 48, 25, 2, 1, 1, 12, 2, 97, 107, 2, 8, 20, 10, 119, 115, 0, 0, 0, 1, 48, 0, 48, 0, 48, 0, 49, 0, 5, 0, 2, 1, 0, 1, 1, 0, 5, 0, 5, 0, 5, 0, 48, 0, 49, 65, 48, 17, 12, 13, 116, 114, 117, 115, 116, 101, 100, 95, 114, 105, 110, 103, 115, 49, 0, 48, 44, 12, 4, 85, 85, 73, 68, 12, 36, 50, 69, 65, 49, 51, 54, 67, 70, 45, 50, 68, 65, 57, 45, 52, 54, 69, 50, 45, 57, 56, 49, 65, 45, 48, 69, 53, 68, 67, 57, 48, 65, 52, 53, 53, 70

Neither CyberChef or bbpb can parse it. Converting to raw data via CyberChef, returns:


0•æ␂␁␈1s0␓
    OSVersion
␆21E2580␗
    ModelName
␊Mac Studio0␛
␖MessageProtocolVersion␂␁␀0&

ComputerName
␖Androidâ••s Mac Studio0␙␂␁␁
␂ak␂␈␔␊ws␀␀␀␁0␀0␀0␀1␀␅␀␂␁␀␁␁␀␅␀␅␀␅␀0␀1A0␑
␍trusted_rings1␀0,
␄UUID
$2EA136CF-2DA9-46E2-981A-0E5DC90A455F```

mandiant / macos-UnifiedLogs

Support parsing encoded/binary protobuf data #30