jglim / CaesarSuite

Library and applications to work with Dаіmlеr diagnostics CBF files.
MIT License
125 stars 33 forks source link

Failure to parse some CBF's #57

Closed prj closed 1 year ago

prj commented 1 year ago

For example ME28.cbf.

Bunch of errors like this:

Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_12
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_10
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_10
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_11
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_16
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_16
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_16
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_16
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_16
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_16
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_16
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_16
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_16
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_11
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_10
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_10
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_11
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_11
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_10
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_10
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_10
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_14
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_14
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_5E
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_11
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_58
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_12
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_58
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_12
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_11
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_11
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_12
Parsing experimental 0x410 prep with sysparam 17 at #0
0x410 : sys param: 17 for qualifier #0 could not find referenced DiagService with index DT_IOLID_10

Followed by this:

System.IO.IOException
  HResult=0x80131620
  Message=An attempt was made to move the position before the beginning of the stream.
  Source=mscorlib
  StackTrace:
   at System.IO.MemoryStream.Seek(Int64 offset, SeekOrigin loc)
   at Caesar.CaesarReader.ReadBitflagDumpWithReader(UInt64& bitFlags, BinaryReader reader, Int32 dumpSize, Int64 virtualBase) in C:\Tuning\Tools\repos\CaesarSuite\Caesar\Caesar\CaesarReader.cs:line 45
   at Caesar.ECUVariantPattern..ctor(BinaryReader reader, Int64 baseAddress) in C:\Tuning\Tools\repos\CaesarSuite\Caesar\Caesar\ECUVariantPattern.cs:line 82
   at Caesar.ECUVariant.CreateVariantPatterns(BinaryReader reader) in C:\Tuning\Tools\repos\CaesarSuite\Caesar\Caesar\ECUVariant.cs:line 247
   at Caesar.ECUVariant..ctor(BinaryReader reader, ECU parentEcu, CTFLanguage language, Int64 baseAddress, Int32 blockSize) in C:\Tuning\Tools\repos\CaesarSuite\Caesar\Caesar\ECUVariant.cs:line 171
   at Caesar.ECU.CreateEcuVariants(BinaryReader reader, CTFLanguage language) in C:\Tuning\Tools\repos\CaesarSuite\Caesar\Caesar\ECU.cs:line 543
   at Caesar.ECU..ctor(BinaryReader reader, CTFLanguage language, CFFHeader header, Int64 baseAddress, CaesarContainer parentContainer) in C:\Tuning\Tools\repos\CaesarSuite\Caesar\Caesar\ECU.cs:line 367
   at Caesar.CaesarContainer.ReadECU(BinaryReader fileReader) in C:\Tuning\Tools\repos\CaesarSuite\Caesar\Caesar\CaesarContainer.cs:line 204
   at Caesar.CaesarContainer..ctor(Byte[] fileBytes) in C:\Tuning\Tools\repos\CaesarSuite\Caesar\Caesar\CaesarContainer.cs:line 49
   at Caesar.Program.Main(String[] args) in C:\Tuning\Tools\repos\CaesarSuite\Caesar\Caesar\Program.cs:line 17

  This exception was originally thrown at this call stack:
    [External Code]
    Caesar.CaesarReader.ReadBitflagDumpWithReader(ref ulong, System.IO.BinaryReader, int, long) in CaesarReader.cs
    Caesar.ECUVariantPattern.ECUVariantPattern(System.IO.BinaryReader, long) in ECUVariantPattern.cs
    Caesar.ECUVariant.CreateVariantPatterns(System.IO.BinaryReader) in ECUVariant.cs
    Caesar.ECUVariant.ECUVariant(System.IO.BinaryReader, Caesar.ECU, Caesar.CTFLanguage, long, int) in ECUVariant.cs
    Caesar.ECU.CreateEcuVariants(System.IO.BinaryReader, Caesar.CTFLanguage) in ECU.cs
    Caesar.ECU.ECU(System.IO.BinaryReader, Caesar.CTFLanguage, Caesar.CFFHeader, long, Caesar.CaesarContainer) in ECU.cs
    Caesar.CaesarContainer.ReadECU(System.IO.BinaryReader) in CaesarContainer.cs
    Caesar.CaesarContainer.CaesarContainer(byte[]) in CaesarContainer.cs
    Caesar.Program.Main(string[]) in Program.cs

This is an older controller, that is on K-Line, not on CAN. But then for example CR2 and CR3 parse just fine.

prj commented 1 year ago

Seems like only the Unk16 field fails. I did this in CaesarReader.ReadBitFlagDumpWithHeader and it seems that everything else gets parsed okay. The readout "dumpOffset" is a huge int value. I wonder if in this cbf it's a direct value and not an offset at all actually.

if (CheckAndAdvanceBitflag(ref bitFlags))
{
    byte[] result;
    // read the dump's offset relative to our current block
    int dumpOffset = reader.ReadInt32();
    // save our reading cursor
    long readerPosition = reader.BaseStream.Position;
    try {
        // seek to the specified offset, then read out the dump
        reader.BaseStream.Seek(dumpOffset + virtualBase, SeekOrigin.Begin);
        result = reader.ReadBytes(dumpSize);
        // restore our reading cursor
    } catch (IOException) {
        Console.WriteLine("Failed to read BitflagDump!");
        result = new byte[] { };
    }
    reader.BaseStream.Seek(readerPosition, SeekOrigin.Begin);
    return result;
}

EDIT: Not okay. On those that this fails, the entire structure ends up blank. No variant data. Seems like something else entirely goes very wrong.

prj commented 1 year ago

So the offset is always screwed. It's way higher than the file size. The reason it fails on this one, is because you're using an int and it rolls over and becomes negative. Format/order is probably different in this...

Pretty sure now it's some kind of different style of identification of the controller. Here's KWP vendor ID vs unk16 as an uint in hex:

8101 - 70454502
0100 - 70454502
8103 - 70454502
0101 - 70454502
0100 - 93457502
8103 - 93457502
0100 - 71454502
0100 - 94457502
0102 - 58450503
0102 - 06532111
0101 - 60450503
0100 - 00451503
0000 - 00451503
0101 - 59450503
0101 - 61450503
0100 - 01451503
0000 - 01451503
0000 - 00532111
0000 - 01532111
0000 - 02532111
0000 - 03532111
0000 - 00533111
0000 - 01533111
0000 - 02533111
0000 - 08532111
0000 - 09532111
0000 - 10532111
0000 - 13532111
0000 - 04533111
0000 - 05533111
0000 - 06533111
0000 - 11533111
0000 - 16533111
0000 - 22533111
0000 - 23533111
0000 - 22533111
0000 - 07532111
0000 - 26532111
0000 - 20532111
0000 - 15532111
0000 - 12532111
0000 - 14532111
0000 - 27532111
0000 - 12533111
0000 - 19533111
0000 - 13533111
0000 - 14533111
0000 - 33532111
0000 - 34532111
0000 - 36532111
0000 - 35532111
0000 - 29532111
0000 - 27533111
0000 - 28533111
0000 - 29533111
0000 - 37532111
0000 - 38532111
0000 - 30533111
0000 - 32533111
0000 - 40532111
0000 - 30532111
0000 - 31532111
0000 - 32532111
0000 - 41532111
0000 - 42532111
0000 - 47532111
0000 - 48532111
0000 - 05532111
0000 - 21532111
0000 - 54532111
0000 - 57532111
0000 - 58532111
0000 - 33533111
0000 - 34533111
0000 - 35533111
0000 - 36533111
0000 - 48533111
0000 - 49533111
0000 - 53533111
0000 - 54533111
0000 - 59532111
0000 - 53532111
0000 - 43532111
0000 - 49532111
0000 - 50532111
0000 - 52532111
0000 - 44532111
0000 - 51532111
0000 - 66532111
0000 - 40533111
0000 - 46533111
0000 - 45533111
0000 - 44533111
0000 - 47533111
0000 - 70532111
0000 - 78532111
0000 - 77532111
0000 - 68533111
0000 - 70533111
0000 - 73532111
0000 - 74532111
0000 - 75532111
0000 - 76532111
0000 - 83532111
0000 - 74533111
0000 - 86533111
0000 - 75533111
0000 - 76533111
0000 - 73533111
0000 - 89533111
0000 - 91533111
0000 - 92533111
0000 - 93533111
0000 - 81532111
0000 - 82532111
0000 - 85532111
0000 - 86532111
0000 - 93532111
0000 - 95532111
0000 - 96532111
0000 - 95533111
0000 - 96533111
0000 - 97533111
0000 - 10533111
0000 - 11533111
0000 - 12533111
0000 - 19533111
0000 - 32533111
0000 - 94532111
0000 - 92532111
prj commented 1 year ago

Mystery solved. This field is just the first 4 bytes of the IDENT request. If it's present then that's what should be used as ident. It's deffo not a pointer to somewhere.

Comms log:

[>] 82 10 F1 1A 86 23
[<] 92 F1 10 5A 86 11 21 53 26 79 08 01 37 01 03 01 05 00 15 02 02 FA

And from the previous list there is 26532111 in there.

The logic is basically, check if both vendor id and this byte array match, and if not just match the byte array to get the variant. I'll make PR.

jglim commented 1 year ago

Nice to see you again prj, and thanks for investigating and solving this mystery. Looking forward to your PR!

prj commented 1 year ago

I made it a byte array, because it is literally the first bytes of a 1A 86 ident response. Converting it to an uint or some other datatype does not make sense, as it creates unnecessary endianness issues.

jglim commented 1 year ago

Byte array is a good fit. I might tweak the names slightly in the future as that value appears to be specific to KWP2000.

Thanks again for your contribution!

prj commented 1 year ago

Yeah, I don't really know what to name it. Maybe EcuId1A86 did not seem to be much better.

It can be present only in KWP2000PE/KWP20C3PE controllers and those that use the longer 1A 86 for identification, not 1A 87.