search

jglim / ODB

Utilities for Softing binary ObjectDB files
MIT License
31 stars 15 forks source link

Extract Com Params from smr-d #9

Open Flo354 opened 3 months ago

Flo354 commented 3 months ago

Hello,

I studied the program because I would like to automate the extraction of com params for a personal project. For instance, CP_UniqueRespIdTable, CP_StMin...

I noticed in the strings extracted that there are these parameters, but they are not always accurate. Let's take for example two ECUs for the UniqueRespIdTable:

In the case of MED1775, you will see something like this:

ISO_15765_2.smr-cs::ISO_15765_2.CP_UniqueRespIdTable
normal segmented 11-bit transmit with FC
2016
normal segmented 11-bit receive with FC
2024
normal unsegmented 11-bit receive
UDS_CAN_D

As expected, we can see the can ID and the tester ID

But for BC_F222, this is a complete different story:

ISO_15765_2.smr-cs::ISO_15765_2.CP_UniqueRespIdTable
normal segmented 11-bit transmit with FC
normal segmented 11-bit receive with FC
1417
normal unsegmented 11-bit receive
UDS_CAN_D

Here, we have only the tester ID. The can ID should be below normal segmented 11-bit transmit with FC but it is not. This is one example. On some ECUs, the UniqueRespIdTable is not present, or both of the can ID and tester ID are not present.

For BC_F222, we can still find the can ID. going through database differ we can see that the can id is "1545" image

The line \00 1545 \00 (or \n if replaced) is well defined: image

But I am yet to understand how can I parse this accordingly.

Do you have any idea how can I parse this accurately as Database Differ does please?

Thanks,

jglim commented 2 months ago

Hello Florian,

I did not look further into the format after I decrypted the main sections. Within your SMR files, there are..

https://github.com/jglim/ODB/blob/783e666dd612106b1abf8566c390fc482ed729dc/ODB/ObjectDB/ODBFile.cs#L172

Your attachments come from the strings block, which seems to purely consist of a string table. I assume that one of the earlier 2 blocks contains a tree-like structure which is what we see in DatabaseDiffer. For string fields in that structure, there is probably a pointer or offset to the string table.

When the strings are serialized, I am fairly certain that there is deduplication involved (esp. since many variants share identical values), which should explain why some values are missing.

Unfortunately beyond this, I do not know more about the tree structure. I am very interested in knowing how it works too, but I am also stuck at this point.

Flo354 commented 2 months ago

When the strings are serialized, I am fairly certain that there is deduplication involved (esp. since many variants share identical values), which should explain why some values are missing.

This is actually very interesting and can explain what I am facing.

Do you have any clue on how we could eventually be able to tackle this? How did you find about the decryption process? Reverse engineering? Resources on the web? ...

jglim commented 2 months ago

This project was implemented from reversing DTS, specifically version 8.14. The bulk of the interesting bits (such as decryption, keys, hashes, segments) were found in ODBase.dll. I am not sure if the higher-level parsing is still contained within this DLL.

I don't have any good ideas on moving forward, but DatabaseDiffer (and the DLLs that it depends on) is a good candidate to dig into, since DatabaseDiffer is relatively lightweight as compared to Monaco.

Flo354 commented 2 months ago

I found something:

ODBase.dll is embedding StdUtil.dll which itself is including these three DLLs:

First guess: the unknown data section is an AVL Tree which is used to regenerate the XML ODB file. There is also the TrafoXmlConfig.xsd which seems to contain rules of ODB file. Guess it would be used by Xalan after AVL tree.

EDIT: TrafoCoApp.exe 

PS C:\Program Files (x86)\Softing\Diagnostic Tool Set 8\8.16\bin> .\TrafoCoApp.exe

Error: Calling parameters wrong or incomplete!
No arguments at the command line!

 ************************************   Usage of command line application   *************************************

 TrafoCoApp.exe /InFiles <list of files> [options-A] [options-B]
 TrafoCoApp.exe /InDir <inDir name> [options-A] [options-B]
 TrafoCoApp.exe /xmlConfig <file> [options-A]
 TrafoCoApp.exe /InFiles <list of files> /xmlConfig <file> /outDir <output folder> [options-A]
 TrafoCoApp.exe /fsplit <ODX-F file> [options-F] [options-A]
 TrafoCoApp.exe /xmlConfig <file> /fsplit /splitall [options-A]

 Possible options for the console application are:

   [/help]
options-A :
   [/log]
   [/forceTrafo]
   [/CheckerRS <rule set>]
   [/resultUI]
   [/no-encrypt] [/encrypt <method>]
   [/dbscheme <scheme nr>]
options-B :
   [/version <new version>]
   [/omitTextId] [/omitSdg] [/omitDesc] [/omitCompanyAdminData]
   [/hidePaths]
   [/d]
   [/p-fg] [/p-p] [/p-c] [/p-flash] [/p-ecu] [/p-e] [/p-fd]
   [/bp-job <list of paths>] [/bp-lib <list of paths>]
   [/bp-flash <list of paths>]
   [/excludeEcuVariants <listOfEcuVariantNames>]
   [/outDir <path>]
   [/tiInputFile <tmx_file>]
options-F :
   [/splitall]
   [/idents <listOfIdents>]

 Short description of options:

 /help        -- A short usage summary is shown
 /version     -- Used to set an user-defined revision information
 /dbscheme    -- Provide desired scheme for database output
 /xmlConfig   -- Xml file that contains a configuration for the trafo run
 /log         -- A default log file is generated if there is no log file specified after this option
 /forceTrafo  -- Transformation will not stop if checker-tool detects ruleset violations in odx input files
 /CheckerRS   -- Used to specify the rule set used by checker-tool call (when Trafo.ini is present)
 /resultUI    -- Display a pop-up with the transformation result and an option to consult trafo log
 /no-encrypt  -- Suppress encryption of databases during transformation
 /encrypt     -- Requests encryption of databases with provided method during transformation
 /omitTextId  -- TextIDs are not included into the output format
 /omitSdg     -- SDGs are not included into the output format
 /omitDesc    -- DESCs are not included into the output format
 /omitCompanyAdminData
              -- ADMIN-DATA and COMPANY-DATA are not included into the output format
 /hidePaths   -- Used to hide full paths for output directories, exe-path and input files (or directories) in the meta information
 /d           -- A regular transformation will be performed, but no actual output data will be created
 /p-fg        -- The package type for the output content is FunctionalGroup
 /p-p         -- The package type for the output content is Protocol
 /p-c         -- The package type for the output content is ComparamSpec
 /p-flash     -- The package type for the output content is Flash
 /p-ecu       -- The package type for the output content is selfcontained ECU
 /p-e         -- The package type for the output content is EcuConfig
 /p-fd        -- The package type for the output content is FunctionDictionary
 /bp-job      -- Additional base paths, separated by ";" and used to resolve java job files
 /bp-lib      -- Additional base paths, separated by ";" and used to resolve library referenced files
 /bp-flash    -- Additional base paths, separated by ";" and used to resolve flash data files
 /excludeEcuVariants
              -- List with EcuVariants names, separated by ";" , that will be excluded from the output content
 /outDir      -- The directory where the transformed output is stored. The option is mandatory
 /tiInputFile -- Text Identifiers shall be added in smr during a trafo run, from the specified TMX file
 /fsplit      -- Flash file will be splitted by idents(partnumbers) selected from a dialog or provided in cmdline
 /splitall    -- Used together with /fsplit to output a separate smr-f file for each ident from ODX-F input
 /idents      -- List with idents names, separated by ";" , used together with /fsplit to split a flash file
*****************************************************************************************************************
jglim commented 2 months ago

That's interesting. I'm struggling to understand if it is possible and practical to fit an XML file into an AVL tree. The XSLT transformation does make sense though, but that probably comes after the first layer is removed?

TrafoCoApp has also been a mystery to me. I haven't been able to get it do achieve anything meaningful, and there's little documentation for that application.

Flo354 commented 2 months ago

I think the AVL is somehow a mapper tree to the string tree. I guess there must be a path in the binary tree which when you follow it guides you toward the correct string. Since it's optimized binary search, that would explain why the needed strings are not always at the same place.

We need now to understand how to parse this AVL tree and how to map it in order to rebuild the XML ODX.

I think this is where TrafoCoApp.exe is useful. In my opinion, this app is used somehow to generate smr files from the OTX Studio. Maybe it would be easier trying to convert ODX to SMR. Since it must be almost the same operation, when we know how to go from A to B, it would be easier to go from B to A.

What do you think?

jglim commented 2 months ago

If I had to guess, strings in the tree are a bunch of offsets pointing at the strings blob. Non-primitive types (e.g. byte arrays) are probably embedded in the other binary blob in the same manner. Primitive fixed-length types e.g. int32 could possibly be embedded directly in the tree. I do not know, or have any assumptions about the actual tree format, though there's a good chance it is embedded in the first unknown block.

However, this is mostly guesswork. I got stuck here on static RE; running DTS applications with an attached debugger might yield more useful leads.

Flo354 commented 2 months ago

Yes, this is out of my field either, I will continue to extract data through Database Differ :p