DTCO 4.0 Structure ?

[cbrede]

Hi, from 14 JUN 2019 new Tacho will be install in the trucks. There also will new drivercard. Has some already some test data? How will be the implemtation?

BR

Carsten

[Bobertin89]

Hi, I try to use the program with a new V1B file but I've got this error : "First unrecognised region with magic 0 at 0x0000" "There were 26765 unmatched regions (magics) in the file." I attach the file in this post.

newDTCO.zip

Can you tell me if you have an issue?

BR Sylvain

[jugglingcats]

I assume V1B is a different file format, ie. not DDD. If it's a different format entirely (as opposed to just a DDD format with a different extension in the filename), this project doesn't currently support it.

[Bobertin89]

Hi, thank you for your reply. I thought it was the same format with different extension for different country. With my tachygraph, I can take the file with TGD, DDD or V1B format. I take the files with the three extensions and they are strictely identical.

I have some trucks with the old file format (DTCO3) and some trucks with the new file format (DTCO4, since june 2019). When I use your application with the V1B or DDD extensions, it work with the DTCO3 and I've the same result.

I attach the file with the three format in dtco3 in a zip and with the three format in dtco4 in other zip.

dtco3.zip dtco4.zip

Thank you in advance for your Reply! BR

[jugglingcats]

Hi @Bobertin89, I understand thank you. Unfortunately I don't know anything about the new standard or the effort required to implement it. Perhaps it's something @davispuh has looked at.

[davispuh]

Haven't really looked into it, but most structures in gen2 is different with extra fields and signature validation is completely different.

[Bobertin89]

Hi all, thanks for reply. In your project, the two xml configuration file appears to contain the file structure. In order to read the new file, if I make an other xml config file with the new additionnal field, it should' be work? I don't see anything for the identifier... I find this documentation https://dtc.jrc.ec.europa.eu/iot_doc/EU%202016-799-EN.pdf that seems to contain the structure of the new file, but not user friendly... Do you know if an other "user friendly" documentation exist? Sorry for my poor english ^^

[davispuh]

if I make an other xml config file with the new additionnal field, it should' be work?

Most likely it won't be that easy, but you can try.

I find this documentation https://dtc.jrc.ec.europa.eu/iot_doc/EU%202016-799-EN.pdf Do you know if an other "user friendly" documentation exist?

That's the most user friendly you'll find, I've been going over it for ages... Also it's not even latest, here's latest one with amendments http://data.europa.eu/eli/reg_impl/2016/799/2018-04-17

[jugglingcats]

Unfortunately these specs are not written to be developer friendly, which was my motivation for publishing this project in the first place (because the XML config file provides a good interpretation of the first digital spec!).

[Bobertin89]

Hi, thank's for your reply. I'm trying to make the xml config file for the gen2. I think I understand the purpose. However, I don't understand why some data are with the "global value" attributes and other are not. Can you explain me why, and in addition, can you tell me more about the validation signature?

Best regards and thank you in advance!

[Sh0ckwav3]

Hey Guys, Im trying to implemet your solution in my Office. But my Problem is simmilar to @Bobertin89. Some of our cars are Driving with the new TachoGraph and new DriverCards named G2 with the format 1C on it. Someone of you already created a new XML Config file for those new Versions?
Or has another way, how to read the new Generation DDD-Files?

Ty verry much und Best regards. Dominic

[jugglingcats]

Hi @Bobertin89, I don't really understand your comment about "global value". The XML config for a card (driver or vehicle) contains definitions for the elementary files defined by the gen 1 spec. These elementary files (think of them as sections) follow each other in the ddd file. Each one starts with a magic, and this selects which bit of the config will be used to interpret that section. For example, here is the config for the driver card "CardIccIdentification" elementary file. It has a magic of 0x0002.

    <ElementaryFile Name="CardIccIdentification" Identifier="0x0002" Unsigned="true">
        <UInt8 Name="ClockStop" Length="1"/>
        <ExtendedSerialNumber Name="CardExtendedSerialNumber"/>
        <SimpleString Name="CardApprovalNumber" Length="8"/>
        <UInt8 Name="CardPersonaliserId" Length="1"/>
        <Object Name="EmbedderIcAssemblerId">
            <SimpleString Name="CountryCode" Length="2" />
            <BCDString Name="ModuleEmbedder" Size="2" />
            <UInt8 Name="ManufacturerInformation" Length="1" />
        </Object>
        <UInt16 Name="IcIdentifier" Length="2"/>
    </ElementaryFile>

It is the responsibility of the config above to consume the entire elementary file. This is very important, because when this section is consumed we expect to put the read head at the exact start of the next elementary file, otherwise we will fail to read the correct magic for the next section and things will not work.

Within the config you can consume primitive values, such as UInt8 or UInt16 but there are also helpers to consume more complex structs in the spec, such as ExtendedSerialNumber. These helpers know how to read and output (in the XML) these specific bits of structure.

You can introduce further nesting in the XML using Object, which is just a convenient way of grouping properties together.

There are also facilities for reading repeated data from the elementary file given a count already encountered. If you need I can find examples of this. There are more esoteric facilities available too, such as reading from a cyclic buffer within the elementary file. Hopefully the gen 2 spec doesn't have these, but the feature is there!

I hope this gives a reasonable overview of how the config files work. I have no idea whether the capabilities provided are sophisticated enough to process the gen 2 spec. Does it even have the concept of elementary files with magics, because this is fundamental to the current way of processing.

And finally, @davispuh wrote the signature checking and I believe it is switched off by default.

[Sh0ckwav3]

I created a new ConfigFile, which contains now the C1 or Gen2 structure. But the main problem is, that the Driver-DDD file contains some informations twice. One time in Gen1-Format and a second time in Gen2-Format, but with the extenden Informations. So when i will read the Magic Identifier="0x0501" i will get more than one result in my XML file, this also happens by other Magics. I hope i can modify the code, that this will not happen anymore. If someone is familiar with this problem and wants to help me pls write me a message.

[jugglingcats]

The code to pick the region which is going to be used for processing (based on magic) is here:

https://github.com/jugglingcats/tachograph-reader/blob/master/src/DataFile.cs#L100

Clearly if the new standard has the same magic used more than once in the file there must be a way choose the correct region XML in the config to be used. I am not sure how this could/should be determined.

[nick-tn]

Does anybody knows why Gen2 files have Objects ID's like 0x7621, 0x7622, etc. Old files have 0x7601, 0x7602..?

I understand that it could specify Gen2 file but I can't find any document proving this idea. Moreover the document https://dtc.jrc.ec.europa.eu/iot_doc/EU%202016-799-EN.pdf describes that it must be 0x7601, 0x7602 because there's no difference for TREP in messages sending to old or new DTCO4.0 tachographs.

[WallHackUp01]

Greetings,

There is an other problem that i met. Gen 2 have new Elementary Files nammed DIR, ATR/INFO and EXTENDED_LENGHT. When the documentation redirect the user to TCS_145, TCS_146 and TCS_147 in order to complete the EF, I don't undersatnd what we really have to put in. Which type of data did i have to put in the EF ?

Thanks in advance.

[jugglingcats]

@WallHackUp01 it looks like the information needed is in this ISO spec: https://www.iso.org/standard/54550.html. Unfortunately it is not free 👎. And it is annoying because the EF is variable length so in order to skip it we need to know the structure.

[andreasringdal]

There is some information regarding the ISO spec available her https://gnupg.org/ftp/specs/openpgp-card-2.1.pdf Don't know if it is of any help.

[davispuh]

Does anybody knows why Gen2 files have Objects ID's like 0x7621, 0x7622, etc. Old files have 0x7601, 0x7602..?

I understand that it could specify Gen2 file but I can't find any document proving this idea. Moreover the document https://dtc.jrc.ec.europa.eu/iot_doc/EU%202016-799-EN.pdf describes that it must be 0x7601, 0x7602 because there's no difference for TREP in messages sending to old or new DTCO4.0 tachographs.

That's not true, it just looks like this isn't included in document you're looking at but if you look at latest one, there are described those TREP IDs.

http://data.europa.eu/eli/reg_impl/2016/799/2018-04-17

See

Appendix  7
DATA  DOWNLOADING  PROTOCOLS
2.2.2 Message types
2.2.2.9 Transfer Data Request (SID 36)

[nick-tn]

Thank you for pointing to the information! Now it is clear.

[patientzerouk]

I'm about to dive into this myself, and it would be really handy if others have something to start with.

Once/if I get this working, I intend to submit this back into the project. Hopefully others will as well!

Does anyone have a working example for GEN2 stuff?

[patientzerouk]

Hi All, Just posting a note to say I haven't forgotten about my GEN2 Stuff, but the holidays and work overtook me. I'll be back on it in the new year. Hope all is well with everyone.

[mpi-wl]

@WallHackUp01 it looks like the information needed is in this ISO spec: https://www.iso.org/standard/54550.html. Unfortunately it is not free 👎. And it is annoying because the EF is variable length so in order to skip it we need to know the structure.

It's not true. All the information you need is here: https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1468399756621&uri=CELEX:32016R0799 amending: https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1524124831090&uri=CELEX:32018R0502

and here: https://dtc.jrc.ec.europa.eu/

The length of all EF is known. Also DIR, ATR / INFO (these EF do not appear in ddd files).

Other documentation needed is ASN.1, Elliptic curve cryptography (Brainpool, NIST), etc.

Does anyone have a working example for GEN2 stuff?

Yes :) (app is not completed): https://esmreader.tachosfera.eu/

[Sh0ckwav3]

@mpi-wl are you going to share the code for gen2 stuff?

[mpi-wl]

@Sh0ckwav3 my library is not available as open source. But if you have a problem with 2G (or 1G) I will answer your questions.

[Sh0ckwav3]

Well so i will give it a second try... I got the specification for Driver File: https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1468399756621&uri=CELEX:32016R0799

but i cant find Specifications for Vehile, anyone got one for Vehicle DDD Files?

[mpi-wl]

The file structure of vu is in the same documentation. Search for "DDP_029". This documentation has changes: https://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1524124831090&uri=CELEX:32018R0502 e.g. (ii) | the heading ‘Data structure generation 1’ is replaced by the following: ‘Data structure generation 1 (TREP 01 Hex)’; (iii) | the heading “Data structure generation 2” is replaced by the following: ‘Data structure generation 2 (TREP 21 Hex)’;

The file structure from vu is differently described than the file structure from the driver card. Here, to know the length of the fields you must search for specific data types in Appendix 1

[Sh0ckwav3]

@mpi-wl is our library as .net as well? i added and edited now all settings for the new format for the driver cards and added regions to parse the data. but i only get the gen 1 data all the gen 2 will not be read. why is this, how is it possible to have 2 times an Identifier with the same code but different data behind? how can i solve this, that i can get only the g1 or only the g2?

Ty for your help

[mpi-wl]

@mpi-wl is our library as .net as well?

JavaScript

why is this, how is it possible to have 2 times an Identifier with the same code but different data behind?

The structure of the ddd file for the driver card

AA BB CC DD EE XX XX ... XX XX

AA (or BB, CC, DD ... XX.) is 1 byte

AA BB - EF file id CC (third byte) - 00: content of EF file for gen 1, 01: signature for gen 1, 02: content of EF file for gen 2, 03: EF signature for gen 2.

DD EE - length (data or signature)

XX XX ... XX XX - data or signature

Not all EF are signed (e.g. certificates) The correct ddd file for the 2G driver card contains gen 1 and gen 2 data

Ddd file for the VU 76 01, 76 02, 76 03, 76 04, 76 05 - gen 1 76 21, 76 22, ​​76 23, 76 24, 76 25 - gen 2

The VU ddd file contains the data of gen 1 or gen 2

[Sh0ckwav3]

Well i have to ask again sorry, maybe im dumb or whatever.

So when i have a look at my DDD File in HexEdit, then i can see that Gen1 "CardVehiclesUsed" starts somewhere at "00004620" but the Identifier is "0505" (as meantioned in documentation), this would not match with the data i can see in this image

the same for gen2 ... it starts somewhere at 0000b380 and not 0505, there are only 2 entries but where can i find the correct identifier to jump to this point and read all the data ...

AA BB - EF file id CC (third byte) - 00: content of EF file for gen 1, 01: signature for gen 1, 02: content of EF file for gen 2, 03: EF signature for gen 2.

I understand in this example that the gen 1 and gen 2 information are directly in line, unfortunately it does not look like this according to these pictures.

so if u have a tip for me how i can find only the gen 2 data or booth at the same time and pare it with the given code from this project ... it would help me a lot. i understand what the code from this repo is doing and how it works, but i dont get it how to identify the correct offset for each starting region.

[mpi-wl]

Well i have to ask again sorry, maybe im dumb or whatever.

Don't worry :) it's not easy.

Everything is ok: 05 04 - Driver Activity Data 01 - means there is still a signature (gen 1) 00 80 - 0x0080 = 128 dec

After 128 bytes (containing the signature for 05 04) there is 05 05 - so it's ok

You can only find data for gen 2 by checking 3 byte. You could also check the length or content of a given EF file and in case of an error it means that it is gen 2 but it is not an optimal solution.

Unfortunately, I don't know C# well and I can't help you with the code from this project. If I find some time, I can look at this code, but I don't know if I can help you.

[Sh0ckwav3]

Well TY so much for this example! It opened my eyes :D

But there are already new Questions :P

You mentioned 0504 is the identifier for DriverActivityData, but it should be VehiclesUsed Data there. In which Documentation you found which start means exactly what? I cant find this in any documentation :/

What i also don't understand are the red marked bytes (except the vuDataBlockCounter)

And all the yellow in this image are placeholders for more data until the ef is finished correct?

The next block that shows up is with identifier 0505 so what i meant in the first question, where you get the information about that what this is. the rest is clear then.

[mpi-wl]

You mentioned 0504 is the identifier for DriverActivityData, but it should be VehiclesUsed Data there.

Identifier for Vehicles Used is 05 05. Even in your cases you read 05 05.

In which Documentation you found which start means exactly what? I cant find this in any documentation :/

All the time in the same documentation :)

Search for TCS_152 and for gen 1 TCS_148

You read the red bytes incorrectly. 25 82 is the data length for this EF (9602 bytes). The next two bytes (00 01) are vehiclePointerNewestRecord. The oldest data is overwritten with newer ones. vehiclePointerNewestRecord is the byte number at which the latest data begins. In most cases, you can ignore it - read all CardVehicleRecord and sort them in an array. The exception is DriverActivityData because there the record length is variable. The last red byte (00) is the beginning of the first CardVehicleRecord. vehicleOdometerBegin is 3 bytes: 00 0B 09. vehicleOdometerEnd also is 3 bytes 00 0B 38. The dates of first and last use are well marked next is 1 byte specifying the country of registration 0A for Switzerland : https://dtc.jrc.ec.europa.eu/dtc_nation_codes.php Next is the code page (01 - ISO / IEC 8859-1 Latin-1) for the registration number. The registration number is 13 bytes (52 44 ... 20 20). Next is the vuDataBlock Counter (2 bytes) and vehicleIdentificationNumber (17 bytes). The red byte (00) is the beginning of the next record (00 0B 38 - vehicleOdeometerBegin etc ...)

And all the yellow in this image are placeholders for more data until the ef is finished correct?

These are the default values. Example for vehicle Registration Number (as in the picture above) 00 (for code page) and 20 20 20 .... 20 20. The length of Vehicles Used for your driver card is 9602 and so many bytes are in the ddd file. If there are only 2 records, the default data will still be (00, 20, FF etc.)

[mpi-wl]

@Sh0ckwav3 in the code of this project, the third byte is checked in /src/regions/ElementaryFileRegion.cs from 41 line

[Sh0ckwav3]

Nice Ty again for your help,

So i created now a new XML Config with gen 1 and gen 2 and looks good so far but one of the biggest problems that i'm facing is, that the application is not made for "2 generations in one file"

So the code will now find the start of the gen 2 EF twice: This one is Wrong because this is Data from gen 1 This is the correct start of GEN 2 File

Then he tries to parse allways from the first one and fails with all the given information in the config because they make no sense.

So i have to find a way, to read gen1 and 2 but has to tell gen 2 where the min start byte offset is. or i already thougt about a complete nev version where we split the gen 1 and 2 in separate bytearrays and then run once gen 1 and once gen 2 but this is not a clean solution.

[mpi-wl]

My library reads gen 1 and gen 2 (if there are two generations in one file) and this is how the file should be verified. It is not a good idea to search for EF only for the 2nd generation. You don't know if the entire file is correct. But if you only need EF for 2nd generation you should probably modify the if ... else statement in /src/regions/ElementaryFileRegion.cs from line 41

[Sh0ckwav3]

My library reads gen 1 and gen 2 (if there are two generations in one file) and this is how the file should be verified.

How do you handle the problem I'm facing that he find twice an identifier but only one is correct?

Im trying also to read booth if booth are present and only one if there is only one. I already modified the if statement in this file, but there is a long way to go untillits finished. Im facing some problems with the XML configuration, its reading not what expected and its hard to find out why.

[Sh0ckwav3]

BTW im stuck here:

When o try to read the VehicleRegistrationNumber i receive two Bytes to much in the front and some to few in the back, so i think the byteoffset is not correct:

I dont know where it comes from, in the xml I think i configured it correctly:

<ElementaryFile Name="CardVehiclesUsed" Identifier="0x0505">
    <UInt16 Name="VehiclePointerNewestRecord"/>
    <Repeat Name="CardVehicleRecords" CountRef="#NoOfCardVehicleRecords">
      <Object Name="CardVehicleRecord">
        <UInt24 Name="VehicleOdometerBegin"/>
        <UInt24 Name="VehicleOdometerEnd"/>
        <TimeReal Name="VehicleFirstUse"/>
        <TimeReal Name="VehicleLastUse"/>
        <Object Name="VehicleRegistration">
          <UInt8 Name="VehicleRegistrationNation"/>
          <InternationalString Name="VehicleRegistrationNumber" Length="13"/>
        </Object>
        <BCDString Name="VuDataBlockCounter" Size="2"/>
        <InternationalString Name="vehicleIdentificationNumber" Size="17"/>
      </Object>
    </Repeat>
  </ElementaryFile>

i have to check now all manually and try to fix this.

[mpi-wl]

How do you handle the problem I'm facing that he find twice an identifier but only one is correct?

Im trying also to read booth if booth are present and only one if there is only one. I already modified the if statement in this file, but there is a long way to go untillits finished. Im facing some problems with the XML configuration, its reading not what expected and its hard to find out why.

In my opinion, you don't want to solve your problem. The code for this project was created before 2G and it is probably not enough to change only the configuration file. You must make changes to the code - e.g. checking the third byte, changes in certificates and signatures. In addition, some data structures have changed and it is not enough to change the field length in the configuration file.

I dont know where it comes from, in the xml I think i configured it correctly:

Incorrect: VehicleRegistrationNumber is 14 bytes long. The first byte is the code page and the next (13 bytes) is the registration number. You must check the project code to make sure it matches the documentation. The registration number is filled with spaces (0x20), which you then delete (Trim ()).

[ramasofficial]

Sh0ckwav3 - do you have code with gen2 where is working correctly now?

[Sh0ckwav3]

Hey @ramasofficial, I dont have any code... sorry. I know mostly everything about the files now but i stoped developing.

[Dilemma725]

Hi guys,

I've spent some time looking at the G2 vehicle files and they don't seem too hard to decode. I wanted to validate the correctness of the layout of the file, not to get the full dataset out of them. I've written some code in C# to get all the segments out of the file, which can be translated to the data quite easily if you use the code on this project (tachograph-reader).

See attached file for example code on how to decode G2 vehicle files.

Hope this helps someone!

TachoReader.txt

[Dilemma725]

Btw, does anyone have a G2 drivercard file that he can share? I've got G2 vehicle files, but no drivercard files. I'd like to have a look at one of those as well.

Thanks

[ramasofficial]

Btw, does anyone have a G2 drivercard file that he can share? I've got G2 vehicle files, but no drivercard files. I'd like to have a look at one of those as well.

Thanks

Hi, i have gen2 driver files, you can download in this issue: https://github.com/jugglingcats/tachograph-reader/issues/55

If you need more, write me.

[Dilemma725]

Hi!

Thanks a lot. Time for some binary exploring :)

[ramasofficial]

Hi!

Thanks a lot. Time for some binary exploring :)

You’re welcome. If you write a code, share code here.

[Dilemma725]

The structure test code is working for G1 & G2 VU files and G1 & G2 card files. Need to clean up the code a bit before i post it here.

[ddan87]

Hi @Dilemma725 , your example is very interesting. In my case there are some problems with decoding the ddd file, for example a lot of invalid values for VIN or CompanyName and so on.

I noticed that you process the data file without the xml writer

df = VehicleUnitDataFileG2.Create();
df.Process(filename);

Is it just because an extract of your code? Or isn't your goal to parse the ddd in xml? :) Thank so much.

[ramasofficial]

The structure test code is working for G1 & G2 VU files and G1 & G2 card files. Need to clean up the code a bit before i post it here.

We waiting code. :)

[Dilemma725]

Is it just because an extract of your code? Or isn't your goal to parse the ddd in xml? :)

My goal is not to parse the files to XML. It is to validate the structure of the file. This means that i must find a way to decode the files and check if all blocks are present and that there are no structural problems in the file. I'm not intenting to check the signatures, rather to check all pointers and blocks in the files to make sure these are correct. This way, i can save the file if correct and request a new remote download if i detect corruption in the file.

This way, i get a better chance to get a complete set of data for all vehicles and drivers (supporting an FMS system).

I fixed a few bugs in the cyclic buffer for the driver cards in my code. I will try to post the complete code next week.

[Dilemma725]

Found some time :) TachoFileChecker.zip

It is a Visual Studio 2017 project btw.