mf4 file read Big-endian buffer not supported on little-endian compiler

[zorce]

Try to read a mf4 file (version 4.10) But get the following exception. Unexpected error: (<class 'ValueError'>, ValueError('Big-endian buffer not supported on little-endian compiler',),<traceback object at 0x055D8328>)

[ratal]

Hi, Thanks for the report but coudl you give more details on the error, complete trace ? It will firstly allow me to understand in which file and around where you have this issue.

[zorce]

Hi, Can't get the full stacktrace for some reason.. fetched within the module? Something in file md4reader.py:1896 -> def linearConv(vect, cc_val): The full printout if I let it run for quite some time:

Unexpected error: (<class 'ValueError'>, ValueError('Big-endian buffer not supported on little-endian compiler',),<traceback object at 0x054D8300>) dataRead crashed, back to python data reading Unexpected error: (<class 'ValueError'>, ValueError('Big-endian buffer not supported on little-endian compiler',),<traceback object at 0x04A45788>) dataRead crashed, back to python data reading Unexpected error: (<class 'ValueError'>, ValueError('Big-endian buffer not supported on little-endian compiler',),<traceback object at 0x04A45788>) dataRead crashed, back to python data reading Unexpected error: (<class 'ValueError'>, ValueError('Big-endian buffer not supported on little-endian compiler',),<traceback object at 0x054E4738>) dataRead crashed, back to python data reading Unexpected error: (<class 'ValueError'>, ValueError('Big-endian buffer not supported on little-endian compiler',),<traceback object at 0x054E48A0>) dataRead crashed, back to python data reading C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36-32\lib\site-packages**mdfreader\mdf4reader.py:1896*: Run timeWarning: invalid value encountered in multiply return vect P2 + P1

[zorce]

Seems to be fetched by the try: at line 1113 in md4reader.py try: return self.readBitarray(fid.read(self.CGrecordLength * self.numberOfRecords), channelSet) except: # still memory efficient but takes time print('Unexpected error:', exc_info(), file=stderr) print('dataRead crashed, back to python data reading', file=stderr)

[ratal]

Can you try to reinstall mdfreader and check if cython module dataRead is properly compiled ? (it should have correct management of byteorder) I also added byteorder handling in case dataRead crashes lines 1201:1255, please check if it solved your issue because byteorder handling could also be missing from lines 1116:1136.

[zorce]

Have reinstalled it now, and investigated a bit more. It seems to be only some records that failes. All channels are read. Could it be that some signals is big-endian and most is little-endian?

[ratal]

Yes, it could be. You can confirm by analysing your file using MDFValidator from Vector, freely downloadable. dataRead is still crashing after your reinstall ? If you are on windows, please make sure you have an appropriate compiler for cython (issue #50 )

[zorce]

Found some issues in MDFValidator after conversion from blf2mdf "Property 7: Hidden VTAB after linear conversion (CANape specific feature, not specified in MDF, will be ignored by other tools)" The values are then not parsed correctly in mdfreader? For example I have a signal with some ambient temperature. In etas mda and vSignalizer the value is displayed correctly rangeing between 15.5 - 20.0 degrees. But when displaying this signal in mdfreader as totaly weird numbers. eg. first few samples [774900239 791677455 791677455 808454671 808454671 825231887 825231887 842009103 842009103 858786319 858786319 875563535 875563535 892340751 892340751 909117967 909117967 925895183 925895183 942672399 942672399 959449615 959449615 976226831 976226831 993004047 993004047 3213839 3213839 19991055 19991055 36768271 36768271 53545487 53545487 70322703 70322703 87099919 87099919 103877135 103877135 120654351

[ratal]

Hi Zorce, First what has changed since 15 days ago when you had timeWarning: invalid value encountered in multiply ? (hard to follow) If you have a file not compliant with MDF specification, it is difficult to guess how to process the channel. VTAB is most probably a CC block property (conversion, maybe value table) What is cc_type value ? (1 = linear conversion) cctype = [4,10] are related to tab cc precision and cc_flags are not yet considered, what are its values ?

[zorce]

Hi, I still have those timeWarnings. But now trying with some more logfiles containing more signals. Cannot see the cc_type, cc_precision ore flags in the conversion section in mdfValidator. One more thing that have changed is that i now have mdf files converted from blf with vector logging converter in vSignalizer.

[ratal]

Hi Zorce, But this is not anymore a mdf 4.x I can not tweak mdf3.x reader when linear conversion, I do not find a way to discriminate a normal linear conversion to hidden VTAB (I do not have it specification or a file to retro-engineer). Number of parameters are same. Seems anyway minor information, just some kind of text to say you are out of range. But do you have errors reading the mdf3 with mdfreader ? If we go back to mdf4 reading issue ; As you start to get familiar to MDFValidator, could you:

1. put a print(chan) line 1195
2. compare output with MDFVAlidator and check for discrepencies
3. before return vect * P2 + P1, put a print(P1,P2,vect). To confirm there might be some NaN

[ratal]

More than a month without feedback, does not seems so critical. Do not hesitate to repoen the issue if needed.

[zorce]

Hi, Sorry have been on vacations without my computer at hand.. Back with the mdf 4.x issue. Printed chan at line 1195. Printout of the vect takes ages... But after some time it prints a hell of alot. Values seem okey. If I concentrate on only one signal eg. in my example Odometer. I put a print in convertChannelData4() function at the end to see the converted data of this signal.. it looks totally wrong. If I look at the signal in vector signalizer I get the correct value of a signal only going in positive direction from 10 -> 17 Attached picture below.

IP_IPTotalOdometer {'data': array([ 2560, 655360, 0, 86, 21711, 4474156, 15001083, 10485569, 82432, 4325376, 10, 2560, 655360, 128, 49222, 2113926, 15464503, 5580306, 1315650, 1327616, 4325376, 10, 2560, 655360, 224, 49267, 8425100, 1427812, 6312733, 4267586, 2048512, 4325376, 10, 2560, 655360, 0, 45249, 10500785, 3990472, 868388, 12002370, 2441728, 4325376, 10, 2560, 655360, 208, 8240, 5245209, 10503007, 7853609, 5057090, 2769408, 4325376, 10, 2560, 655360, 16, 97, 13642889, 13421045, 16477231, 14954306, 3162624, 4325376, 10, 2560, 655360, 48, 33023, 1634942, 12064582, 11632946, 12399170, 3293696, 4325376, 10, 2560, 655360, 64, 24695, 6309807, 9455889, 14961717, 8926530, 3490304, 4325376, 10, 2560, 655360, 200, 61685, 8951272, 11896284, 2103352, 5453890, 3686912, 4325376, 10, 2560, 655360, 64, 43177, 5784866, 10468775, 6153018, 1981250, 3883520, 4325376, 10, 2560, 655360, 72, 26674, 7396701, 9304434, 9219645, 15351106, 4080128, 4325376, 10, 2560, 655360, 128, 10439, 13136458, 7805215, 14892096, 5914690, 4211200, 4325376, 10, 2560, 655360, 152, 64532, 10226916, 1945988, 8233537, 12599618, 4276736, 4325376, 10, 2560, 655360, 196, 14440, 1332863, 4672746, 1574979, 2442050, 4407808, 4325376, 10, 2560, 655360, 52, 11332, 3953435, 6744143, 11889988, 9126978, 4473344, 4325376, 10, 2560, 655360, 240, 1136, 4251021, 351355, 2201930, 11946562, 4866560, 4325376, 10, 2560, 655360, 16, 26652, 15492388, 3863264, 12516939, 1854530, 4997632, 4325376, 10, 2560, 655360, 72, 25799, 7896256, 10586950, 5792845, 8539458, 5063168, 4325376, 10, 2560, 655360, 148, 31907, 6077018, 12265388, 16042318, 15158850, 5194240, 4325376, 10, 2560, 655360, 36, 44639, 181898, 13660466, 5849425, 5263682, 5325312, 4325376, 10, 2560, 655360, 202, 35410, 6736984, 3686373, 2356305, 217666, 5390848, 4325376, 10, 2560, 655360, 220, 38128, 13022500, 1641368, 15836754, 11883074, 5390848, 4325376, 10, 2560, 655360, 172, 28179, 13649907, 4250186, 12671315, 6837058, 5456384, 4325376, 10, 2560, 655360, 232, 23686, 133568, 8365821, 9178196, 1791042, 5521920, 4325376, 10, 2560, 655360, 16, 54361, 2398094, 5731760, 6078292, 13521986, 5521920, 4325376, 10, 2560, 655360, 16, 1224, 5301965, 10466167, 10126938, 9787970, 5915136, 4325376, 10, 2560, 655360, 126, 1114, 403351, 6258218, 6633819, 4741954, 5980672, 4325376, 10, 2560, 655360, 98, 29796, 4513892, 6900701, 3337307, 16472898, 6046208, 4325376, 10, 2560, 655360, 58, 35905, 4985653, 16456336, 106332, 11361346, 6046208, 4325376, 10, 2560, 655360, 150, 63100, 3727610, 14870338, 13062493, 6315330, 6111744, 4325376, 10, 2560, 655360, 28, 60481, 14334151, 4108021, 9765982, 1269314, 6177280, 4325376, 10, 2560, 655360, 78, 11003, 16021397, 16087976, 6469470, 13000258, 6177280, 4325376, 10, 2560, 655360, 232, 8842, 14561893, 15223899, 3828319, 7954242, 6242816, 4325376, 10, 2560, 655360, 12, 33395, 4649264, 2165767, 8261216, 1400898, 6308352, 4325376, 10, 2560, 655360, 212, 2293, 7855871, 11826016, 15099744, 7299138, 6308352, 4325376, 11, 2816, 720896, 53, 6127, 15151206, 9951673, 5030240, 13131842, 6308352, 4325376, 11, 2816, 720896, 10, 2619, 1802702, 2507027, 11737697, 2253122, 6373888, 4325376, 11, 2816, 720896, 74, 15422, 4115252, 174188, 1799265, 8085826, 6373888, 4325376, 11, 2816, 720896, 132, 32491, 1859483, 6295238, 8506721, 13918530, 6373888, 4325376, 11, 2816, 720896, 12, 54843, 6648578, 4027679, 15214178, 3039810, 6439424, 4325376, 11, 2816, 720896, 142, 52766, 14733929, 12901496, 5275746, 8872514, 6439424, 4325376, 11, 2816, 720896, 56, 28030, 963535, 8078034, 12048738, 14770754, 6439424, 4325376, 11, 2816, 720896, 169, 54166, 13698615, 11577899, 1913699, 3826498, 6504960, 4325376, 11, 2816, 720896, 58, 7357, 1505436, 3477637, 8621155, 9659202, 6504960, 4325376, 12, 3072, 786432, 134, 43649, 4117251, 8681438, 15394147, 15557442, 6504960, 4325376, 12, 3072, 786432, 61, 49378, 1254507, 6938167, 5390180, 4613186, 6570496, 4325376, 12, 3072, 786432, 200, 56041, 5684433, 16663697, 12097636, 10511426, 6570496, 4325376, 12, 3072, 786432, 171, 41919, 7055673, 1813738, 2093668, 16344130, 6636032, 4325376, 12, 3072, 786432, 70, 54121, 7883424, 4461380, 8801125, 5399874, 6636032, 4325376, 12, 3072, 786432, 225, 64443, 188424, 1997981, 15705189, 11298114, 6636032, 4325376, 12, 3072, 786432, 94, 19639, 8126060, 3334390, 5635685, 353858, 6701568, 4325376, 12, 3072, 786432, 58, 2670, 11642836, 12994384, 12212070, 6252098, 6701568, 4325376, 12, 3072, 786432, 10, 53985, 4005691, 11972265, 2208102, 12084802, 6701568, 4325376, 12, 3072, 786432, 119, 50248, 15962017, 12981251, 8915559, 1140546, 6767104, 4325376, 12, 3072, 786432, 218, 31051, 8509192, 1604700, 15688551, 7038786, 6767104, 4325376, 12, 3072, 786432, 44, 40314, 14624368, 15917749, 5684583, 12871490, 6767104, 4325376, 12, 3072, 786432, 83, 29961, 1185239, 9062927, 12392040, 1992770, 6832640, 4325376, 12, 3072, 786432, 17, 5930, 16345149, 10858600, 2322536, 7825474, 6832640, 4325376, 12, 3072, 786432, 212, 2318, 14668195, 923586, 9095528, 13658178, 6832640, 4325376, 12, 3072, 786432, 138, 6512, 9088010, 15629595, 15802985, 2779458, 6898176, 4325376, 12, 3072, 786432, 117, 50250, 364401, 13231220, 5733481, 8612162, 6898176, 4325376, 12, 3072, 786432, 81, 28813, 16308952, 10832846, 12506473, 14510402, 6898176, 4325376, 12, 3072, 786432, 212, 2051, 10636607, 8303143, 2436970, 3566146, 6963712, 4325376, 12, 3072, 786432, 137, 41440, 9864358, 3218049, 9275498, 9464386, 6963712, 4325376, 12, 3072, 786432, 67, 32262, 1172493, 11567322, 15982954, 15297090, 6963712, 4325376, 12, 3072, 786432, 180, 8219, 8953204, 14608179, 5978987, 4352834, 7029248, 4325376, 12, 3072, 786432, 208, 17958, 3465435, 11685517, 12686443, 10251074, 7029248, 4325376, 12, 3072, 786432, 183, 56615, 12363075, 16561894, 2748011, 16083778, 7029248, 4325376, 12, 3072, 786432, 26, 49939, 1856681, 9247808, 9455468, 5205058, 7094784, 4325376, 12, 3072, 786432, 49, 54826, 10214416, 2655385, 16228460, 11037762, 7094784, 4325376, 12, 3072, 786432, 151, 43310, 159607, 2288626, 6224492, 93506, 7160320, 4325376, 12, 3072, 786432, 155, 6654, 474078, 8409420, 12866413, 5991746, 7160320, 4325376, 12, 3072, 786432, 114, 13891, 5475653, 13940901, 2927981, 11824450, 7160320, 4325376, 12, 3072, 786432, 56, 46649, 1350314, 93753, 2571633, 4223298, 7422464, 4325376, 12, 3072, 786560, 32844, 30490, 2750567, 2236774, 14313841, 7172418, 7422464, 4325376, 12, 3072, 786560, 32823, 8405897, 3420955, 12965010, 9344625, 10121538, 7422464, 4325376, 12, 3072, 786432, 83, 8436564, 12317133, 9537471, 4309873, 13005122, 7422464, 4325376, 12, 3072, 786432, 118, 19661, 3043201, 1260524, 16052081, 15954242, 7422464, 4325376, 12, 3072, 786432, 96, 7826, 3683703, 14102594, 15353206, 4814402, 7750144, 4325376, 12, 3072, 786432, 42, 8440956, 6302508, 12772718, 10449526, 7763522, 7750144, 4325376, 12, 3072, 786560, 32977, 37514, 4330974, 11835291, 5349238, 10712642, 7750144, 4325376, 12, 3072, 786560, 215, 8450335, 9421457, 2640840, 314486, 13596226, 7750144, 4325376, 12, 3072, 786560, 151, 8453967, 6615364, 9240308, 12122230, 16545346, 7815680, 4325376, 12, 3072, 786560, 32855, 5064, 1465080, 16167201, 7021943, 2717506, 7815680, 4325376, 13, 3328, 852096, 61, 50840, 15148460, 10904910, 2052727, 5601090, 7815680, 4325376, 13, 3328, 852096, 42, 6292, 4862815, 924027, 13794935, 8550210, 7815680, 4325376, 13, 3328, 851968, 32832, 25143, 1182483, 11390119, 8825719, 11499330, 7815680, 4325376, 13, 3328, 852096, 231, 57705, 8653766, 3702996, 3790967, 14382914, 7815680, 4325376, 13, 3328, 852096, 98, 43021, 9481849, 4797185, 15467640, 555074, 7881216, 4325376, 13, 3328, 851968, 32821, 40513, 14081836, 8381997, 10498424, 3504194, 7881216, 4325376, 13, 3328, 851968, 232, 8432100, 7045600, 2398810, 5463672, 6453314, 7881216, 4325376, 13, 3328, 852096, 32778, 8397336, 14202515, 5590407, 428920, 9336898, 7881216, 4325376, 13, 3328, 851968, 32936, 8418677, 13839432, 14614964, 12302200, 12286018, 7881216, 4325376, 14, 3584, 917504, 32768, 14634, 3393786, 8500448, 7267448, 15235138, 7881216, 4325376, 14, 3584, 917632, 68, 49561, 3291566, 10053389, 2167161, 1341762, 7946752, 4325376, 14, 3584, 917504, 33019, 8452107, 6712673, 1055546, 13909369, 4290882, 7946752, 4325376, 14, 3584, 917632, 1, 8419964, 12324373, 12373606, 8940153, 7240002, 7946752, 4325376, 14, 3584, 917504, 72, 8439776, 8617672, 2065043, 3905401, 10123586, 7946752, 4325376, 14, 3584, 917504, 178, 44107, 6417019, 9254336, 15713145, 13072706, 7946752, 4325376, 14, 3584, 917632, 33, 8401564, 12995119, 15788268, 10678393, 16021826, 7946752, 4325376, 14, 3584, 917632, 32987, 8454075, 3407330, 2071833, 5643642, 2193986, 8012288, 4325376, 14, 3584, 917504, 21, 3982, 11677590, 13193030, 608890, 5077570, 8012288, 4325376, 15, 3840, 983040, 32913, 8427313, 13684553, 5374835, 12351098, 8026690, 8012288, 4325376, 15, 3840, 983168, 32832, 8403550, 13236476, 10729119, 7381882, 10975810, 8012288, 4325376, 15, 3840, 983040, 32827, 8476, 12953776, 6580940, 2412666, 13859394, 8012288, 4325376, 15, 3840, 983040, 33006, 8568, 1382244, 15277561, 14154874, 31554, 8077824, 4325376, 15, 3840, 983168, 32827, 29784, 16257816, 10146085, 9120123, 2980674, 8077824, 4325376, 15, 3840, 983168, 32877, 8406360, 3272395, 5211474, 4150907, 5864258, 8077824, 4325376, 15, 3840, 983040, 32792, 14957, 11422335, 11221119, 15893115, 8813378, 8077824, 4325376, 15, 3840, 983040, 219, 8448700, 6831922, 15919787, 10792827, 11762498, 8077824, 4325376, 15, 3840, 983040, 94, 57636, 2160356, 8953560, 5823611, 14711618, 8077824, 4325376, 15, 3840, 983168, 33004, 8449223, 8465816, 13652229, 723324, 818242, 8143360, 4325376, 15, 3840, 983040, 32923, 8445278, 13084491, 4850994, 12465532, 3767362, 8143360, 4325376, 15, 3840, 983168, 173, 8437231, 11502847, 5355870, 7561852, 6716482, 8143360, 4325376, 16, 4096, 1048704, 32951, 8441672, 9695154, 12414091, 2527100, 9600066, 8143360, 4325376, 16, 4096, 1048704, 117, 8411403, 4969574, 10297528, 14334844, 12549186, 8143360, 4325376, 16, 4096, 1048576, 32961, 31575, 3994649, 11326436, 9300092, 15498306, 8143360, 4325376, 16, 4096, 1048576, 32779, 62001, 5677005, 7636753, 4199805, 1604930, 8208896, 4325376, 17, 4352, 1114112, 32931, 59618, 3508353, 2308926, 16007549, 4554050, 8208896, 4325376, 17, 4352, 1114240, 32915, 8389659, 13537843, 1961578, 10907261, 7503170, 8208896, 4325376, 17, 4352, 1114112, 77, 8403331, 10424296, 11837847, 6003581, 10452290, 8208896, 4325376, 17, 4352, 1114240, 252, 8403429, 12021146, 742596, 903293, 13335874, 8208896, 4325376, 17, 4352, 1114240, 28, 37520, 874574, 15534065, 12711037, 16284994, 8208896, 4325376, 17, 4352, 1114240, 32914, 32177, 3163649, 15645213, 7610750, 2457154, 8274432, 4325376, 17, 4352, 1114240, 32770, 45593, 903092, 5533258, 2641534, 5340738, 8274432, 4325376, 17, 4352, 1114240, 32808, 8396309, 8618600, 15868279, 14383742, 8289858, 8274432, 4325376, 17, 4352, 1114112, 32846, 2797, 5086747, 3593635, 9348990, 11238978, 8274432, 4325376, 17, 4352, 1114112, 80, 8443248, 11804111, 13666512, 4379774, 14122562, 8274432, 4325376, 17, 4352, 1114240, 32987, 22079, 12705666, 4668669, 16056447, 294722, 8339968, 4325376, 17, 4352, 1114240, 233, 8452151, 8508213, 9039657, 11087231, 3243842, 8339968, 4325376, 17, 4352, 1114240, 248, 8425547, 7306217, 2794326, 6052479, 6192962, 8339968, 4325376, 17, 4352, 1114240, 32925, 8438029, 7842972, 5068419, 1017727, 9076546, 8339968, 4325376, 17, 4352, 1114112, 11, 31871, 13708624, 527024, 12825471, 12025666, 8339968, 4325376, 17, 4352, 1114112, 0, 0, 7171429, 670819, 7366002, 7233901, 7097967, 3105647, 6648436, 6516589, 0, 0, 0, 2315352, 0, 6648691, 0, 0, 7171429, 3101784, 6254706, 670821, 7566177, 2258034, 3106110, 2248037, 2126447, 3944293, 7233392, 4065852, 7618048, 0, 0, 2315352, 0, 0, 0, 0, 0, 6549511, 0, 0, 0, 66560, 0, 80, 13270850, 0, 0, 2315352, 0, 7239265, 0, 0, 7304289, 7479396, 6909472, 2315352, 0, 5267273, 7169396, 2310990, 0, 6549511, 0, 1769480, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 16760129, 4303112, 524288, 83, 6969863, 9178434, 4368648, 524288, 51, 12696657, 1252674, 4368648, 524288, 12314, 833871, 11543874, 4368648, 524288, 243, 3584886, 14101570, 4368648, 524288, 2255, 252110, 16724290, 4368648, 524288, 45307, 15946594, 9581890, 4368648, 524288, 55434, 8112629, 2505026, 4368648, 524288, 41215, 8537481, 12139586, 4368648, 524288, 45130, 11191836, 2506818, 4368648, 524288, 29833, 3153240, 15745346, 4368648, 524288], dtype=uint32), 'unit': '', 'description': 'the total odometer displayed in IP', 'master ': 't71', 'masterType': 0, 'conversion': {'type': 7, 'parameters': {'cc_val': 16777215.0, 'cc_ref': ['Invalid', 0]} }, 'axis': []}

[ratal]

Hi Zorce, Thanks for trying. Could you please also try to print this channel before conversion by convertChannelData4() , to confirm the channel is well read. I made recently some bug fix for records reading in last commit -> what version are you trying with ? Could you please update to Master ? What is worrying me is that t71 is supposed to be a master channel (time) but its conversion type is 7, meaning a value to text/scale conversion table. It does not make sense to define a time channel from a table (which seems to have only one pair of values, cc_val and cc_ref). So either header reading is not correct (mdfinfo) or the file has issues or not compliant to specification. --> Could you check in MDFValidator for this time channel ? (should be in datagroup # 72) What is its channel description and conversion description ?

[zorce]

Hi, Printed in the beginning of the method also, the values are the same. No conversion done? Also now tried to upgrade mdfreader, but it failed to upgrade numpy. Now I can't get numpy running. Under windows failed to import multiarray numpy extension module. Tried to remove numpy and reinstall, but same issue again. :/

[zorce]

After reinstallation of python.. helped with python 3.6.2 instead of 3.6.0 I managed to install latest version of numpy and mdfreader again.. Did a print of the channel again, before and after the convertChanelData4, and the channels are equal before and after the conversion.

[ratal]

So it seems there is an error reading your file. One way to figure out is to print record class each time it initialised by loadInfo() method and compare with MDFValidator. Looking at your vector, does not seem to be a big endian issue, rather a record which is malformed.

[zorce]

Might be some indexing error? Class content print shows t71 contains the IP_IPTotalOdometer channel as you can see bellow. But in fact in MDFvalidator it is in t72! Record class content print Total number of channels : 2 t71 t71 0 (('t71_title', 't71'), '<f8') <d ChannelType 2 DataType 4 IP_IPTotalOdometer IP_IPTotalOdometer 1 (('IP_IPTotalOdometer_title', 'IP_IPTotalOdometer'), '>u4') >I ChannelType 0 DataType 1 Total bits : 88 Calculated total bytes : 12 CG block record bytes length : 11 Datagroup number : 71 Byte aligned : True Hidden bytes : False Master channel : t71 Numpy records format : (('t71_title', 't71'), '<f8') (('IP_IPTotalOdometer_title', 'IP_IPTotalOdometer'), '>u4') VLSD_CG{}

[ratal]

Hi Zorce, MDFValidator counts datagroup from 1 and python from 0, the reason for this offset. Clearly, there is mismatch between expected number of bytes for the record and the computed one nu loadinfo() I guess IP_IPTotalOdometer is in 22bits so readable within standard 4bytes by a PC but actually 3 bytes. You try the following, adding after if self.CGrecordLength < self.recordLength and self.byte_aligned\ and not self.hiddenBytes: self.hiddenBytes = True It will trigger reading with dataRead cython extension thant should better handle this case.

[zorce]

Hi again, And thank you for taking the time! This gave me a more seasonable array of numbers in an rising fashion. [655360,...,720896,...,786432,...,851968,...,917504,...,983040,...,1048576,...,1114112,...]

This matches the graph above but the byte order seem flipped 655360 -> A0000 (hex) -> little endian 0000A => uint32 decimal 10. if i do this for all values: 655360 = 10 720896 = 11 786432 = 12 851968 = 13 917504 = 14 983040 = 15 1048576 = 16 1114112 = 17

Perfect match ^^ so now we are back at the endian-ness then?

[zorce]

Btw. still get those warnings: Unexpected error: (<class 'ValueError'>, ValueError('Big-endian buffer not supported on little-endian compiler',), <traceback object at 0x000000001273B448>)

[ratal]

Yes, unfortunately. Actually, studying dataRead code I wrote some time ago, I could read it should handle properly internally the endian. However, by mistake, I think I tried to correct again afterwards by python. Can you try to comment line 1198 to 1200 ?

if (buf[self[chan].recAttributeName].dtype.byteorder == '>' and byteorder == 'little') or \ buf[self[chan].recAttributeName].dtype.byteorder == '<' and byteorder == 'big': buf[self[chan].recAttributeName] = buf[self[chan].recAttributeName].newbyteorder()

[zorce]

That did not help unfortunately. The values become int16 big endian ore something 2560, 2816, 3072.. Also in MDFvalidator I can see the cn_data_type is set to 1 = UNSIGNED_MOTOROLA for all the values that ends up in the exception case here for the dataRead. In that code you are defaulting the endian of bitarray to little. Is it possible to make it based on the cn_data_type instead?

[ratal]

Just to confirm, you do a getChannelData to visualise the array or a plot ? The raw data in your file are unint16, so normal for those values, it has to be converted by 'conversion': {'type': 7, 'parameters': {'cc_val': 16777215.0, 'cc_ref': ['Invalid', 0]}}

[zorce]

Have been using a for loop to get all the stuff. eg. `for channel in yop.masterChannelList: for signal in yop.masterChanelList[channel]:

do stuff here`

But tested quickly to use the getChannelData also to see if it differ. After the conversion for this particular signal only values equal to 16777215.0 will be translated to 'invalid' via the valueToTextConv(...) No endianess convertions in the convertChannelData.

[ratal]

Hi Zorce, I had a deeper look to your issue and I understood the empty recarray passed to dataRead using big endian was problematic (meaning of error). Even though dataRead is internally able to cope with endianess, the buffer can nnot pass through the buffer in cython code unsing another endian than the native one. So I had to fake everything native in channel format description and endian handled internally in dataRead, outputing native byte order You should find this modification in latest commit. Can you try it (after recompilation of dataRead, python setup.py develop)?

[zorce]

Hi, Had some installation issues, but after installing pip install lxml I could run the latest version. I get this printout "Cannot compress data, please install bcols and blosc" The mdf file reads now without the exception: Unexpected error: (<class 'ValueError'>, ValueError('Big-endian buffer not supported on little-endian compiler',),<traceback object at 0x055D8328>) So that is good! But seem to still be something missing printout like this: line 1603: if channelName == 'IP_IPTotalOdometer': print(channel) gives me the following instead of [655360,...,720896,...,786432,...,851968,...,917504,...,983040,...,1048576,...,1114112,...] [10, 11, 12, 13, 14, 15, 16, 17]

[ratal]

Hi Zorce, printout is for new feature implemented to compress data, it should not impact your issue. For the displayed vector, I guess it is not yet converted ? Conversion you showed on 6th July was 'description': 'the total odometer displayed in IP', 'master ': 't71', 'masterType': 0, 'conversion': {'type': 7, 'parameters': {'cc_val': 16777215.0, 'cc_ref': ['Invalid', 0]} }, 'axis': []} For a type 7 corresponding to value to text/scale conversion, cc_val and cc_ref seems too short, this might be another issue. --> can you show in mdfvalidator what is the channel block ?

[zorce]

Hi, I think the conversion type 7 is only for that one value to indigate invalid data. But all the rest of the data seem to be endian flipped? as you can see in the picture below cn_data_type is motorola, but many other signals in same mdf file is signed intel for example. IP_IPTotalOdometer channelblock in mdfvalidator:

[ratal]

Dear Zorce, Sorry, I did not ask for the correct block. Conversion details are in the conversion block, pointed by cn_cc_conversion in this channel block. Inside you should see conversion type 7 corresponding to text/scale. I am interested to compare conversion block content versus what mdfreader parsed. I expect discrepencies. Regarding endian, I think this should be solved as I could reproduce similar issue as you had with other file and this seems fixed. Also your data are continuously increasing.

[zorce]

Hi, Attached below the conversion block also. Hope this gives some clue.

[ratal]

Hi Zorce, Looking at this conversion block, I do not understand how you can have integers values as you expect. In my understanding that default text is empty txt '' and otherwise 'invalid'. It also corresponds to what you sent 15 days ago and parsed by mdfreader. Could you check your file ? However, I identified bugs with this kind of conversion I am going to fix.

[ratal]

Hi Zorce, I fixed bugs for this kind of conversion TABX. You can have a try but I doubt it would change anything, I guess there is still a conversion missing.

[zorce]

Hi Ratal, Will try to take some more logs and try out your latest commits next week, due to a business trip. Also I will try to take some more logs. Do you really think it is a conversion issue or just that the data is displayed with the wrong endian. The latest picture above also shows the correct physical values for cc_phy_range_min = 10 and cc_phy_range_max = 17. That's the same as I see in vSignalizyer (graph) previously. Maybe still some issue with the readBitarray method not using the correct cn_data_type for some reason. Could not verify this until next week, just a taught.

[ratal]

Hi Zorce, I am not 100% sure but I think I fixed the endian issue using other files having similar issue as you have. But let's do: odo=numpy.array([655360,720896, 786432,851968],dtype='u4') target=numpy.array([10,11, 12,13],dtype='u4') odo/target -> array([65536, 65536, 65536, 65536], dtype=uint32) after a odo.byteswap(True), odo/target -> array([256, 256, 256, 256], dtype=uint32) So maybe indeed endian is not yet correct (256 coefficient is most likely), I will look at it but still a conversion factor is missing and I think it is inside next conversion block cc_ref[1] (or cc_ref[0]), can you show it ? By the way, dataRead was properly recompiled ? (not having ValueError('Big-endian buffer.. is mostly coming from python but inside dataRead was also introduced modifications for endian handling)

[zorce]

Hi, Just did a test with the latest, pull --rebase; python setup.py develop. I get an error at line 2131 Traceback (most recent call last): File "PreProcessing.py", line 142, in main(args) File "PreProcessing.py", line 28, in main yop = mdfreader.mdf(file) File "C:\mypath\mdfreader\mdf. py", line 147, in init compression=compression) File "C:\mypath\mdfreader\mdfr eader.py", line 392, in read convertAfterRead, filterChannelNames, compression) File "C:\mypath\mdfreader\mdf4 reader.py", line 1534, in read4 self._convertAllChannel4() File "C:\mypath\mdfreader\mdf4 reader.py", line 1646, in _convertAllChannel4 [self._convertChannel4(channelName) for channelName in self] File "C:\mypath\mdfreader\mdf4 reader.py", line 1646, in [self._convertChannel4(channelName) for channelName in self] File "C:\mypath\mdfreader\mdf4 reader.py", line 1634, in _convertChannel4 self.setChannelData(channelName, self._getChannelData4(channelName)) File "C:\mypath\mdfreader\mdf4 reader.py", line 1561, in _getChannelData4 channelName, self.convert_tables)[channelName] File "C:\mypath\mdfreader\mdf4 reader.py", line 1610, in _convertChannelData4 vect = valueToTextConv(vect, conversion_parameter['cc_val'], conversion_parameter['cc_ref']) File "C:\mypath\mdfreader\mdf4 reader.py", line 2131, in valueToTextConv val_count = int(len(cc_val)) TypeError: object of type 'float' has no len()

val_count = int(len(cc_val))

[zorce]

The cc_ref[0] only points to the text block. added below cc_ref[1] (default value) points to 0x0000 source address of the channel data? the dataRead seems to be correctly recompiled. How can i confirm this the best way?

[ratal]

Hi Zorce, I am bit lost on this one. I tested with other files having similar endian channels and have correct handling. Also checked with other files for the lookup table for text/Scaling and goes ok. The cc meaning is that if the value is higher than 16777215; value is 'invalid' The channel block says that cc_phy_range_min and max are directly within what you expect. So my guess is there is still a raw data reading issue, not conversion issue finally. Most probably as you identified, still an endian issue, with dataRead. Could you show maybe the input given to dataRead for this channel (around line 1242) ? By comparing with channel block, we could identify the issue.

[zorce]

Hi ratal,

Did a printout for the input parameters, bytesdata is large so omitted here.

self[chan].bitCount = 24 self[chan].signalDataType = 1 self[chan].nativeRecordFormat[1] = u4 self.numberOfRecords = 1598 self.CGrecordLength = 11 self[chan].bitOffset = 0 self[chan].posByteBeg = 8 self[chan].posByteEnd = 12

The output again from dataRead is [2560... 2816... 3072... etc] 2560 toHex = A00 A00 as uint16 - big endian (AB) = 2560 A00 as uint16 - little endian (BA) = 10 (correct value)

When building latest version I get some warnings for dataRead and numpy deprecated api. Full printout below:

C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\Lib\site-packages\mdfreader-git>python setup.py develo p Compiling dataRead.pyx because it changed. [1/1] Cythonizing dataRead.pyx running develop running egg_info creating mdfreader.egg-info writing mdfreader.egg-info\PKG-INFO writing dependency_links to mdfreader.egg-info\dependency_links.txt writing entry points to mdfreader.egg-info\entry_points.txt writing requirements to mdfreader.egg-info\requires.txt writing top-level names to mdfreader.egg-info\top_level.txt writing manifest file 'mdfreader.egg-info\SOURCES.txt' reading manifest file 'mdfreader.egg-info\SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '*' under directory 'docs\build\html' warning: no files found matching 'mdfreader.pdf' under directory 'docs\build\latex' warning: no files found matching 'README' writing manifest file 'mdfreader.egg-info\SOURCES.txt' running build_ext building 'dataRead' extension creating build creating build\temp.win-amd64-3.6 creating build\temp.win-amd64-3.6\Release C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC :\Users\dennis.delin\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy\core\include -IC:\Users\dennis. delin\AppData\Local\Programs\Python\Python36\include -IC:\Users\dennis.delin\AppData\Local\Programs\Python\Python36 \include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Microsoft Vis ual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.14393.0\ucrt" "-IC:\Progr am Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.14393 .0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.14393.0\um" "-IC:\Program Files (x86)\Windows Kit s\10\include\10.0.14393.0\winrt" /TcdataRead.c /Fobuild\temp.win-amd64-3.6\Release\dataRead.obj dataRead.c c:\users\dennis.delin\appdata\local\programs\python\python36\lib\site-packages\numpy\core\include\numpy\npy_1_7_dep recated_api.h(12) : Warning Msg: Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_17 API_VERSION dataRead.c(5910): warning C4244: '=': conversion from 'unsigned long' to 'unsigned char', possible loss of data dataRead.c(6294): warning C4244: '=': conversion from 'unsigned long' to 'unsigned char', possible loss of data dataRead.c(6312): warning C4334: '<<': result of 32-bit shift implicitly converted to 64 bits (was 64-bit shift int ended?) dataRead.c(6492): warning C4244: '=': conversion from '__int64' to 'long', possible loss of data dataRead.c(11598): warning C4244: 'initializing': conversion from 'double' to 'float', possible loss of data dataRead.c(11604): warning C4244: 'initializing': conversion from 'double' to 'float', possible loss of data creating C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\Lib\site-packages\mdfreader-git\build\lib.win -amd64-3.6 C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\x86_amd64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /M ANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\libs /LIBP ATH:C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Mic rosoft Visual Studio 14.0\VC\LIB\amd64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\LIB \amd64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.14393.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86 )\Windows Kits\NETFXSDK\4.6.1\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.14393.0\um\x64" /EXPORT:PyInit_dataRead build\temp.win-amd64-3.6\Release\dataRead.obj /OUT:build\lib.win-amd64-3.6\dataRead.cp36-w in_amd64.pyd /IMPLIB:build\temp.win-amd64-3.6\Release\dataRead.cp36-win_amd64.lib dataRead.obj : warning LNK4197: export 'PyInit_dataRead' specified multiple times; using first specification Creating library build\temp.win-amd64-3.6\Release\dataRead.cp36-win_amd64.lib and object build\temp.win-amd64-3. 6\Release\dataRead.cp36-win_amd64.exp Generating code Finished generating code copying build\lib.win-amd64-3.6\dataRead.cp36-win_amd64.pyd -> Creating c:\users\dennis.delin\appdata\local\programs\python\python36\lib\site-packages\mdfreader.egg-link (link to .) mdfreader 0.2.6 is already the active version in easy-install.pth Installing mdfconverter-script.py script to C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\Scripts Installing mdfconverter.exe script to C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\Scripts

Installed c:\users\dennis.delin\appdata\local\programs\python\python36\lib\site-packages\mdfreader-git Processing dependencies for mdfreader==0.2.6 Searching for lxml==3.8.0 Best match: lxml 3.8.0 Adding lxml 3.8.0 to easy-install.pth file

Using c:\users\dennis.delin\appdata\local\programs\python\python36\lib\site-packages Searching for Cython==0.26 Best match: Cython 0.26 Adding Cython 0.26 to easy-install.pth file Installing cygdb-script.py script to C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\Scripts Installing cygdb.exe script to C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\Scripts Installing cython-script.py script to C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\Scripts Installing cython.exe script to C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\Scripts Installing cythonize-script.py script to C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\Scripts Installing cythonize.exe script to C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\Scripts

Using c:\users\dennis.delin\appdata\local\programs\python\python36\lib\site-packages Searching for sympy==1.1.1 Best match: sympy 1.1.1 Adding sympy 1.1.1 to easy-install.pth file

Using c:\users\dennis.delin\appdata\local\programs\python\python36\lib\site-packages Searching for numpy==1.13.1 Best match: numpy 1.13.1 Adding numpy 1.13.1 to easy-install.pth file

Using c:\users\dennis.delin\appdata\local\programs\python\python36\lib\site-packages Searching for mpmath==0.19 Best match: mpmath 0.19 Adding mpmath 0.19 to easy-install.pth file

Using c:\users\dennis.delin\appdata\local\programs\python\python36\lib\site-packages Finished processing dependencies for mdfreader==0.2.6

[zorce]

Hi, Looked a bit at the dataRead also and in this case I guess we end up at line 86 in dataRead.pyx, bit count <= 32 with datatype 1. I did a print out of byteorder = little. we have byteorder = little and signalDataType = 1 (unsigned motorola) but we end up in the else case here and therefore swap bytes. Typo here maybe? Maybe the swap is done in a mid-little-endian (CDAB) instead of Little-endian (DCBA)

Good online analyzer http://www.scadacore.com/field-tools/programming-calculators/online-hex-converter/ put A0000 as hex input

[ratal]

Hi Zorce, The previous error ValueError('Big-endian buffer not supported on little-endian compiler') forced to pass to DataRead native order (so on your computer, little endian) numpy arrays as input (dataRead takes a numpy array already initiliased as input, modifies its values and returns it). Our target is to output native order array with correct processing in dataRead, so swapping is needed. Here it is done by numpy using .byteswap(). In https://docs.scipy.org/doc/numpy-1.13.0/user/basics.byteswapping.html Change the byte-ordering of the underlying data, leaving the dtype interpretation as it was. This is what arr.byteswap() does. So dataRead should return data in native order after swapped bytes. So this is the thinking but there might still be some flaw as you spotted out with mid-little-endian as the data are on 24bits=3bytes stored in the end in 4bytes. ABCD should become CBAD but we do DCBA, something like this. Probably in this case, numpy byteswap() is not appropriate and bytes should be swapped in cython, I will study this. Never know, you could also print .dtype of the returned array of dataRead and confirm it little endian ? For the warnings, I also have it and comes from .c compilation, I have little leverage on it.

[zorce]

Hi ratal, See that you have done some changes! The values for odometer now looks corrcet now!! Awesome! Only now have one small new issue ^^ The valueToTextConv or something before it have got broken. Stacktrace Traceback (most recent call last): File "PreProcessing.py", line 142, in main(args) File "PreProcessing.py", line 28, in main yop = mdfreader.mdf(file) File "C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\lib\site-packages\mdfreader-git\mdfreader\mdf. py", line 147, in init compression=compression) File "C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\lib\site-packages\mdfreader-git\mdfreader\mdfr eader.py", line 392, in read convertAfterRead, filterChannelNames, compression) File "C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\lib\site-packages\mdfreader-git\mdfreader\mdf4 reader.py", line 1536, in read4 self._convertAllChannel4() File "C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\lib\site-packages\mdfreader-git\mdfreader\mdf4 reader.py", line 1646, in _convertAllChannel4 [self._convertChannel4(channelName) for channelName in self] File "C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\lib\site-packages\mdfreader-git\mdfreader\mdf4 reader.py", line 1646, in [self._convertChannel4(channelName) for channelName in self] File "C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\lib\site-packages\mdfreader-git\mdfreader\mdf4 reader.py", line 1634, in _convertChannel4 self.setChannelData(channelName, self._getChannelData4(channelName)) File "C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\lib\site-packages\mdfreader-git\mdfreader\mdf4 reader.py", line 1563, in _getChannelData4 channelName, self.convert_tables)[channelName] File "C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\lib\site-packages\mdfreader-git\mdfreader\mdf4 reader.py", line 1612, in _convertChannelData4 vect = valueToTextConv(vect, conversion_parameter['cc_val'], conversion_parameter['cc_ref']) File "C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36\lib\site-packages\mdfreader-git\mdfreader\mdf4 reader.py", line 2139, in valueToTextConv val_count = int(len(cc_val)) TypeError: object of type 'float' has no len()

IP_IPTotalOdometer printed in _convertChannelData4 at valueToTextConv function call cc_val 16777215.0 cc_ref ['Invalid', 0]

[ratal]

Hi Zorce, Good to hear we finally progressed ! Can you try to change ? : val_count = int(len(cc_val)) by

if isinstance(cc_val, tuple):
        val_count = len(cc_val)
 else:
        val_count = 1

[zorce]

Hi,

Thanks, did that and found even one more small issue.. Not entirely your fault but seem to be some weird unicode values in cc_ref so for some values it cant encode the cc_ref to ascii. Maybe because of thes string initialization? temp = empty(len(vect), dtype='S' + str(maxlen))

I guess we could ether ignore special chars when we read them from file and store in channel or read them with python .encode()

[zorce]

Hi again, After some more investigation it seems like the valueToTextConv is not working quite as it suppose I guess? Or is the point of it to actually replace all values with default once if not match the cc_ref. Currently it looks like all the values not found in cc_ref gets a default value instead of only update the values that have a cc_ref.

Example an other signal IP_Hour values in vector are 10..... then 11... Conversion type 7 = valueToTextConv. cc_val = (24, 25, 26, 27, 28, 29, 30, 31) cc_ref ['invalid'...., 0] After conversion the channel data will be all 0.

My expectation would be that if the vector contained say 10, 10, 24, 11, 11 the data array would be 10, 10, 'invalid', 11, 11.

also regarding the encoding I removed all non ascii like so cc_ref = [str(ref).encode('ascii', 'ignore') for ref in cc_ref] before the maxlen calc

This is okey if there is also a way to get the raw unconverted values for each channel. eg. change the yop.getChannelData to getChannelDataRaw and getChannelDataPhys?

[ratal]

Hi, Yes, getChannelDataRaw could be interesting addition but I think you should have what you expect out of the box. Also, you can have it easily in the mean time by deactivating convertAfterRead=False argument and accessing raw data directly from the mdfreader dictionnary. Indeed code should be corrected. Last cc_ref is default and a null link means 1:1 conversion. Can you try the following shot ?:

    maxlen = max([len(str(ref)) for ref in cc_ref])
    temp = empty(len(vect), dtype='U' + str(maxlen))  # initialize empty array with max length
    # checks for scaling
    try:
        from sympy import lambdify, symbols
    except:
        print('Please install sympy to convert channel ', file=stderr)
    X = symbols('X')
    for ref in range(len(cc_ref)):
        if isinstance(cc_ref[ref], CCBlock):
            if cc_ref[ref]['cc_type'] == 3:
                # formula to be applied
                cc_ref[ref] = lambdify(X, cc_ref[ref]['cc_ref']['Comment']
                                       , modules='numpy', dummify=False)
            elif cc_ref[ref]['cc_type'] == 1: # linear conversion
                cc_ref[ref] = lambdify(X, str(cc_val[1]) + '* X + ' + str(cc_val[0])
                                       , modules='numpy', dummify=False)
            else:
                print('implement missing conversion formulae')
        elif (PythonVersion > 3 and not isinstance(cc_ref[ref], str)) or \
                (PythonVersion < 3 and not isinstance(cc_ref[ref], unicode)):  # identity, non conversion
            cc_ref[ref] = lambdify(X, 'X'
                                    , modules='numpy', dummify=False)
    key_index = where(vect[0] == cc_val)[0]  # look up for first value in vect
    if not len(key_index) == 0:  # value corresponding in cc_val
        temp[0] = cc_ref[key_index[0]]
    else:  # default value
        if callable(cc_ref[-1]):
            temp[0] = cc_ref[-1](vect[0])
        else:
            temp[0] = cc_ref[-1]
    for Lindex in range(1, len(vect)):
        if vect[Lindex] == vect[Lindex - 1]:  # same value as before, no need to look further
            temp[Lindex] = temp[Lindex - 1]
        else:  # value changed from previous step
            key_index = where(vect[Lindex] == cc_val)[0]
            if not len(key_index) == 0:  # found match
                temp[Lindex] = cc_ref[key_index[0]]
            else:  # default
                if callable(cc_ref[-1]):
                    temp[Lindex] = cc_ref[-1](vect[Lindex])
                else:
                    temp[Lindex] = cc_ref[-1]
    return asarray(temp)

[zorce]

Hi,

Will test the convertAfterRead flag also but, first the snippet you sent me. I get the following stack trace: vect = valueToTextConv(vect, conversion_parameter['cc_val'], conversion_para meter['cc_ref']) File "C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36-32\lib\site -packages\mdfreader\mdf4reader.py", line 2158, in valueToTextConv cc_ref[ref] = lambdify(X, str(cc_val[1]) + '* X + ' + str(cc_val[0]) TypeError: 'float' object is not subscriptable

I get your point, if only convert some and therefore mixing the types in the array you will lose the possibility to do numpy calculations on it (eg. average, mean, min, max etc.) hence it would be good with the getChannelDataRaw for those kind of things and get physical values for other lookups etc.

Also noted on some of my signals (many) they include multiple conversion references. First a type 7 valueToText and a linearConv. Temperature signal cc_ref[0] cc_ref[1] linear conv

The convertChannelData only does first conversion.

[ratal]

Hi, My test files do not implement your case so I could not confirm code before you try it. Maybe following should work ?

                cc_ref[ref] = lambdify(X, str(cc_ref[ref]['cc_val'][1]) + '* X + ' 
                                       + str(cc_ref[ref]['cc_val'][0])
                                       , modules='numpy', dummify=False)

Indeed valueToText function will convert your numerical data into text according to conversion type purpose, sometimes not easy to use but there are use cases were it makes sense like gear name, etc.

[zorce]

Hi,

That worked better but still had some issue with the array initialization File "C:\Users\dennis.delin\AppData\Local\Programs\Python\Python36-32\lib\site -packages\mdfreader\mdf4reader.py", line 2145, in valueToTextConv temp = empty(len(vect), dtype='U' + str(maxlen)) # initialize empty array w ith max length MemoryError

[ratal]

Hi, I know, I have an urgent todo reducing memory consumption. But just for info, can you give ?:

• file size
• number of channels
• your platform available memory If you are courageous and to quickly troubleshoot you, you can try to gain lot of memory in case you have many channels or metadata by not saving in CommentBlock the 'xml_tree' key and the 'Comment' key for raw text (convert them into temporary variable) For some files, I could reduce memory by 75%

[zorce]

Hi,

Got it to work quite good now when running with on demand conversion as you mentioned. For this file: Filesize: 134 311 112 bytes (128mb) numeber of channels 591 (183 master channels) Memory on computer Total 16268 (16gb) Chached 9927 Available 9996 Free 133

I will take a look at the CommentBlock as the logFiles i use is converted from blf with dbc databases to mdf before analyzis with mdfreader.