search

EmotiBit / ofxEmotiBit

OpenFrameworks library and examples for the EmotiBit
MIT License
33 stars 7 forks source link

DataParser parsing error leading to extra columns in parsed file #219

Open produceconsumerobot opened 11 months ago

produceconsumerobot commented 11 months ago

Describe the bug image From dataset 'XenboX at TRI 2023-04-16/Data/Bob/2023-04-16_15-42-28-096912'

produceconsumerobot commented 11 months ago

Raw data:

6948200,20977,11,PI,1,100,1395,1366,1364,1369,1408,1399,1394,1379,1353,1338,1343 6948200,20978,11,PR,1,100,1488,1495,1487,1489,1517,1498,1507,1490,1469,1476,1491 6948200,20979,11,PG,1,100,289,282,272,265,298,274,294,266,329,259,253 6948185,20980,2,EA,1,100,0.030296,0.030296 6948185,20981,2,EL,1,100,26396.400391,26396.400391 6948179,20982,1,T1,1,100,34.919 6948194,20983,1,TH,1,100,26.513 6948187,20984,3,AX,1,100,0.743,0.745,0.744 6948187,20985,3,AY,1,100,0.002,0.004,0.003 6948187,20986,3,AZ,1,100,-0.503,-0.501,-0.503 6948187,20987,3,GX,1,100,0.366,0.366,0.397 6948187,20988,3,GY,1,100,0.214,0.214,0.244 6948187,20989,3,GZ,1,100,0.214,0.214,0.214 6948187,20990,3,MX,1,100,21,21,21 6948187,20991,3,MY,1,100,55,54,54 6948187,20992,3,MZ,1,100,-107,-108,-105 6948301,20993,10,PI,1,100,1373,1400,1410,1370,1410,1382,1377,1391,1399,1349 6948301,20994,10,PR,1,100,1490,1485,1504,1478,1483,1476,1497,1471,1495,1455 6948301,20995,10,PG,1,100,274,278,276,270,266,302,271,294,295,276 6948259,20996,1,EA,1,100,0.030295 6948259,20997,1,EL,1,100,26396.599609 6948301,20998,3,AX,1,100,0.744,0.745,0.743 6948301,20999,3,AY,1,100,0.001,0.005,0.002 6948301,21000,3,AZ,1,100,-0.499,-0.501,-0.500 6948301,21001,3,GX,1,100,0.397,0.336,0.305 6948301,21002,3,GY,1,100,0.244,0.244,0.244 6948301,21003,3,GZ,1,100,0.214,0.214,0.244 6948301,21004,3,MX,1,100,22,21,21 6948301,21005,3,MY,1,100,54,54,55 6948301,21006,3,MZ,1,100,-108,-105,-107 6948330,21007,4,EM,1,100,RS,RB,2023-04-16_15-42-28-096912.csv,PS,MN 6948392,21008,1,SF,1,100,0.0000 6948407,21009,10,PI,1,100,1374,1374,1381,1373,1367,1349,1349,1370,1382,1386 6948407,21010,10,PR,1,100,1492,1486,1494,1491,1491,1532,1474,1501,1486,1502 6948407,21011,10,PG,1,100,268,270,266,293,266,266,285,271,258,276 6948392,21012,2,EA,1,100,0.030296,0.030296 6948392,21013,2,EL,1,100,26396.400391,26396.000000 6948319,21014,1,T1,1,100,34.930 6948334,21015,1,TH,1,100,26.490 6948380,21016,2,AX,1,100,0.743,0.743 6948380,21017,2,AY,1,100,0.002,0.002 6948380,21018,2,AZ,1,100,-0.500,-0.500 6948380,21019,2,GX,1,100,0.397,0.336 6948380,21020,2,GY,1,100,0.244,0.214 6948380,21021,2,GZ,1,100,0.244,0.214 6948380,21022,2,MX,1,100,20,21 6948380,21023,2,MY,1,100,53,54 6948380,21024,2,MZ,1,100,-109,-107 6948500,21025,9,PI,1,100,1382,1373,1374,1383,1389,1399,1411,1376,1374 6948500,21026,9,PR,1,100,1504,1459,1490,1469,1490,1505,1536,1489,1470 6948500,21027,9,PG,1,100,294,281,270,269,276,273,257,267,315 6948459,21028,1,EA,1,100,0.030296 6948459,21029,1,EL,1,100,26396.000000 6948452,21030,1,T1,1,100,34.863 6948467,21031,1,TH,1,100,26.490 6948474,21032,2,AX,1,100,0.743,0.744 6948474,21033,2,AY,1,100,0.004,0.002 6948474,21034,2,AZ,1,100,-0.503,-0.501 6948474,21035,2,GX,1,100,0.336,0.366 6948474,21036,2,GY,1,100,0.214,0.214 6948474,21037,2,GZ,1,100,0.183,0.214 6948474,21038,2,MX,1,100,21,21 6948474,21039,2,MY,1,100,54,53 6948474,21040,2,MZ,1,100,-108,-106 6948532,21041,4,EM,1,100,RS,RB,2023-04-16_15-42-28-096912.csv,PS,MN 6948607,21042,11,PI,1,100,1371,1371,1369,1421,1364,1357,1379,1369,1394,1399,1376 6948607,21043,11,PR,1,100,1473,1484,1489,1460,1483,1477,1493,1486,1512,1508,1471 6948607,21044,11,PG,1,100,266,261,267,261,253,253,266,236,274,281,281 6948592,21045,2,EA,1,100,0.030296,0.030296 6948592,21046,2,EL,1,100,26396.000000,26396.199219 6948585,21047,1,T1,1,100,34.870 6948600,21048,1,TH,1,100,26.460 6948594,21049,3,AX,1,100,0.744,0.744,0.743 6948594,21050,3,AY,1,100,0.004,0.002,0.002 6948594,21051,3,AZ,1,100,-0.501,-0.501,-0.501 6948594,21052,3,GX,1,100,0.366,0.366,0.366 6948594,21053,3,GY,1,100,0.214,0.214,0.214 6948594,21054,3,GZ,1,100,0.214,0.244,0.244 6948594,21055,3,MX,1,100,21,21,20 6948594,21056,3,MY,1,100,55,54,54 6948594,21057,3,MZ,1,100,-107,-107,-106 6948659,21058,1,SF,1,100,0.0000 6948700,21059,9,PI,1,100,1379,1341,1396,1369,1433,1364,1367,1368,1378 6948700,21060,9,PR,1,100,1469,1469,1490,1489,1479,1474,1459,1498,1488 6948700,21061,9,PG,1,100,340,259,288,269,258,278,244,271,239 6948659,21062,1,EA,1,100,0.030296 6948659,21063,1,EL,1,100,26396.400391 6948674,21064,2,AX,1,100,0.745,0.744 6948674,21065,2,AY,1,100,0.003,0.004 6948674,21066,2,AZ,1,100,-0.500,-0.504 6948674,21067,2,GX,1,100,0.366,0.366 6948674,21068,2,GY,1,100,0.214,0.183 6948674,21069,2,GZ,1,100,0.244,0.275 6948674,21070,2,MX,1,100,21,22 6948674,21071,2,MY,1,100,54,55 6948674,21072,2,MZ,1,100,-108,-107 6948900,21073,4,EM,1,100,RS,RB,2023-04-16_15-42-28-096912.csv,PS,MN 6948895,21074,19,PI,1,100,1397,1396,1378,1361,1357,1344,1358,1369,1381,1347,1369,1416,1378,1368,1373,1376,1372,1346,1323 6948895,21075,19,PR,1,100,1496,1510,1486,1460,1473,1488,1495,1470,1478,1486,1492,1505,1519,1482,1487,1483,1492,1528,1502 6948895,21076,19,PG,1,100,271,280,298,303,243,247,244,269,290,265,264,270,282,260,268,270,270,258,261 6948873,21077,3,EA,1,100,0.030296,0.030296,0.030296 6948873,21078,3,EL,1,100,26396.199219,26396.000000,26396.400391 6948866,21079,2,T1,1,100,34.873,34.945 6948882,21080,2,TH,1,100,26.459,26.503 6948875,21081,5,AX,1,100,0.745,0.746,0.744,0.744,0.744 6948875,21082,5,AY,1,100,0.002,0.003,0.002,0.003,0.004 6948875,21083,5,AZ,1,100,-0.502,-0.503,-0.503,-0.499,-0.501 6948875,21084,5,GX,1,100,0.336,0.397,0.397,0.366,0.366 6948875,21085,5,GY,1,100,0.183,0.244,0.214,0.183,0.214 6948875,21086,5,GZ,1,100,0.214,0.214,0.244,0.244,0.214 6948875,21087,5,MX,1,100,21,21,21,21,21 6948875,21088,5,MY,1,100,55,53,55,56,53 6948875,21089,5,MZ,1,100,-106,-107,-107,-108,-107 6949000,21090,11,PI,1,100,1390,1360,1379,1388,1364,1377,1375,1417,1365,1400,1371 6949000,21091,11,PR,1,100,1461,1505,1501,1518,1481,1510,1492,1463,1472,1506,1487 6949000,21092,11,PG,1,100,257,241,270,262,270,294,265,242,259,270,272 6948945,21093,1,EA,1,100,0.030296 6948945,21094,1,EL,1,100,26396.000000 6948987,21095,3,AX,1,100,0.743,0.743,0.743 6948987,21096,3,AY,1,100,0.003,0.004,0.003 6948987,21097,3,AZ,1,100,-0.500,-0.500,-0.502 6948987,21098,3,GX,1,100,0.336,0.366,0.397 6948987,21099,3,GY,1,100,0.183,0.214,0.214 6948987,21100,3,GZ,1,100,0.183,0.183,0.214 6948987,21101,3,MX,1,100,21,21,20 6948987,21102,3,MY,1,100,54,55,54 6948987,21103,3,MZ,1,100,-106,-106,-107 6948978,21104,1,BV,1,100,3.73 6948978,21105,1,B%,1,100,49 6949100,21106,4,EM,1,100,RS,RB,2023-04-16_15-42-28-096912.csv,PS,MN 6949079,21107,1,SF,1,100,0.0000 6949093,21108,9,PI,1,100,1398,1403,1378,1371,1390,1379,1358,1379,1373 6949093,21109,9,PR,1,100,1491,1518,1485,1487,1477,1501,1459,1486,1517 6949093,21110,9,PG,1,100,271,238,253,288,301,267,265,265,273 6949079,21111,2,EA,1,100,0.030296,0.030296 6949079,21112,2,EL,1,100,26396.000000,26396.000000 6949005,21113,1,T1,1,100,34.941 6949020,21114,1,TH,1,100,26.503 6949067,21115,2,AX,1,100,0.745,0.746 6949067,21116,2,AY,1,100,0.003,0.003 6949067,21117,2,AZ,1,100,-0.502,-0.502 6949067,21118,2,GX,1,100,0.366,0.366 6949067,21119,2,GY,1,100,0.244,0.214 6949067,21120,2,GZ,1,100,0.214,0.214 6949067,21121,2,MX,1,100,22,21 6949067,21122,2,MY,1,100,54,54 6949067,21123,2,MZ,1,100,-109,-107 6949200,21124,11,PI,1,100,1382,1392,1400,1371,1376,1385,1410,1377,1386,1376,1361 6949200,21125,11,PR,1,100,1493,1493,1495,1489,1512,1499,1480,1489,1513,1456,1476 6949200,21126,11,PG,1,100,255,283,273,271,287,275,306,256,288,288,244 6949145,21127,1,EA,1,100,0.030296 6949145,21128,1,EL,1,100,26396.000000 6949139,21129,1,T1,1,100,34.974 6949154,21130,1,TH,1,100,26.395 6949187,21131,3,AX,1,100,0.743,0.744,0.746 6949187,21132,3,AY,1,100,0.004,0.005,0.001 6949187,21133,3,AZ,1,100,-0.500,-0.501,-0.502 6949187,21134,3,GX,1,100,0.336,0.305,0.336 6949187,21135,3,GY,1,100,0.214,0.183,0.214 6949187,21136,3,GZ,1,100,0.275,0.275,0.244 6949187,21137,3,MX,1,100,21,21,21 6949187,21138,3,MY,1,100,54,54,54 6949187,21139,3,MZ,1,100,-107,-109,-108 6949301,21140,4,EM,1,100,RS,RB,2023-04-16_15-42-28-096912.csv,PS,MN 6949293,21141,9,PI,1,100,1368,1374,1353,1385,1375,1379,1372,1379,1393 6949293,21142,9,PR,1,100,1491,1520,1453,1480,1491,1504,1485,1461,1501 6949293,21143,9,PG,1,100,268,250,265,262,266,265,244,277,255 6949279,21144,2,EA,1,100,0.030296,0.030296 6949279,21145,2,EL,1,100,26396.000000,26396.000000 6949272,21146,1,T1,1,100,34.918 6949287,21147,1,TH,1,100,26.395 6949307,21148,3,AX,1,100,0.745,0.744,0.744 6949307,21149,3,AY,1,100,0.004,0.003,0.004 6949307,21150,3,AZ,1,100,-0.500,-0.500,-0.504 6949307,21151,3,GX,1,100,0.366,0.397,0.336 6949307,21152,3,GY,1,100,0.214,0.183,0.214 6949307,21153,3,GZ,1,100,0.275,0.244,0.275 6949307,21154,3,MX,1,100,21,21,21 6949307,21155,3,MY,1,100,55,55,54 6949307,21156,3,MZ,1,100,-107,-108,-107 6949345,21157,1,SF,1,100,0.0000 6949400,21158,10,PI,1,100,1401,1404,1378,1372,1358,1392,1382,1352,1374,1373 6949400,21159,10,PR,1,100,1506,1500,1504,1492,1473,1505,1482,1486,1480,1474 6949400,21160,10,PG,1,100,291,296,285,269,254,286,239,256,271,276 6949345,21161,1,EA,1,100,0.030296 6949345,21162,1,EL,1,100,26395.800781 6949405,21163,1,T1,1,100,34.929 6949387,21164,2,AX,1,100,0.743,0.745 6949387,21165,2,AY,1,100,0.003,0.003 6949387,21166,2,AZ,1,100,-0.500,-0.501 6949387,21167,2,GX,1,100,0.397,0.336 6949387,21168,2,GY,1,100,0.183,0.183 6949387,21169,2,GZ,1,100,0.244,0.244 6949387,21170,2,MX,1,100,21,21 6949387,21171,2,MY,1,100,55,55 6949387,21172,2,MZ,1,100,-107,-106 6949502,21173,4,EM,1,100,RS,RB,2023-04-16_15-42-28-096912.csv,PS,MN 6949507,21174,11,PI,1,100,1400,1395,1406,1353,1378,1375,1369,1370,1405,1386,1377 6949507,21175,11,PR,1,100,1499,1485,1484,1487,1504,1498,1494,1486,1488,1501,1484 6949507,21176,11,PG,1,100,261,275,293,246,258,283,271,273,294,271,271 6949479,21177,2,EA,1,100,0.030296,0.030296 6949479,21178,2,EL,1,100,26396.000000,26395.800781 6949420,21179,1,TH,1,100,26.354 6949507,21180,3,AX,1,100,0.745,0.742,0.744 6949507,21181,3,AY,1,100,0.004,0.003,0.002 6949507,21182,3,AZ,1,100,-0.498,-0.501,-0.502 6949507,21183,3,GX,1,100,0.305,0.336,0.336 6949507,21184,3,GY,1,100,0.244,0.214,0.183 6949507,21185,3,GZ,1,100,0.244,0.214,0.244 6949507,21186,3,MX,1,100,21,21,21 6949507,21187,3,MY,1,100,54,54,54 6949507,21188,3,MZ,1,100,-107,-107,-108 6949600,21189,9,PI,1,100,1360,1381,1354,1374,1363,1342,1348,1377,1389 6949600,21190,9,PR,1,100,1475,1445,1472,1490,1480,1468,1484,1482,1480 6949600,21191,9,PG,1,100,255,268,249,269,316,269,267,265,302 6949545,21192,1,EA,1,100,0.030296 6949545,21193,1,EL,1,100,26396.000000 6949539,21194,1,T1,1,100,34.898 6949554,21195,1,TH,1,100,26.354 6949587,21196,2,AX,1,100,0.743,0.746 6949587,21197,2,AY,1,100,0.004,0.001 6949587,21198,2,AZ,1,100,-0.503,-0.501 6949587,21199,2,GX,1,100,0.336,0.305 6949587,21200,2,GY,1,100,0.214,0.214 6949587,21201,2,GZ,1,100,0.275,0.244 6949587,21202,2,MX,1,100,21,21 6949587,21203,2,MY,1,100,53,53 6949587,21204,2,MZ,1,100,-108,-107 6949703,21205,4,EM,1,100,RS,RB,2023-04-16_15-42-28-096912.csv,PS,MN 6949679,21206,1,SF,1,100,0.0000 6949707,21207,11,PI,1,100,1378,139

Parsed data:

412518.499281,1681695374.749144,6947919.000,20947,2,EA,1,100,0.030296 412518.566281,1681695374.816145,6947986.000,20947,2,EA,1,100,0.030296 412518.632281,1681695374.882145,6948052.000,20963,1,EA,1,100,0.030295 412518.698781,1681695374.948645,6948118.500,20980,2,EA,1,100,0.030296 412518.765281,1681695375.015145,6948185.000,20980,2,EA,1,100,0.030296 412518.839281,1681695375.089146,6948412520.125281,1681695376.375150,6949545.000,21192,1,EA,1,100,0.030296 412520.192281,1681695376.442150,6949612.000,21210,2,EA,1,100,0.030296 412520.259281,1681695376.509151,6949679.000,21210,2,EA,1,100,0.030296 412520.325782,1681695376.575651,6949745.500,21226,2,EA,1,100,0.030296 412520.392282,1681695376.642151,6949812.000,21226,2,EA,1,100,0.030296 412520.459282,1681695376.709151,6949879.000,21243,1,EA,1,100,0.030296 412520.525782,1681695376.775652,6949945.500,21258,2,EA,1,100,0.030296 412520.592282,1681695376.842152,6950012.000,21258,2,EA,1,100,0.030296

produceconsumerobot commented 11 months ago

This may be a situation where the parser falls behind on writing data

produceconsumerobot commented 11 months ago

Re-parsed the file and got a similar issue, but in a different location in the file:

413257.319560,1681696113.571783,7686739.000,8642,1,EA,1,100,0.030297 413257.386060,1681696113.638283,7686805.500,8659,2,EA,1,100,0.030297 413257.452560,1681696113.704783,7686872.000,8659,2,EA,1,100,0.030297 413257.519560,1681696113.771783,7686939.000,8676,1,EA,1,100,0.030297 413257.586060,1681696113.413258.719560,1681696114.971788,7688139.000,8871,1,EA,1,100,0.030297 413258.786060,1681696115.038288,7688205.500,8886,2,EA,1,100,0.030297 413258.852560,1681696115.104788,7688272.000,8886,2,EA,1,100,0.030297 413258.919560,1681696115.171788,7688339.000,8903,1,EA,1,100,0.030297 413258.986060,1681696115.238289,7688405.500,8919,2,EA,1,100,0.030297