Invalid _pstate_changes.csv files

[mpoquet]

While running some experiments on Grid'5000, I got two invalid _pstate_changes.csv files (out of 1282).

Data: buggy_pstate_changes.zip

Some IO problem probably occurred during the execution of these instances. But it might be possible to improve the exportation system such that the risk to see this kind of problem is reduced.

Currently running these instances again to make sure the problem is not more grave. Update: got the same problem after a new execution of the instances, the problem seems deterministic regarding Batsim inputs!

[mpoquet]

Detected those files with the following python command:

filenames = glob.glob('./*_pstate_changes.csv')

for f in filenames:
    try:
        data = pd.read_csv(f)
    except:
        print('cannot read {}'.format(f))

[mpoquet]

The following script displays (maybe not every) invalid lines.

import re
import glob

filenames = glob.glob('./*_pstate_changes.csv')
r = re.compile('''^[^,]+,[^,]+,[^,]+$''')

for fname in filenames:
    with open(fname) as f:
        line_nb = 1
        for line in f:
            line = line.strip()
            if r.match(line) is None:
                print('{f}:{l}... "{c}"'.format(f=fname, l=line_nb, c=line))
            line_nb += 1

Result:

./90e1b789_pstate_changes.csv:14993... "4.53444e+06,117,4.54.53445e+06,117,13"
./90e1b789_pstate_changes.csv:18539... "5.89263e+06,7-105.85.89272e+06,4,-2"
./90e1b789_pstate_changes.csv:126461... "2 32,-1"
./224b32a4_pstate_changes.csv:17309... "1.82085e+06,34-35 38-65,1.81.82086e+06,34-35 38-65,13"
./224b32a4_pstate_changes.csv:20182... "2.36649e+06,43-46 120-122.3662.36649e+06,43-46 120-121,13"
./224b32a4_pstate_changes.csv:26022... "3.23396e+06,44-47 91-1023.233.2355e+06,0-3,-1"
./224b32a4_pstate_changes.csv:33923... "4.2114e+06,0-3 71-79 88-4.2114e4.21141e+06,4 10 31-37 48-70,13"
./224b32a4_pstate_changes.csv:36489... "4.584.58614e+06,41-4.58614e+06,122,13"
./224b32a4_pstate_changes.csv:41350... "5.02998e+06,2-17 34-47 75.02998e5.02998e+06,18-33,-1"
./224b32a4_pstate_changes.csv:48524... "5.63634e+06,97 100-101 15.63635.63635e+06,14-15 38-39 72-73 80-84 92-96,13"
./224b32a4_pstate_changes.csv:190715... "2.39788e+07,53-2.39774e+072.39788e+07,0-9 42-44,0"
./224b32a4_pstate_changes.csv:195702... "2.44957e2.44928e+07,2.44957e+07,0-31,-1"
./224b32a4_pstate_changes.csv:198302... "2.49832.49675e+07,712.49833e+07,68-70 103-104,0"
./224b32a4_pstate_changes.csv:200944... "2.5284e+07,13-29 33 42.52864e+07,13-29 33 46-47,0"
./224b32a4_pstate_changes.csv:203362... "2.55185e+07,102.55016e+07,107,13"
./224b32a4_pstate_changes.csv:206164... "2.58601e+07,7 11 33-34 2.58569e+07,7 35-37,13"
./224b32a4_pstate_changes.csv:206165... "2.58574e+07,9,-2.58603e+07,7 11 33-34 37-40,0"
./224b32a4_pstate_changes.csv:208721... "2.60793e+07,412.6083e+07,41-44 49 61-68 70-71 73-76 79-82 86 89-104,0"
./224b32a4_pstate_changes.csv:210932... "2.63797e+07,94,2.63729e+07,25-26 45 47-48 119,13"
./224b32a4_pstate_changes.csv:210933... "2.63731e+07,46 92-93 992.63797e+07,88,-2"
./224b32a4_pstate_changes.csv:213570... "2.65234e+07,16-2.65217e+07,0-15,-1"
./224b32a4_pstate_changes.csv:213572... "2.65219e+07,2.65235e+07,16-23 113 116-122,0"
./224b32a4_pstate_changes.csv:221586... "2.76408e+07,84 93 109-114,-22.76268e+07,117-118,-2"
./224b32a4_pstate_changes.csv:224513... "2.80387e+07,51 66,12.80601e+07,0 13 51,13"
./224b32a4_pstate_changes.csv:227456... "2.83643e+07,49-50 62 62.83628e+07,63-64 68-69 93-95 98-103 106 113-114,0"
./224b32a4_pstate_changes.csv:230407... "2.87928e+07,0-2 52.87815e+07,7,-1"
./224b32a4_pstate_changes.csv:233104... "2.94078e+07,2.94499e+07,111-114,-1"
./224b32a4_pstate_changes.csv:235454... "2.95795e+07,12.9582e+07,16-19,-1"
./224b32a4_pstate_changes.csv:238048... "2.9834e+07,8-21 56-2.9831e+07,8-14,0"
./224b32a4_pstate_changes.csv:238050... "2.9831e+07,12.98341e+07,8-39 56-59,-2"
./224b32a4_pstate_changes.csv:240644... "3.00814e+07,0-14 33-36 99-101 118-123.00799e+07,0-15 32-33 36-39 82-86 94-96,-2"
./224b32a4_pstate_changes.csv:240645... "3.00799e+07,34-3.00816e+07,0-14,0"
./224b32a4_pstate_changes.csv:243040... "3.01775e+07,82 92-110 126,-23.01744e+07,0-15,-2"
./224b32a4_pstate_changes.csv:243042... "3.01744e+07,93.01775e+07,1-15 82 92-99,-1"
./224b32a4_pstate_changes.csv:245727... "3.05009e+07,96-103.05175e+07,100-103 105,-2"
./224b32a4_pstate_changes.csv:248060... "3.07052e+07,104-107,133.07035e+07,124-125,13"
./224b32a4_pstate_changes.csv:250480... "3.08777e+07,17-28 3.08761e+07,47-54,-2"
./224b32a4_pstate_changes.csv:250482... "3.08761e+07,47-53.08777e+07,120-121,13"
./224b32a4_pstate_changes.csv:252799... "3.10208e+07,0,-3.10101e+07,19-24 31-32 34-43 98-103,0"
./224b32a4_pstate_changes.csv:252801... "3.10101e+07,903.1021e+07,0,0"
./224b32a4_pstate_changes.csv:257223... "3.13478e+07,42-43.13577e+07,112-113,-1"
./224b32a4_pstate_changes.csv:259966... "3.16462e+07,0 4-5 12-13 16-18 24-39 41-46 56-63 3.1701e+07,79-110,0"
./224b32a4_pstate_changes.csv:262527... "3.20923e+07,64-63.20842e+07,21-27 32-34 52 56-57 59-68,-2"
./224b32a4_pstate_changes.csv:265219... "3.25775e+07,48-793.25816e+07,48-79,-2"
./224b32a4_pstate_changes.csv:268005... "3.31198e+07,78 83.31161e+07,2-3,0"
./224b32a4_pstate_changes.csv:272396... "3.34909e+07,0-3 53.34883e+07,0-3 5-6 8-9,-2"
./224b32a4_pstate_changes.csv:274950... "3.36368e+07,8-11,3.36397e+07,8-11,13"
./224b32a4_pstate_changes.csv:277521... "3.38642e+07,93.38502e+07,38-53 62-77,-1"
./224b32a4_pstate_changes.csv:285560... "3.48365e+07,2-3,13.48467e+07,91,0"
./224b32a4_pstate_changes.csv:288268... "3.53387e+07,3.53108e+07,13-44,13"
./224b32a4_pstate_changes.csv:290908... "3.55301e+07,0-3.55332e+07,0-7,-2"
./224b32a4_pstate_changes.csv:293831... "3.58305e+07,8-3.58266e+07,0-3,-1"
./224b32a4_pstate_changes.csv:299028... "3.62843e+07,83-863.62942e+07,87-88 90 98-100,0"
./224b32a4_pstate_changes.csv:301526... "3.65626e+07,60-63 77 83-88 90-92 96 3.65949e+07,52-55,0"
./224b32a4_pstate_changes.csv:307330... "3.77216e+07,77-92 102-1163.77259e+07,2-3 77-92 102-115,-1"
./224b32a4_pstate_changes.csv:310402... "3.81732e+07,64-73 79-80 82-893.81589e+07,64-68,-2"
./224b32a4_pstate_changes.csv:319200... "3.98334e+07,3.98399e+07,32-33,0"
./224b32a4_pstate_changes.csv:324581... "4.05427e+07,0-3 18-24 29-35,-24.05386e+07,0 13-15 17-24,0"
./224b32a4_pstate_changes.csv:327529... "4.06373e+07,0-5,04.06342e+07,0-7 18-24 29-31,-2"
./224b32a4_pstate_changes.csv:330031... "4.09131e+07,18-19 26-31 48-49 75-76 120-1234.09113e+07,120-123,13"
./224b32a4_pstate_changes.csv:332663... "4.10853e+07,22-23 30-31,4.10909e+07,24 49,0"
./224b32a4_pstate_changes.csv:334809... "4.12999e+07,95-96 103-104 1184.12895e+07,22-25 30-57,-2"
./224b32a4_pstate_changes.csv:334813... "4.12897e+07,24.13001e+07,22-25 27-44 79-81 95-96 103-104 118-120,0"
./224b32a4_pstate_changes.csv:339521... "4.16817e+07,4.16791e+07,2-3 15-25 27-29,0"
./224b32a4_pstate_changes.csv:339524... "4.16792e+07,92-99 14.16819e+07,2-3 15-16 23-25 27-29 88-93,0"
./224b32a4_pstate_changes.csv:344458... "4.20136e+07,54-55 4.20109e+07,48-49,0"
./224b32a4_pstate_changes.csv:346824... "4.22999e+07,49-50 92-98 14.22782e+07,0-11 18-26 32-38 43-56 122-123,-2"
./224b32a4_pstate_changes.csv:346827... "4.22782e+07,0-11 18-26 32-38,4.23e+07,51-56 108-109 119-122,0"
./224b32a4_pstate_changes.csv:349222... "4.25673e+07,32-41 48-49 96-974.25648e+07,42-47 96-97,-1"
./224b32a4_pstate_changes.csv:351849... "4.30257e+07,0-31,-24.29989e+07,111-112,-2"
./224b32a4_pstate_changes.csv:354641... "4.34185e+07,12-4.34128e+07,10-11,-2"
./224b32a4_pstate_changes.csv:357067... "4.40696e+07,100-101 105 4.40867e+07,93-95,13"
./224b32a4_pstate_changes.csv:365331... "4.52885e+07,32-44.52924e+07,40 42-52 121-124,0"
./224b32a4_pstate_changes.csv:369807... "4.57109e+07,40-41 53-4.57088e+07,86-87 103 112 122,13"
./224b32a4_pstate_changes.csv:369809... "4.57088e+07,40-41 53-57 72-75 86-87 103 112 1224.57111e+07,86-87 103 112 122,-2"
./224b32a4_pstate_changes.csv:377795... "4.66254e+07,4.66294e+07,79-85 96-120,13"
./224b32a4_pstate_changes.csv:380469... "4.72012e+07,1124.7189e+07,100-107,13"
./224b32a4_pstate_changes.csv:380473... "4.71895e+07,974.72014e+07,113-114,-2"
./224b32a4_pstate_changes.csv:388010... "4.82126e+07,44 49-50 54.82094e+07,40-43 45-48 51-58,0"
./224b32a4_pstate_changes.csv:388012... "4.82094e+07,44 494.82128e+07,44 49-50 59-60 65-75,-2"
./224b32a4_pstate_changes.csv:390787... "4.87429e+07,4.86961e+07,98,-1"
./224b32a4_pstate_changes.csv:393638... "4.95328e+07,74.95306e+07,109-115 127,13"
./224b32a4_pstate_changes.csv:393640... "4.95307e+07,40-59 72 76-77 814.9533e+07,109-115 127,13"
./224b32a4_pstate_changes.csv:396253... "4.98141e+07,4.98112e+07,0-15 32-39 72-79,0"
./224b32a4_pstate_changes.csv:396257... "4.98112e+07,0-14.98143e+07,0-15 32-39 72-79,0"
./224b32a4_pstate_changes.csv:399291... "5.09969e+07,84-5.09607e+07,0-31,-1"
./224b32a4_pstate_changes.csv:402217... "5.17108e+07,0-1 5.17163e+07,2-5,-1"
./224b32a4_pstate_changes.csv:410884... "5.38004e+07,11 15.3797e+07,6-9,-2"
./224b32a4_pstate_changes.csv:416609... "5.4967e+07,7-9 86 88-89 106-105.49698e+07,7 10-11 72-76 85 110-111,0"
./224b32a4_pstate_changes.csv:418999... "5.51439e+07,57-58 61-66,5.51472e+07,57-58 61-66,-2"
./224b32a4_pstate_changes.csv:421664... "5.56271e+07,111-15.56242e+07,122-127,13"
./224b32a4_pstate_changes.csv:421666... "5.56243e+07,43-47 95.56271e+07,39-42,13"
./224b32a4_pstate_changes.csv:428603... "5.62656e+07,4-7 12-19 64-5.62638e+07,73-79 93-95 102 111-115,0"
./224b32a4_pstate_changes.csv:430759... "5.66959e+07,5.67044e+07,40-42,-1"
./224b32a4_pstate_changes.csv:432768... "5.69795e+07,92 96-99 111-113 115-117,135.69779e+07,23 64 66-69 92 96-99 111-1135.69795e+07,23 64 66-69 92 96-99 111-113 115-120,-1"
./224b32a4_pstate_changes.csv:434333... "5.70266e+07,20-22 64-5.70248e+07,23-24 30-31 62-63,-25.70268e+07,64-66 92 96-99 111-113 115-120,-2"
./224b32a4_pstate_changes.csv:436972... "5.73169e+07,22-5.73403e+07,20,-1"
./224b32a4_pstate_changes.csv:441556... "5.77766e+07,25 30 32-34 46-48 575.77741e+05.77766e+07,25 30 32-34 46-48,-1"

[mpoquet]

Some data to reproduce the error (not clean yet). reproduce.zip

[mickours]

It looks like 2 IO streams are mixed. Maybe stream the outputs correctly to the file at runtime instead of at the end of the simulation would fix the issue?

[mpoquet]

The issue is old, the exportation of data has been improved since the issue has been opened.

We should check if the problem still exists...

[mpoquet]

We should really check this with current Batsim.

[Mommessc]

We definitely should check this.

[mpoquet]

What about checking this @Mommessc? :D

[Mommessc]

After struggling with your very old data I managed to run the batsim simulation but the python script did not report any faulty line.

[mpoquet]

OK, I close this issue, this has probably been fixed a long time ago.