wa6gz
commented
3 weeks ago - Adds necessary code for open once/read many tests for CCDEF
- Fixes issue in CCDEF where some NaNs weren't writing correctly due to an error in the baseline correction
Closed wa6gz closed 2 weeks ago
wa6gz
commented
3 weeks ago 
github-actions[bot]
commented
3 weeks ago Benchmark results:
________________________________________________________________
Format: waveform_benchmark.formats.ccdef.CCDEF_Compressed
(CCDEF compressed format)
Record: ./data/waveforms/mimic_iv/waves/p100/p10079700/85594648/85594648
214981 seconds x 6 channels
255177600 timepoints, 199126720 samples (78.0%)
________________________________________________________________
Channel summary information:
signal fs(Hz) Bit resolution Channel length(s)
II 249.89 0.005(mV) 212497
III 249.89 0.005(mV) 5
V 249.89 0.005(mV) 212497
aVR 249.89 0.005(mV) 212492
Pleth 124.94 0.000244(NU) 212486
Resp 62.47 0.000244(Ohm) 212497
________________________________________________________________
Output size: 102219 KiB (4.21 bits/sample)
CPU time: 10.5000 sec
Wall Time: 10.5013 s
________________________________________________________________
Fidelity check via read_waveforms:
Chunk Numeric Samples NaN Samples
# Errors / Total % Eq SNR NaN Values Match
Signal: II
0 0/ 51212896 100.000 151.9 Y (7584)
1 0/ 58944 100.000 153.0 Y (1216)
2 0/ 1818752 100.000 151.8 Y (1408)
Signal: III
0 0/ 960 100.000 151.1 Y (320)
Signal: V
0 0/ 51213248 100.000 152.0 Y (7232)
1 0/ 58944 100.000 153.0 Y (1216)
2 0/ 1818752 100.000 151.7 Y (1408)
Signal: aVR
0 0/ 3982176 100.000 151.8 Y (1184)
1 0/ 47229696 100.000 152.2 Y (6144)
2 0/ 58944 100.000 153.0 Y (1216)
3 0/ 1818752 100.000 152.6 Y (1408)
Signal: Pleth
0 0/ 25610240 100.000 inf Y (0)
1 0/ 29440 100.000 inf Y (0)
2 0/ 909440 100.000 inf Y (0)
Signal: Resp
0 0/ 12805120 100.000 154.8 Y (0)
1 0/ 15040 100.000 178.5 Y (0)
2 0/ 455040 100.000 155.2 Y (0)
________________________________________________________________
Fidelity check via open/read/close:
Chunk Numeric Samples NaN Samples
# Errors / Total % Eq SNR NaN Values Match
Signal: II
0 0/ 51212896 100.000 151.9 Y (7584)
1 0/ 58944 100.000 153.0 Y (1216)
2 0/ 1818752 100.000 151.8 Y (1408)
Signal: III
0 0/ 960 100.000 151.1 Y (320)
Signal: V
0 0/ 51213248 100.000 152.0 Y (7232)
1 0/ 58944 100.000 153.0 Y (1216)
2 0/ 1818752 100.000 151.7 Y (1408)
Signal: aVR
0 0/ 3982176 100.000 151.8 Y (1184)
1 0/ 47229696 100.000 152.2 Y (6144)
2 0/ 58944 100.000 153.0 Y (1216)
3 0/ 1818752 100.000 152.6 Y (1408)
Signal: Pleth
0 0/ 25610240 100.000 inf Y (0)
1 0/ 29440 100.000 inf Y (0)
2 0/ 909440 100.000 inf Y (0)
Signal: Resp
0 0/ 12805120 100.000 154.8 Y (0)
1 0/ 15040 100.000 178.5 Y (0)
2 0/ 455040 100.000 155.2 Y (0)
________________________________________________________________
Open Once Read Many performance (median of N trials):
OP #seek #read KiB CPU(s) Wall(s) [N]
open 0 -1 0 0.0002 0.0002 [5] open 1 x 214981s, all channels
read 0 -1 0 2.0405 2.0408 [5] open 1 x 214981s, all channels
close 0 -1 0 0.0002 0.0003 [5] open 1 x 214981s, all channels
open 0 -1 0 0.0002 0.0002 [173] open 5 x 500s, all channels
read 0 -1 0 0.0485 0.0486 [173] open 5 x 500s, all channels
close 0 -1 0 0.0001 0.0001 [173] open 5 x 500s, all channels
open 0 -1 0 0.0002 0.0002 [34] open 50 x 50s, all channels
read 0 -1 0 0.2847 0.2847 [34] open 50 x 50s, all channels
close 0 -1 0 0.0002 0.0002 [34] open 50 x 50s, all channels
open 0 -1 0 0.0002 0.0002 [4] open 500 x 5s, all channels
read 0 -1 0 2.6395 2.6398 [4] open 500 x 5s, all channels
close 0 -1 0 0.0002 0.0002 [4] open 500 x 5s, all channels
open 0 -1 0 0.0002 0.0003 [36] open 1 x 214981s, rand channel
read 0 -1 0 0.2759 0.2759 [36] open 1 x 214981s, rand channel
close 0 -1 0 0.0002 0.0002 [36] open 1 x 214981s, rand channel
open 0 -1 0 0.0002 0.0002 [566] open 5 x 500s, rand channel
read 0 -1 0 0.0084 0.0085 [566] open 5 x 500s, rand channel
close 0 -1 0 0.0001 0.0001 [566] open 5 x 500s, rand channel
open 0 -1 0 0.0002 0.0002 [172] open 50 x 50s, rand channel
read 0 -1 0 0.0487 0.0488 [172] open 50 x 50s, rand channel
close 0 -1 0 0.0001 0.0001 [172] open 50 x 50s, rand channel
open 0 -1 0 0.0002 0.0002 [22] open 500 x 5s, rand channel
read 0 -1 0 0.4481 0.4482 [22] open 500 x 5s, rand channel
close 0 -1 0 0.0002 0.0002 [22] open 500 x 5s, rand channel
open 0 -1 0 0.0002 0.0002 [20] open 1 x 214981s, one channel
read 0 -1 0 0.5098 0.5099 [20] open 1 x 214981s, one channel
close 0 -1 0 0.0002 0.0002 [20] open 1 x 214981s, one channel
open 0 -1 0 0.0002 0.0002 [515] open 5 x 500s, one channel
read 0 -1 0 0.0100 0.0100 [515] open 5 x 500s, one channel
close 0 -1 0 0.0001 0.0001 [515] open 5 x 500s, one channel
open 0 -1 0 0.0002 0.0002 [166] open 50 x 50s, one channel
read 0 -1 0 0.0509 0.0509 [166] open 50 x 50s, one channel
close 0 -1 0 0.0001 0.0001 [166] open 50 x 50s, one channel
open 0 -1 0 0.0002 0.0002 [22] open 500 x 5s, one channel
read 0 -1 0 0.4522 0.4522 [22] open 500 x 5s, one channel
close 0 -1 0 0.0001 0.0001 [22] open 500 x 5s, one channel
________________________________________________________________
Read performance (median of N trials):
#seek #read KiB CPU(s) Wall(s) [N]
0 -1 0 2.0720 2.0722 [5] open 1 x 214981s, all channels
0 -1 0 0.0504 0.0504 [186] open 5 x 500s, all channels
0 -1 0 0.3075 0.3075 [33] open 50 x 50s, all channels
0 -1 0 2.8872 2.8875 [4] open 500 x 5s, all channels
0 -1 0 0.2778 0.2779 [36] open 1 x 214981s, one channel
0 -1 0 0.0094 0.0094 [803] open 5 x 500s, one channel
0 -1 0 0.0587 0.0587 [162] open 50 x 50s, one channel
0 -1 0 0.5532 0.5533 [18] open 500 x 5s, one channel
________________________________________________________________
________________________________________________________________
Format: waveform_benchmark.formats.ccdef.CCDEF_Uncompressed
(CCDEF uncompressed format)
Record: ./data/waveforms/mimic_iv/waves/p100/p10079700/85594648/85594648
214981 seconds x 6 channels
255177600 timepoints, 199126720 samples (78.0%)
________________________________________________________________
Channel summary information:
signal fs(Hz) Bit resolution Channel length(s)
II 249.89 0.005(mV) 212497
III 249.89 0.005(mV) 5
V 249.89 0.005(mV) 212497
aVR 249.89 0.005(mV) 212492
Pleth 124.94 0.000244(NU) 212486
Resp 62.47 0.000244(Ohm) 212497
________________________________________________________________
Output size: 401262 KiB (16.51 bits/sample)
CPU time: 1.2237 sec
Wall Time: 1.2238 s
________________________________________________________________
Fidelity check via read_waveforms:
Chunk Numeric Samples NaN Samples
# Errors / Total % Eq SNR NaN Values Match
Signal: II
0 0/ 51212896 100.000 151.9 Y (7584)
1 0/ 58944 100.000 153.0 Y (1216)
2 0/ 1818752 100.000 151.8 Y (1408)
Signal: III
0 0/ 960 100.000 151.1 Y (320)
Signal: V
0 0/ 51213248 100.000 152.0 Y (7232)
1 0/ 58944 100.000 153.0 Y (1216)
2 0/ 1818752 100.000 151.7 Y (1408)
Signal: aVR
0 0/ 3982176 100.000 151.8 Y (1184)
1 0/ 47229696 100.000 152.2 Y (6144)
2 0/ 58944 100.000 153.0 Y (1216)
3 0/ 1818752 100.000 152.6 Y (1408)
Signal: Pleth
0 0/ 25610240 100.000 inf Y (0)
1 0/ 29440 100.000 inf Y (0)
2 0/ 909440 100.000 inf Y (0)
Signal: Resp
0 0/ 12805120 100.000 154.8 Y (0)
1 0/ 15040 100.000 178.5 Y (0)
2 0/ 455040 100.000 155.2 Y (0)
________________________________________________________________
Fidelity check via open/read/close:
Chunk Numeric Samples NaN Samples
# Errors / Total % Eq SNR NaN Values Match
Signal: II
0 0/ 51212896 100.000 151.9 Y (7584)
1 0/ 58944 100.000 153.0 Y (1216)
2 0/ 1818752 100.000 151.8 Y (1408)
Signal: III
0 0/ 960 100.000 151.1 Y (320)
Signal: V
0 0/ 51213248 100.000 152.0 Y (7232)
1 0/ 58944 100.000 153.0 Y (1216)
2 0/ 1818752 100.000 151.7 Y (1408)
Signal: aVR
0 0/ 3982176 100.000 151.8 Y (1184)
1 0/ 47229696 100.000 152.2 Y (6144)
2 0/ 58944 100.000 153.0 Y (1216)
3 0/ 1818752 100.000 152.6 Y (1408)
Signal: Pleth
0 0/ 25610240 100.000 inf Y (0)
1 0/ 29440 100.000 inf Y (0)
2 0/ 909440 100.000 inf Y (0)
Signal: Resp
0 0/ 12805120 100.000 154.8 Y (0)
1 0/ 15040 100.000 178.5 Y (0)
2 0/ 455040 100.000 155.2 Y (0)
________________________________________________________________
Open Once Read Many performance (median of N trials):
OP #seek #read KiB CPU(s) Wall(s) [N]
open 0 -1 0 0.0002 0.0002 [13] open 1 x 214981s, all channels
read 0 -1 0 0.8107 0.8108 [13] open 1 x 214981s, all channels
close 0 -1 0 0.0002 0.0002 [13] open 1 x 214981s, all channels
open 0 -1 0 0.0002 0.0002 [282] open 5 x 500s, all channels
read 0 -1 0 0.0259 0.0260 [282] open 5 x 500s, all channels
close 0 -1 0 0.0001 0.0001 [282] open 5 x 500s, all channels
open 0 -1 0 0.0002 0.0002 [47] open 50 x 50s, all channels
read 0 -1 0 0.2073 0.2073 [47] open 50 x 50s, all channels
close 0 -1 0 0.0001 0.0001 [47] open 50 x 50s, all channels
open 0 -1 0 0.0002 0.0002 [5] open 500 x 5s, all channels
read 0 -1 0 2.0213 2.0215 [5] open 500 x 5s, all channels
close 0 -1 0 0.0001 0.0001 [5] open 500 x 5s, all channels
open 0 -1 0 0.0002 0.0003 [80] open 1 x 214981s, rand channel
read 0 -1 0 0.1047 0.1048 [80] open 1 x 214981s, rand channel
close 0 -1 0 0.0002 0.0002 [80] open 1 x 214981s, rand channel
open 0 -1 0 0.0002 0.0002 [724] open 5 x 500s, rand channel
read 0 -1 0 0.0047 0.0047 [724] open 5 x 500s, rand channel
close 0 -1 0 0.0001 0.0001 [724] open 5 x 500s, rand channel
open 0 -1 0 0.0002 0.0002 [223] open 50 x 50s, rand channel
read 0 -1 0 0.0354 0.0355 [223] open 50 x 50s, rand channel
close 0 -1 0 0.0001 0.0001 [223] open 50 x 50s, rand channel
open 0 -1 0 0.0002 0.0002 [29] open 500 x 5s, rand channel
read 0 -1 0 0.3395 0.3395 [29] open 500 x 5s, rand channel
close 0 -1 0 0.0001 0.0001 [29] open 500 x 5s, rand channel
open 0 -1 0 0.0002 0.0002 [46] open 1 x 214981s, one channel
read 0 -1 0 0.2064 0.2067 [46] open 1 x 214981s, one channel
close 0 -1 0 0.0002 0.0002 [46] open 1 x 214981s, one channel
open 0 -1 0 0.0002 0.0002 [701] open 5 x 500s, one channel
read 0 -1 0 0.0050 0.0050 [701] open 5 x 500s, one channel
close 0 -1 0 0.0001 0.0001 [701] open 5 x 500s, one channel
open 0 -1 0 0.0002 0.0002 [220] open 50 x 50s, one channel
read 0 -1 0 0.0360 0.0360 [220] open 50 x 50s, one channel
close 0 -1 0 0.0001 0.0001 [220] open 50 x 50s, one channel
open 0 -1 0 0.0002 0.0002 [29] open 500 x 5s, one channel
read 0 -1 0 0.3412 0.3412 [29] open 500 x 5s, one channel
close 0 -1 0 0.0001 0.0001 [29] open 500 x 5s, one channel
________________________________________________________________
Read performance (median of N trials):
#seek #read KiB CPU(s) Wall(s) [N]
0 -1 0 0.8228 0.8232 [13] open 1 x 214981s, all channels
0 -1 0 0.0274 0.0274 [325] open 5 x 500s, all channels
0 -1 0 0.2235 0.2236 [44] open 50 x 50s, all channels
0 -1 0 2.1601 2.1603 [5] open 500 x 5s, all channels
0 -1 0 0.1020 0.1020 [86] open 1 x 214981s, one channel
0 -1 0 0.0054 0.0055 [1175] open 5 x 500s, one channel
0 -1 0 0.0451 0.0451 [206] open 50 x 50s, one channel
0 -1 0 0.4393 0.4394 [23] open 500 x 5s, one channel
________________________________________________________________
briangow
commented
2 weeks ago Thanks @wa6gz , this looks good to me!