error in getChannelData when noDataLoading=True

[danielhrisca]

Benchmark environment

• 3.6.2 (default, Jul 20 2017, 03:52:27) [GCC 7.1.1 20170630]
• Linux-4.9.40-1-MANJARO-x86_64-with-glibc2.3.4

The files used are the test files.

With mdf version 3 I get the error:

Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/daniel/.config/spyder-py3/temp.py", line 198, in get_all_reader3_nodata
    for s in x:
RuntimeError: dictionary changed size during iteration

With mdf version 4 I get the error:

Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/daniel/.config/spyder-py3/temp.py", line 151, in get_all_reader4_nodata
    for s in x:
RuntimeError: dictionary changed size during iteration

[ratal]

Hi Daniel, Situation should have improved for version 4, still working on version 3

[ratal]

Hi Daniel, Both versions should not throw any error.

[danielhrisca]

Hello Aymeric,

no more errors with the latest commit, but you probably want to investigation the performance.

Benchmark environment

• 3.6.2rc1 (heads/3.6:268e1fb, Jun 17 2017, 19:01:44) [MSC v.1900 64 bit (AMD64)]
• Windows-10-10.0.15063-SP0
• Intel64 Family 6 Model 69 Stepping 1, GenuineIntel
• 16GB installed RAM

Notations used in the results

• compression = mdfreader mdf object created with compression=blosc
• compression bcolz 6 = mdfreader mdf object created with compression=6
• noDataLoading = mdfreader mdf object read with noDataLoading=True

Files used for benchmark

• 183 groups
• 36424 channels

Get all channels (36424 calls)	Time [ms]	RAM [MB]
mdfreader 0.2.6 mdfv3	95	542
mdfreader 0.2.6 nodata mdfv3	114142	413
mdfreader 0.2.6 compression mdfv3	757	264
mdfreader 0.2.6 compression bcolz 6 mdfv3	357	1037
mdfreader 0.2.6 mdfv4	98	1315
mdfreader 0.2.6 nodata mdfv4	168593	1185
mdfreader 0.2.6 compression mdfv4	750	1029
mdfreader 0.2.6 compression bcolz 6 mdfv4	411	1750