ratal / mdfreader

Read Measurement Data Format (MDF) versions 3.x and 4.x file formats in python
Other
169 stars 74 forks source link

Resampling of integer channels #101

Closed cristi-neagu closed 6 years ago

cristi-neagu commented 6 years ago

Resampling integer channels should not be done with linear interpolation (np.interp). Consider using scipy.interpolate.interp1d(kind='zero') instead. This could be hardcoded, or, if more flexibility is desired, give the user the option to specify zero order interpolation for integer signals.

The downside is that it would introduce an extra dependency on scipy in the mdf module.

danielhrisca commented 6 years ago

Hello Cristi,

this is similar to issue #90

cristi-neagu commented 6 years ago

It is, yes. I didn't notice it because it was a secondary issue, different from the original title. In any case, i feel it merits its own tracker.

ratal commented 6 years ago

Thanks for the tip Cristi yes indeed already issued, thanks Daniel, definitely needed :) scipy is already an optional dependency for export to netcdf or matlab, so why not adding this interpolation. however I am wondering how it performs against searchsorted used by Daniel ?

cristi-neagu commented 6 years ago

On my system, scipy does perform 2-3 times worse than searchsorted, over about 765000 data points. The worst performance i got was about 60ms, which is still reasonably fast. Both methods work just as well. Searchsorted has the advantage of speed, while scipy is more readable, in my opinion.

danielhrisca commented 6 years ago

@cristi-neagu nice find. It does exactly what my implementation does; I'm thinking of adding script to the dependency list and use interp1d

danielhrisca commented 6 years ago

After a few tests it seems that interp1d is 3x slower and raises exception for new timestamps outside the initial timestamp interval.

cristi-neagu commented 6 years ago

interp1d can be asked to either replace any values outside of bounds with given values, or extrapolate data, by using fill_value.

danielhrisca commented 6 years ago

You're right. One has to decide if the speed penalty is acceptable or not.

ratal commented 6 years ago

Hi, I put Daniel's snippet (thanks) for interpolation in last mdfreader commit. Can you check if that suits you ?

cristi-neagu commented 6 years ago

That code works perfectly. Thanks.

cristi-neagu commented 6 years ago

Fixed in d1d9a45