search

OpenDrift / opendrift

Open source framework for ocean trajectory modelling
https://opendrift.github.io
GNU General Public License v2.0
248 stars 121 forks source link

Speeding up allocation of self.history in import_file #993

Closed manuelaghito closed 2 years ago

manuelaghito commented 2 years ago

for some reason numpy.ma.array allocation is extremely slow when mask=True is passed

knutfrode commented 2 years ago

Ah, yes! In fact, we should aim to remove masked arrays completely from OpenDrift.

manuelaghito commented 2 years ago

now it should be ok, hopefully :-) was very eager when I found the problem :-) image

manuelaghito commented 2 years ago

by the way if we don't need all values to be masked then we can remove the line self.history[:] = np.ma.masked then it is even faster. But at least like it is proposed now the initially allocated self.history should be exactly the same as before, as far as I can see, all zeros, all masked

manuelaghito commented 2 years ago

just noticed that I didn't pull the latest master with the recent commit https://github.com/OpenDrift/opendrift/commit/1d1df93aad71d111f313af34f763001e3bf7ef86 before I worked on this

manuelaghito commented 2 years ago

this should be ready to merge. I confirm that the bottlneck was using mask=[True] when allocating masked array. Tried slicing also, that didn't give any further improvement. I think there were problems with PPI on wednesday and that was everything was still slow