search

djanloo / cmepda

An LSTM + Encoder network for UHECR AirShowers
1 stars 2 forks source link

DataFeeders are utterly too slow #10

Closed djanloo closed 2 years ago

djanloo commented 2 years ago

TODO: lineprofile DataFeederKeras because I suspect the most time-expensive line is the np.load() one.

luciapapalini commented 2 years ago

I'm a stupid person and committed to the wrong issue...... That all was relative to issue #11

djanloo commented 2 years ago

As I suspected loading a file costs approx. 3 ms.

The total 'loading' time over an epoch should be roughly 3.5 mins . Clearly this time is reduced by the mutliproc. features inherited by keras.Sequence, since a typical epoch takes 1.5 mins.

WhatsApp Image 2022-07-11 at 13 25 32