search

elliottd / GroundedTranslation

Multilingual image description
https://staff.fnwi.uva.nl/d.elliott/GroundedTranslation/
BSD 3-Clause "New" or "Revised" License
46 stars 25 forks source link

Faster processing with buffered dataset yielding ? #9

Closed elliottd closed 9 years ago

elliottd commented 9 years ago

Our code has an unfortunate pause while we yield the next big_batch, especially when we have to resize the array.

@scfrank do you think we can adapt this approach to buffering the next big_batch? Exact code is here.

scfrank commented 9 years ago

Could you quantify the length of the unfortunate pause? (Do you have timings that you can s hare?) The resizing is an issue that might be seperately fixable and I'll look into it. I'm not entirely convinced adding more dependencies to this model is what we should be focusing on.

On 14 September 2015 at 12:47, Desmond Elliott notifications@github.com wrote:

Our code has an unfortunate pause while we yield the next big_batch, especially when we have to resize the array.

@scfrank https://github.com/scfrank do you think we can adapt this approach http://jimgoo.github.io/buffered-gens/ to buffering the next big_batch? Exact code is here https://github.com/benanne/kaggle-ndsb/blob/11a66cdbddee16c69514b9530a727df0ac6e136f/buffering.py .

— Reply to this email directly or view it on GitHub https://github.com/elliottd/GroundedTranslation/issues/9.

elliottd commented 9 years ago

Fixed by 366b6874