search

scrapinghub / exporters

Exporters is an extensible export pipeline library that supports filter, transform and several sources and destinations
BSD 3-Clause "New" or "Revised" License
40 stars 10 forks source link

Remove retry from non-idempotent function #304

Closed eliasdorneles closed 8 years ago

eliasdorneles commented 8 years ago

Hey fellows,

So, this is just a first step to address #303, just to avoid having incorrect results silently.

We should be mindful where to put retries in this code, because retries are really meant for code that will (probably, but as far as the code knows, always) return the same result if it is retried.

I couldn't find a good place to put more retries in this code, at least not as it is now, because it's depending on state for a good bunch of things. So, I don't know what could be a next step to address this (leave it without retries, refactor some place?) -- would love to hear some ideas.

bbotella commented 8 years ago

I'm merging this, and we can continue discussion at the issue https://github.com/scrapinghub/exporters/issues/303