URLFetch fails irregularly causing too many support requests

akvo / akvo-flow

A data collection and monitoring tool that works anywhere.

http://akvo.org/products/akvoflow/

GNU Affero General Public License v3.0

65 stars 31 forks source link

URLFetch fails irregularly causing too many support requests #601

Closed mtwestra closed 4 years ago

mtwestra commented 10 years ago

We are getting a lot of email notifications about zip files not being processed due to a URLFetch problem, such as: Device File Processing Error: wfp28669099259418.zip java.io.IOException: Could not fetch URL: http://akvoflow-12.s3.amazonaws.com/devicezip/wfp28669099259418.zip

We know that URLFetch fails in some occasions. This might be worsened by the fact that our s3 storage and the GAE serves are in different continents.

We already have a solution in place that reschedules the task after 5 minutes. However, this still sends out email notifications for each failure.

Proposed solution:

In the device zip processing code, reschedule the task in 5 minutes upon failure, with a maximum retry of 3. Don't send a notification email yet
For each retry, decrease the retry with one.
If during the last retry the task still fails, send out an error message.

iperdomo commented 10 years ago

@mtwestra we already have code to retry with a delay of 5min. See the the following lines:

https://github.com/akvo/akvo-flow/blob/v1.6.17.1/GAE/src/org/waterforpeople/mapping/app/web/TaskServlet.java#L129-L144

cc/ @muloem

mtwestra commented 10 years ago

@iperdomo I suppose seeing the error message means then that all the retries have failed? Or is that not the case?

On 12 Jun 2014, at 10:23, Iván Perdomo notifications@github.com wrote:

@mtwestra we already have code to retry with a delay of 5min. See the the following lines:

https://github.com/akvo/akvo-flow/blob/v1.6.17.1/GAE/src/org/waterforpeople/mapping/app/web/TaskServlet.java#L129-L144 cc/ @muloem

— Reply to this email directly or view it on GitHub.

muloem commented 10 years ago

The files are processed after several retries. Retry mechanism is already in place as implemented for issue #541. Closing this issue.

mtwestra commented 10 years ago

@muloem,

We also want to avoid that we get all these error messages as tender requests. Is that the case now?

On 12 Jun 2014, at 12:55, Emmanuel Mulo notifications@github.com wrote:

Closed #601.

— Reply to this email directly or view it on GitHub.

muloem commented 10 years ago

Agreed. For now the setup is such that every failure is sent to tender as a support request.

mtwestra commented 10 years ago

So we need to change that, because we will get very annoyed :-)

mtwestra commented 10 years ago

Due to the above discussion, the issue text was reformulated. Therefore, some comments above might have lost their context.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.