search

lanrat / czds

simple golang API and tools to interact with czds.icann.org
https://pkg.go.dev/github.com/lanrat/czds
GNU General Public License v3.0
72 stars 14 forks source link

Added Retry Logic #3

Closed kingforaday closed 5 years ago

kingforaday commented 5 years ago

CZDS had previously shown some instability in their infrastructure, which seems to have stabilized, but should a connection timeout occur during zone download or other disconnect, this will re-add the zone location back to the processing queue for further retries.

kingforaday commented 5 years ago

Take a look at the latest commit cb6a11a. To not interrupt the addLinks channel management there is a separate one for any retries required.

lanrat commented 5 years ago

Unfortunately this does not fix the deadlock either.

I tested your branch returning an error randomly on calls to zoneDownload(zi) and after the first few zones the program stopped in a deadlock.

The problem is that once the retryChan is full we are back where we started where the worker threads are blocked adding new items to the channel and cant take items out of either retryChan or inputChan.

lanrat commented 5 years ago

The best solution I can think of is to take the code from commit a7ea2867fb9cc34ed83c8c84a1688e92d602bf94 and change the line: 

inputChan <- zi // requeue

to:

go func() {
    inputChan <- zi // requeue
}()

This adds the ZoneInfo back to inputChan in its own goroutine and does not block the worker. Its not ideal, but it will never deadlock.

lanrat commented 5 years ago

This has been added in #4