evazion / translate-pixiv-tags

MIT License
35 stars 6 forks source link

Handle network errors #36

Closed BrokenEagle closed 5 years ago

BrokenEagle commented 5 years ago

I went back to that Pixiv tags page the other day, and I was still noting the occasional error. So I added in a retry mechanism, an early bail for too many network errors, and a backoff mechanism for slower network conditions. The early bail will help detect and stop processing under bad network conditions or Downbooru.

During testing of this pull at the 40 max pending requests, there were on average a few to no network errors, although occasionally I did see more than 10. Also, the rate of network errors always slowed down as the script backed off the current max pending. Therefore I felt like 25 max network errors was probably the right number to use. Especially since during regular operations a user shouldn't be stressing out a system so much with 500 network requests all at once, and the worst that will happen is that the page will load partially. The user could potentially be notified of this, but there currently isn't such a mechanism plus that was outside the scope for this change

Just to add in some additional observations, but the tags page was also tested at 50 max pending requests, and the error rate was much higher, usually at least 20 or more. So if the max pending requests is ever raised, the max network requests will also need to be raised.

BrokenEagle commented 5 years ago

I didn't realize that the spread operator can be used on the parameters list. That's good to know. Also, I put the level last because that's the way I prefer it, but I can also see the value of having it first.

7nik commented 5 years ago

BTW, https://kangax.github.io/compat-table/es6/ here you can find features added to ES6, links to docs, their test cases and codes, and itemized support by browsers. Newer features are in the "2016+" tab.