search

iitc-project / ingress-intel-total-conversion

ingress.com/intel total conversion user script with some new features. Should allow easier extension of the intel map.
http://iitc.jonatkins.com/
ISC License
991 stars 552 forks source link

Data refresh handling - rewrite #493

Closed jonatkins closed 11 years ago

jonatkins commented 11 years ago

At the moment IITC uses a single timer for handling both map data refreshes and chat refreshes. This is not ideal. Chat data is generally quick to retrieve, and a relatively quick refresh cycle is preferred. However, map data can take much longer to retrieve - and on slower machines, quite a while to process. Additionally, the timer for the next refresh cycle is started when the requests are first sent - if things are particularly slow, a new refresh can start before the current cycle is complete.

Additionally, map data can often return an error==TIMEOUT for individual map tiles. The stock site, as far as I can tell, will immediately retry these tiles - IITC does no retries at all.

So - time for a rewrite of code in this area. First, move chat refresh onto it's own cycle (30 seconds?) completely independent of the map data refreshes

Complete rewrite of the map data fetching, cache handling, etc.

  1. Create a list of all desired data tiles
  2. Start requesting these tiles. Limit the number of tiles retrieved in a single request to a reasonable number, and limit the maximum number of active requests to 4. (as XMLHtppRequest usually allows up to 6 parallel requests, this prevents map data requests from blocking chat refresh)
  3. Add any error==TIMEOUT tiles back into the queue, for retrying. (maybe limit the maximum number of retries for individual tiles?)
  4. Start the refresh timer for map data only after the queue is completely processed - this prevents starting a new refresh before the previous one is complete.
  5. Don't cancel pending data tile requests when the map view changes. At least let them complete and be stored in the cache - and if they're still useful (e.g. map was only panned a small amount) process them, rather than a request cancel and then re-request of the same map tiles.

This is a relatively large job. Given the number of protocol changes recently I've created a new branch for this work.

Chat refresh status will be separate from map data status - so will probably be best moved away from the general 'failed request' counter. Perhaps replace the time in the chat input area with a chat refresh status?

jonatkins commented 11 years ago

this is pretty much done now, and merged into the master branch

some cleanups are still required, but isn't that always the case with code...

prok42 commented 11 years ago

Is it possible to expose the timers/thresholds used for automatic refreshes? During an attack I'd like to be able to set (or activate a plugin) which triggers reloads every, say, 30s instead of 120s (REFRESH_CLOSE). Unfortunately, I could not find any way to overwrite the 120s by means of a plugin (i.e. without editing the iitc master file).

jonatkins commented 11 years ago

@prok42 the values can be found in window.mapDataRequest.REFRESH_CLOSE, etc at run-time

I'd like to improve things further in this area though - would be nice to adjust refresh automatically as needed. One thought is to monitor the COMM data and when activity for portals within the map is seen the refresh period drops. When COMM activity stops, it increases again.

This doesn't cover all cases of activity (e.g. upgrades), but perhaps would be good enough for most cases.

prok42 commented 11 years ago

@jonatkins ah thanks a lot - and kudos for the request rewrite! It is already very smooth now with less timeouts