search

SeattleTestbed / softwareupdater

Software updater daemon
MIT License
1 stars 6 forks source link

lots of software updater requests to the webserver in a small time period. #3

Closed choksi81 closed 10 years ago

choksi81 commented 10 years ago

We are seeing lots of requests by the software updater in a short time period. For example, the requests below are all within a 2 second period. This is almost certainly an indication of other, yet to be discovered problems.

128.208.1.186 2009:11:10:01 128.208.4.30 2009:11:10:01 128.208.1.214 2009:11:10:01 128.208.1.225 2009:11:10:01 128.208.4.178 2009:11:10:01 128.208.1.246 2009:11:10:01 128.208.1.235 2009:11:10:01 128.208.1.158 2009:11:10:01 128.208.1.150 2009:11:10:01 128.208.1.185 2009:11:10:01 128.208.1.169 2009:11:10:01 128.208.1.217 2009:11:10:01 128.208.1.239 2009:11:10:01 128.208.1.167 2009:11:10:01 128.208.1.115 2009:11:10:01 128.208.1.183 2009:11:10:01 128.208.1.152 2009:11:10:01 128.208.1.240 2009:11:10:01 128.208.1.131 2009:11:10:01 128.208.1.108 2009:11:10:01 128.208.1.166 2009:11:10:01 128.208.1.153 2009:11:10:01 128.208.1.117 2009:11:10:01 128.208.1.232 2009:11:10:01 128.208.1.161 2009:11:10:01 128.208.1.121 2009:11:10:01 128.208.1.247 2009:11:10:01 128.208.1.168 2009:11:10:01 128.208.1.156 2009:11:10:01 128.208.1.222 2009:11:10:01 128.208.1.179 2009:11:10:01 128.208.1.199 2009:11:10:01 128.208.1.234 2009:11:10:02 128.208.6.165 2009:11:10:02 128.208.1.249 2009:11:10:02 128.208.1.157 2009:11:10:02 128.208.1.130 2009:11:10:02 128.208.1.241 2009:11:10:02 128.208.1.238 2009:11:10:02 128.208.1.159 2009:11:10:02 128.208.1.135 2009:11:10:02 128.208.1.231 2009:11:10:02 128.208.1.224 2009:11:10:02 128.208.1.221 2009:11:10:02 128.208.1.132 2009:11:10:02 128.208.1.163 2009:11:10:02 128.208.1.114 2009:11:10:02

choksi81 commented 10 years ago

Author: couvb These nodes seem to all be from the undergrad labs. I suspect something strange about the lab setup is causing them to generate the same random number sequences (possibly due to synchronized times).

To further investigate this, I will deploy a simple script to all the lab computers that use a random wait scheme similar to the one in the software updater, except it will send debugging information to a server that I control. This will hopefully provide more insight into what exactly is going on, and how we might deal with it.

choksi81 commented 10 years ago

Author: ivan You can read the apache's access logs on seattle.cs here: /var/log/apache2/access.log

To grab just those that hit the updatesite (via the software updater) do: $ cat /var/log/apache2/access.log | grep updatesite | less

You'll find a couple of timestamps at which the nodes are synchronized in hitting the webserver. For example, from the current log file here is a list of top 50 timestamps (just year, and timestamp) along with the total number of requests that occurred on the exact timestamp in the log. Note that the timestamp "07:50:01" has been the most popular, but not on the same day as in the actual log you won't find any day where more than about 15 hosts make requests at exactly this time.

2009:07:50:01 102 2009:08:50:01 98 2009:07:10:01 98 2009:07:20:01 97 2009:09:00:01 96 2009:08:10:01 95 2009:17:00:02 94 2009:08:20:01 94 2009:06:50:01 94 2009:08:40:01 93 2009:08:00:01 93 2009:07:40:01 93 2009:07:30:01 93 2009:10:10:01 92 2009:06:40:01 92 2009:09:20:01 91 2009:09:30:01 90 2009:10:00:01 89 2009:13:10:02 88 2009:09:50:01 88 2009:08:30:01 88 2009:14:00:02 86 2009:13:00:02 86 2009:07:00:01 85 2009:16:50:02 83 2009:13:50:01 83 2009:12:10:01 83 2009:17:40:01 82 2009:14:20:02 82 2009:12:50:02 82 2009:10:30:01 82 2009:10:20:01 82 2009:09:10:01 82 2009:12:20:01 81 2009:11:50:01 81 2009:11:40:01 81 2009:17:50:01 80 2009:14:30:02 80 2009:11:20:01 80 2009:11:00:01 80 2009:10:50:01 80 2009:17:10:02 77 2009:16:20:01 77 2009:12:40:02 77 2009:12:30:02 77 2009:15:20:01 76 2009:13:40:02 76 2009:11:10:01 76 2009:16:40:02 75 2009:10:40:01 75

choksi81 commented 10 years ago

Author: ivan This seems to have resolved itself. We probably also need some way of monitoring this over a long time period.