threaded_scanner.py does not update memcache

[afunix]

I've installed 3 Perspectives servers, 2 of them are configured to cache data in memcached. These 2 servers often return stale data on HTTP requests. I didn't find anything related to memcache in threaded_scanner.py, so I believe that it does not update memcache which causes server to return stale data.

[daveschaefer]

Hey afunix,

If you do a direct query to your database after the scanner runs, does the database show updated data?

The scanner actually updates data directly inside the database. If you have caching turned on the server won't fetch new database entries until the cache entries have expired. The default behaviour is to cache entries for 12 hours - you can control this with the '--cache-expiry' switch.

Have you tried flushing your cache after running a scan and seeing if the new entries show up? You can also try setting your --cache-expiry to something short, like 5 seconds, and check to make sure the server returns the correct, updated data after a scan finishes.

How often are you running your scans? More frequently than every 12 hours?

[afunix]

Hi Dave.

I've configured threaded_scanner to be executed each hour. How do I flush cache? Do you mean whole memcache? Is there a some kind of script to delete only notary-related records? Why doesn't threaded_scanner flush the cache on its own?

I can change cache-expiry to be equal to thread_scanner execution interval, but I will still have stale records in the cache for 'cache-expiry' period in a worst case.

2014-06-07 8:54 GMT+04:00 Dave notifications@github.com:

Hey afunix,

If you do a direct query to your database after the scanner runs, does the database show updated data?

The scanner actually updates data directly inside the database. If you have caching turned on the server won't fetch new database entries until the cache entries have expired. The default behaviour is to cache entries for 12 hours - you can control this with the '--cache-expiry' switch https://github.com/danwent/Perspectives-Server/blob/master/notary_http.py#L90 .

Have you tried flushing your cache after running a scan and seeing if the new entries show up? You can also try setting your --cache-expiry to something short, like 5 seconds, and check to make sure the server returns the correct, updated data after a scan finishes.

How often are you running your scans? More frequently than every 12 hours?

— Reply to this email directly or view it on GitHub https://github.com/danwent/Perspectives-Server/issues/43#issuecomment-45401228 .

[daveschaefer]

Hey, yes, I meant flushing the memcached instance to get rid of notary entries. If you're tracking other data in the cache besides notary results I don't know of an easy way to remove only notary-related items. Perhaps you could do some research?

Why doesn't threaded_scanner flush the cache on its own?

The scanner script is currently only set to update the database. Previous versions of Perspectives did not have any caching layers, so it's possible the scanner was not designed with that in mind. However, not all notaries have the scanner, the cache, and the server running on the same machine, so if you wanted to have the scanner update cache entries we would have to change the system to be able to do that.

I'm interested in why you have your scan running every hour. Do you expect some certificates to change? Or you just want very up-to-date information?

[afunix]

I run one if the notaries on OpenShift where I can easily configure cron to execute something on hourly or daily basis. So I've chosen hourly instead of daily and aligned all notaries to that period. The another (not-such-a-good) reason is that I have an hourly history of certificates at some networks I control, so in theory I can detect targeted MitM which is executed for a small period of time.

So I believe this record can be treated as a whishlist to update memcache along with database from threaded_scanner.

2014-06-09 7:52 GMT+04:00 Dave notifications@github.com:

Hey, yes, I meant flushing the memcached instance to get rid of notary entries. If you're tracking other data in the cache besides notary results I don't know of an easy way to remove only notary-related items. Perhaps you could do some research?

Why doesn't threaded_scanner flush the cache on its own?

The scanner script is currently only set to update the database. Previous versions of Perspectives did not have any caching layers, so it's possible the scanner was not designed with that in mind. However, not all notaries have the scanner, the cache, and the server running on the same machine, so if you wanted to have the scanner update cache entries we would have to change the system to be able to do that.

I'm interested in why you have your scan running every hour. Do you expect some certificates to change? Or you just want very up-to-date information?

— Reply to this email directly or view it on GitHub https://github.com/danwent/Perspectives-Server/issues/43#issuecomment-45459259 .

[netsafe]

fixed. The scanner can accept memcached or redis connection data and destroy the value by key. No, it's not possible to destroy anything in pycache, because it's a program's "inner sanctum". Actually installing memcached in anonymous mode on localhost is too easy to ignore it. Redis support was written by manual, haven't tested it, because I don't use it. Memcached via both interfaces works fine : tested it myself. Threaded scanner not just writes something into database, but also destroys a value in a cache, so it will be properly re-requested out of DB backend.