bibtasklets: log last sync info in prodsync

[tsgit]

this caches the last sync time for each record? what's the cache size implication of that?

I'm asking because we have previously hit the maxmem ceiling on legacy redis, when there are lots of sessions cached. This is using CFG_REDIS_HOST_LABS

....@inspire-prod-cache1.cern.ch:6379...

what's the maxmem setting on that? I'm seeing

maxmemory 5,614,075,904 maxmemory_policy volatile-lru used_memory_peak 1160667808 used_memory_human 219.47M

so I guess that should be plenty

however I note that volatile-lru is not optimal for reliability, since it means keys will get discarded before their TTL expires when OOM conditions occur

[tsgit]

minor niggle: instead of a naked now a keyword argument timestamp= or sync_time= would be more maintainable

anyhow, one possible issue may be that slave is behind master and bibrec table timestamp and tables for xme info are out of sync on slave. you could get old xme depending on details of order of update of these tables on modification and the transfer to mysql slave. this may be particularly true around backup time, when the slave sync is stopped for extended period of time. I don't think this has been properly analyzed

[michamos]

I did it like it's done for AFS basically. I remember we had this master vs slave consistency issue in the past where selecting records was done on the master, but formatting the records was done on the slave. I fixed that to do everything on the slave. Do you think there's a loophole somewhere?

[michamos]

In [1]: from datetime import datetime                                                                                                                                                                                

In [2]: len(str(datetime.now()))                                                                                                                                                                                     
Out[2]: 26

Adding also the recid, let's assume we need to store 40 bytes per record, plus 10 for redis overhead, so if we have 2M records, that's about ~100MB. Nothing to worry about.

[michamos]

We can close this, we figured out where the discrepancy is coming from.