Speed up optimizer

[mdozmorov]

As we have many grsnp_db folders with filtered and unfiltered data, optimizer takes forever.

[mdozmorov]

Crash while running big database. Can't find cause in logs

2014-07-04 19:23:03,617 INFO Running overlapStatistics for all GFs in grsnp_db/hg19 Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/home/genomerunner/genome_runner/grsnp/optimizer.py", line 150, in create_bkg_gf_overlap_db(gf_dir=gfs_dir,background_dir=background_dir,data_dir=args['data_dir']) File "/home/genomerunner/genome_runner/grsnp/optimizer.py", line 64, in create_bkg_gf_overlap_db write_results(results,db_path) File "/home/genomerunner/genome_runner/grsnp/optimizer.py", line 84, in write_results stat_line = [x["queryfile"]+":"+str(x["intersectregions"])+":"+str(x["queryregions"]) for x in stats] TypeError: 'NoneType' object is not iterable

[mdozmorov]

The system is broken.

• Upon restart 'bkg_overlaps.gr' files got zeroed and restarted from 0. It totally screws everything!
• Redis remembers tasks and tries to restart them all, even after all workers on all machines were purged and restarted. Even hard restart of all machines does not help.
• Some 'bkg_overlaps.gr' files got '.tmp' extension appended. But optimizer no longer cares about it and restarts the tasks. We need to make '.tmp' extension either gone or properly handled

[mdozmorov]

We should carefully think about consequences

http://stackoverflow.com/questions/17184244/how-to-remove-task-from-celery-with-redis-broker

1) Delete queue directly from broker. In your case it's Redis. There are two commands that could help you: llen (to find right queue) and del (to delete it).

2) Start celery worker with --purge or --discard options. Here is help: --purge, --discard Purges all waiting tasks before the daemon is started. WARNING: This is unrecoverable, and the tasks will be deleted from the messaging server.