criteo / biggraphite

Simple Scalable Time Series Database
Apache License 2.0
129 stars 36 forks source link

Fix daemon launch by unregistering Twisted reactor #553

Closed mycroft closed 4 years ago

mycroft commented 4 years ago

The recent cassandra_driver update triggered a reactor installation before starting to parse biggraphite configuration. This is triggering a failure because biggraphite can't initialize its own reactor, making the bg daemon fails.

Also, bumping to 0.14.20.

coveralls commented 4 years ago

Coverage Status

Coverage increased (+0.03%) to 81.958% when pulling c70f5ec74a43ae3d53467ef3eb9531b081daeec1 on mycroft:master into bc100512116ba515c7e271436be59b78ef6fb970 on criteo:master.

mycroft commented 4 years ago

Won't this break twisted? https://twistedmatrix.com/documents/8.2.0/api/twisted.internet.reactor.html

Harold, obviously, my goal is to fix it, not to break it. Upgrading cassandra-driver to latest current version won't make bg-carbon-aggregator-cache and its peer to start anymore:

# /opt/graphite/pypy/bin/python /opt/graphite/pypy/bin/bg-carbon-aggregator-cache --debug --nodaemon start
Traceback (most recent call last):
  File "/opt/graphite/pypy/bin/bg-carbon-aggregator-cache", line 12, in <module>
    sys.exit(main())
  File "/opt/graphite/pypy/site-packages/biggraphite/cli/bg_carbon_aggregator_cache.py", line 48, in main
    carbon_util.run_twistd_plugin("carbon-aggregator-cache")
  File "/opt/graphite/pypy/site-packages/carbon/util.py", line 138, in run_twistd_plugin
    config.parseOptions(twistd_options)
  File "/opt/graphite/pypy/site-packages/twisted/application/app.py", line 637, in parseOptions
    usage.Options.parseOptions(self, options)
  File "/opt/graphite/pypy/site-packages/twisted/python/usage.py", line 255, in parseOptions
    self._dispatch[optMangled](optMangled, arg)
  File "/opt/graphite/pypy/site-packages/twisted/python/usage.py", line 411, in <lambda>
    fn = lambda name, value, m=method: m(value)
  File "/opt/graphite/pypy/site-packages/twisted/application/app.py", line 549, in opt_reactor
    raise usage.UsageError(msg)
UsageError: The specified reactor cannot be used, failed with error: stop.
See the list of available reactors with --help-reactors

The problem is that, for some reason, cassandra_driver will now load twisted reactor modules (cassandra_driver is load by biggraphite even before we start it), and loading twisted reactor will install it, even before we load the configuration. See this trace with a custom Exception I had to use to retrieve the stack trace of the initial installation:

# /opt/graphite/pypy/bin/python /opt/graphite/pypy/bin/bg-carbon-aggregator-cache --debug --nodaemon start
Traceback (most recent call last):
  File "/opt/graphite/pypy/bin/bg-carbon-aggregator-cache", line 12, in <module>
    sys.exit(main())
  File "/opt/graphite/pypy/site-packages/biggraphite/cli/bg_carbon_aggregator_cache.py", line 34, in main
    from biggraphite.plugins import carbon as unused_carbon  # noqa
  File "/opt/graphite/pypy/site-packages/biggraphite/plugins/carbon.py", line 38, in <module>
    from biggraphite import settings as bg_settings
  File "/opt/graphite/pypy/site-packages/biggraphite/settings.py", line 19, in <module>
    from biggraphite import accessor_factory as bg_accessor_factory
  File "/opt/graphite/pypy/site-packages/biggraphite/accessor_factory.py", line 17, in <module>
    from biggraphite.drivers import cassandra as bg_cassandra
  File "/opt/graphite/pypy/site-packages/biggraphite/drivers/cassandra.py", line 34, in <module>
    from cassandra import cluster as c_cluster
  File "/opt/graphite/pypy/site-packages/cassandra/cluster.py", line 94, in <module>
    from cassandra.io.twistedreactor import TwistedConnection
  File "/opt/graphite/pypy/site-packages/cassandra/io/twistedreactor.py", line 25, in <module>
    from twisted.internet import reactor, protocol
  File "/opt/graphite/pypy/site-packages/twisted/internet/reactor.py", line 39, in <module>
    default.install()
  File "/opt/graphite/pypy/site-packages/twisted/internet/epollreactor.py", line 245, in install
    p = EPollReactor()
  File "/opt/graphite/pypy/site-packages/twisted/internet/epollreactor.py", line 66, in __init__
    raise Exception('stop')

The fix I'm proposing is to load biggraphite, which will load cassandra then firstly install the reactor module, and if before we start ours it exists in the loaded modules, to unregister it. It seems like a common practice, but if you want me to double check, I'll take a deeper look. However, I've tested this successfully on our testing environment (it connects & runs well against network & cassandra).