search

spotify / cassandra-reaper

Software to run automated repairs of cassandra
235 stars 60 forks source link

Server not starting when using database as storage #132

Closed mlowicki closed 8 years ago

mlowicki commented 8 years ago

Normally when storageType is set to memory it works fine:

INFO   [2015-12-17 10:07:57,164] [main] i.d.a.AssetsBundle - Registering AssetBundle with name: assets for path /webui/*
INFO   [2015-12-17 10:07:57,197] [main] c.s.r.ReaperApplication - initializing runner thread pool with 15 threads
INFO   [2015-12-17 10:07:57,202] [main] c.s.r.ReaperApplication - initializing storage of type: memory
INFO   [2015-12-17 10:07:57,204] [main] c.s.r.ReaperApplication - no JMX connection factory given in context, creating default
INFO   [2015-12-17 10:07:57,208] [main] c.s.r.ReaperApplication - creating and registering health checks
INFO   [2015-12-17 10:07:57,208] [main] c.s.r.ReaperApplication - creating resources and registering endpoints
INFO   [2015-12-17 10:07:58,214] [main] c.s.r.s.SchedulingManager - Starting new SchedulingManager instance
INFO   [2015-12-17 10:07:58,215] [main] c.s.r.ReaperApplication - resuming pending repair runs
INFO   [2015-12-17 10:07:58,228] [main] i.d.s.ServerFactory - Starting cassandra-reaper
_________                                          .___               __________
\_   ___ \_____    ______ ___________    ____    __| _/___________    \______   \ ____ _____  ______   ___________
/    \  \/\__  \  /  ___//  ___/\__  \  /    \  / __ |\_  __ \__  \    |       _// __ \\__  \ \____ \_/ __ \_  __ \
\     \____/ __ \_\___ \ \___ \  / __ \|   |  \/ /_/ | |  | \// __ \_  |    |   \  ___/ / __ \|  |_> >  ___/|  | \/
 \______  (____  /____  >____  >(____  /___|  /\____ | |__|  (____  /  |____|_  /\___  >____  /   __/ \___  >__|
        \/     \/     \/     \/      \/     \/      \/            \/          \/     \/     \/|__|        \/

INFO   [2015-12-17 10:07:58,302] [main] o.e.j.s.SetUIDListener - Opened application@5a9d6f02{HTTP/1.1}{0.0.0.0:8666}
INFO   [2015-12-17 10:07:58,302] [main] o.e.j.s.SetUIDListener - Opened admin@362045c0{HTTP/1.1}{0.0.0.0:8667}
INFO   [2015-12-17 10:07:58,305] [main] o.e.j.s.Server - jetty-9.0.z-SNAPSHOT
INFO   [2015-12-17 10:07:58,394] [main] c.s.j.s.i.a.WebApplicationImpl - Initiating Jersey application, version 'Jersey: 1.18.1 02/19/2014 03:28 AM'
INFO   [2015-12-17 10:07:58,470] [main] i.d.j.DropwizardResourceConfig - The following paths were found for the configured resources:

    GET     /ping (com.spotify.reaper.resources.PingResource)
    DELETE  /cluster/{cluster_name} (com.spotify.reaper.resources.ClusterResource)
    GET     /cluster (com.spotify.reaper.resources.ClusterResource)
    GET     /cluster/{cluster_name} (com.spotify.reaper.resources.ClusterResource)
    POST    /cluster (com.spotify.reaper.resources.ClusterResource)
    DELETE  /repair_run/{id} (com.spotify.reaper.resources.RepairRunResource)
    GET     /repair_run (com.spotify.reaper.resources.RepairRunResource)
    GET     /repair_run/cluster/{cluster_name} (com.spotify.reaper.resources.RepairRunResource)
    GET     /repair_run/{id} (com.spotify.reaper.resources.RepairRunResource)
    POST    /repair_run (com.spotify.reaper.resources.RepairRunResource)
    PUT     /repair_run/{id} (com.spotify.reaper.resources.RepairRunResource)
    DELETE  /repair_schedule/{id} (com.spotify.reaper.resources.RepairScheduleResource)
    GET     /repair_schedule (com.spotify.reaper.resources.RepairScheduleResource)
    GET     /repair_schedule/cluster/{cluster_name} (com.spotify.reaper.resources.RepairScheduleResource)
    GET     /repair_schedule/{id} (com.spotify.reaper.resources.RepairScheduleResource)
    POST    /repair_schedule (com.spotify.reaper.resources.RepairScheduleResource)
    PUT     /repair_schedule/{id} (com.spotify.reaper.resources.RepairScheduleResource)

But if using configuration as follows:

segmentCount: 200
repairParallelism: DATACENTER_AWARE
repairIntensity: 0.9
#scheduleDaysBetween: 7
#daysToExpireAfterDone: 2
repairRunThreadCount: 15
hangingRepairTimeoutMins: 30
storageType: database
enableCrossOrigin: true
incrementalRepair: false

logging:
  level: DEBUG
  loggers:
    io.dropwizard: DEBUG
    org.eclipse.jetty: DEBUG
  appenders:
    - type: console
      logFormat: "%-6level [%d] [%t] %logger{5} - %msg %n"
    - type: file
      logFormat: "%-6level [%d] [%t] %logger{5} - %msg %n"
      currentLogFilename: /var/log/cassandra_reaper/cassandra_reaper.log
      archivedLogFilenamePattern: /var/log/cassandra_reaper/cassandra_reaper.%d.log.gz

server:
  type: default
  applicationConnectors:
    - type: http
      port: 8666
      bindHost: 0.0.0.0
  adminConnectors:
    - type: http
      port: 8667
      bindHost: 0.0.0.0

database:
  driverClass: org.postgresql.Driver
  user: reaper
  password: my_secret_password
  url: jdbc:postgresql://localhost/reaper_db

jmxAuth:
    username: controlRole
    password: XXX

Then debug logs show:

DEBUG  [2015-12-17 10:09:05,918] [main] o.e.j.u.log - Logging to Logger[org.eclipse.jetty.util.log] via org.eclipse.jetty.util.log.Slf4jLog
DEBUG  [2015-12-17 10:09:05,927] [main] o.e.j.u.c.ContainerLifeCycle - i.d.j.MutableServletContextHandler@1fc32e4f{/,null,null} added {org.eclipse.jetty.servlet.ServletHandler@2f67b837,AUTO}
DEBUG  [2015-12-17 10:09:05,928] [main] o.e.j.u.c.ContainerLifeCycle - i.d.j.MutableServletContextHandler@6cce16f4{/,null,null} added {org.eclipse.jetty.servlet.ServletHandler@7efaad5a,AUTO}
DEBUG  [2015-12-17 10:09:05,941] [main] o.e.j.u.c.ContainerLifeCycle - org.eclipse.jetty.servlet.ServletHandler@7efaad5a added {tasks@6907b8e==io.dropwizard.servlets.tasks.TaskServlet,1,true,AUTO}
DEBUG  [2015-12-17 10:09:05,943] [main] o.e.j.u.c.ContainerLifeCycle - org.eclipse.jetty.servlet.ServletHandler@7efaad5a added {[/tasks/*]=>tasks,POJO}
DEBUG  [2015-12-17 10:09:05,946] [main] o.e.j.s.ServletHandler - filterNameMap={}
DEBUG  [2015-12-17 10:09:05,946] [main] o.e.j.s.ServletHandler - pathFilters=null
DEBUG  [2015-12-17 10:09:05,946] [main] o.e.j.s.ServletHandler - servletFilterMap=null
DEBUG  [2015-12-17 10:09:05,946] [main] o.e.j.s.ServletHandler - servletPathMap={/tasks/*=tasks@6907b8e==io.dropwizard.servlets.tasks.TaskServlet,1,true}
DEBUG  [2015-12-17 10:09:05,946] [main] o.e.j.s.ServletHandler - servletNameMap={tasks=tasks@6907b8e==io.dropwizard.servlets.tasks.TaskServlet,1,true}
INFO   [2015-12-17 10:09:05,955] [main] i.d.a.AssetsBundle - Registering AssetBundle with name: assets for path /webui/*
DEBUG  [2015-12-17 10:09:05,985] [main] o.e.j.u.c.ContainerLifeCycle - org.eclipse.jetty.servlet.ServletHandler@2f67b837 added {assets@ac107383==io.dropwizard.servlets.assets.AssetS
ervlet,1,true,AUTO}
DEBUG  [2015-12-17 10:09:05,985] [main] o.e.j.u.c.ContainerLifeCycle - org.eclipse.jetty.servlet.ServletHandler@2f67b837 added {[/webui/*]=>assets,POJO}
DEBUG  [2015-12-17 10:09:05,985] [main] o.e.j.s.ServletHandler - filterNameMap={}
DEBUG  [2015-12-17 10:09:05,985] [main] o.e.j.s.ServletHandler - pathFilters=null
DEBUG  [2015-12-17 10:09:05,985] [main] o.e.j.s.ServletHandler - servletFilterMap=null
DEBUG  [2015-12-17 10:09:05,986] [main] o.e.j.s.ServletHandler - servletPathMap={/webui/*=assets@ac107383==io.dropwizard.servlets.assets.AssetServlet,1,true}
DEBUG  [2015-12-17 10:09:05,986] [main] o.e.j.s.ServletHandler - servletNameMap={assets=assets@ac107383==io.dropwizard.servlets.assets.AssetServlet,1,true}
DEBUG  [2015-12-17 10:09:05,986] [main] c.s.r.ReaperApplication - repairIntensity: 0.9
DEBUG  [2015-12-17 10:09:05,986] [main] c.s.r.ReaperApplication - incrementalRepair:false
DEBUG  [2015-12-17 10:09:05,986] [main] c.s.r.ReaperApplication - repairRunThreadCount: 15
DEBUG  [2015-12-17 10:09:05,986] [main] c.s.r.ReaperApplication - segmentCount: 200
DEBUG  [2015-12-17 10:09:05,986] [main] c.s.r.ReaperApplication - repairParallelism: DATACENTER_AWARE
DEBUG  [2015-12-17 10:09:05,987] [main] c.s.r.ReaperApplication - hangingRepairTimeoutMins: 30
DEBUG  [2015-12-17 10:09:05,987] [main] c.s.r.ReaperApplication - jmxPorts: null
DEBUG  [2015-12-17 10:09:05,987] [main] c.s.r.ReaperApplication - adding signal handler for SIGHUP
INFO   [2015-12-17 10:09:05,987] [main] c.s.r.ReaperApplication - initializing runner thread pool with 15 threads
INFO   [2015-12-17 10:09:05,993] [main] c.s.r.ReaperApplication - initializing storage of type: database
INFO   [2015-12-17 10:09:06,035] [main] c.s.r.ReaperApplication - no JMX connection factory given in context, creating default
DEBUG  [2015-12-17 10:09:06,037] [main] o.e.j.u.c.ContainerLifeCycle - org.eclipse.jetty.servlet.ServletHandler@2f67b837 added {crossOriginRequests,AUTO}
DEBUG  [2015-12-17 10:09:06,040] [main] o.e.j.u.c.ContainerLifeCycle - org.eclipse.jetty.servlet.ServletHandler@2f67b837 added {[/*]/[]==31=>crossOriginRequests,POJO}
DEBUG  [2015-12-17 10:09:06,041] [main] o.e.j.s.ServletHandler - filterNameMap={crossOriginRequests=crossOriginRequests}
DEBUG  [2015-12-17 10:09:06,042] [main] o.e.j.s.ServletHandler - pathFilters=[[/*]/[]==31=>crossOriginRequests]
DEBUG  [2015-12-17 10:09:06,042] [main] o.e.j.s.ServletHandler - servletFilterMap={}
DEBUG  [2015-12-17 10:09:06,042] [main] o.e.j.s.ServletHandler - servletPathMap={/webui/*=assets@ac107383==io.dropwizard.servlets.assets.AssetServlet,1,true}
DEBUG  [2015-12-17 10:09:06,042] [main] o.e.j.s.ServletHandler - servletNameMap={assets=assets@ac107383==io.dropwizard.servlets.assets.AssetServlet,1,true}
DEBUG  [2015-12-17 10:09:06,042] [main] c.s.r.ReaperApplication - using specified JMX credentials for authentication
INFO   [2015-12-17 10:09:06,042] [main] c.s.r.ReaperApplication - creating and registering health checks
INFO   [2015-12-17 10:09:06,043] [main] c.s.r.ReaperApplication - creating resources and registering endpoints
INFO   [2015-12-17 10:09:07,049] [main] c.s.r.s.SchedulingManager - Starting new SchedulingManager instance
INFO   [2015-12-17 10:09:07,050] [main] c.s.r.ReaperApplication - resuming pending repair runs

And server is not started.

mlowicki commented 8 years ago

Keep in mind I've applied https://raw.githubusercontent.com/spotify/cassandra-reaper/master/src/main/db/reaper_db.sql.

mlowicki commented 8 years ago

Looks like issues with our PostgreSQL. Closing.