search

HubSpot / Singularity

Scheduler (HTTP API and webapp) for running Mesos tasks—long running processes, one-off tasks, and scheduled jobs. #hubspot-open-source
http://getsingularity.com/
Apache License 2.0
826 stars 188 forks source link

"driver not started" issue with multiple Singularity instances and racks #896

Closed scalp42 closed 8 years ago

scalp42 commented 8 years ago

Setup:

I noticed in the UI that it keeps saying that the "driver is not started" (all 3 instances are up and running):

Here is the log of a Singularity with "driver is not started":

INFO  [2016-02-16 19:14:30,384] io.dropwizard.assets.AssetsBundle: Registering AssetBundle with name: assets for path /static/*
INFO  [2016-02-16 19:14:30,412] io.dropwizard.assets.AssetsBundle: Registering AssetBundle with name: api-docs for path /api-docs/*
INFO  [2016-02-16 19:14:31,686] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.SingularityManagedScheduledExecutorServiceProvider
INFO  [2016-02-16 19:14:31,686] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.SingularityManagedScheduledExecutorServiceProvider
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.SingularityManagedScheduledExecutorServiceProvider
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.smtp.SingularitySmtpSender
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.SingularityCuratorProvider
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.SingularityHttpClient
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.smtp.SingularityMailer
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.data.ExecutorIdGenerator
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.mesos.SingularityLogSupport$$EnhancerByGuice$$47cc712a
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.SingularityManagedScheduledExecutorServiceFactory
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.SingularityDriverManager
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.SingularityLeaderController
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.SingularityLeaderLatch
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.smtp.SingularityMailRecordCleaner
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.hooks.SingularityWebhookPoller
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.sentry.SingularityExceptionNotifierManaged
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.metrics.SingularityGraphiteReporterManaged
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.scheduler.SingularityCleanupPoller
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.scheduler.SingularityExpiringUserActionPoller
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.data.history.SingularityHistoryPurger
INFO  [2016-02-16 19:14:31,687] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.scheduler.SingularityDeadSlavePoller
INFO  [2016-02-16 19:14:31,688] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.scheduler.SingularityCooldownPoller
INFO  [2016-02-16 19:14:31,688] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.scheduler.SingularityDeployPoller
INFO  [2016-02-16 19:14:31,688] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.scheduler.SingularitySchedulerPoller
INFO  [2016-02-16 19:14:31,688] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.scheduler.SingularityScheduledJobPoller
INFO  [2016-02-16 19:14:31,688] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.scheduler.SingularityTaskShellCommandDispatchPoller
INFO  [2016-02-16 19:14:31,688] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.scheduler.SingularityTaskReconciliationPoller
INFO  [2016-02-16 19:14:31,688] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.data.history.SingularityRequestHistoryPersister
INFO  [2016-02-16 19:14:31,688] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.data.history.SingularityDeployHistoryPersister
INFO  [2016-02-16 19:14:31,688] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected managed Object: com.hubspot.singularity.data.history.SingularityTaskHistoryPersister
INFO  [2016-02-16 19:14:31,688] com.hubspot.dropwizard.guicier.GuiceBundle: Added guice injected health check: com.hubspot.singularity.SingularityDropwizardHealthcheck
INFO  [2016-02-16 19:14:31,693] io.dropwizard.server.ServerFactory: Starting SingularityService
INFO  [2016-02-16 19:14:31,741] org.eclipse.jetty.setuid.SetUIDListener: Opened SingularityService@5e5073ab{HTTP/1.1}{0.0.0.0:8070}
INFO  [2016-02-16 19:14:31,743] org.eclipse.jetty.server.Server: jetty-9.0.z-SNAPSHOT
INFO  [2016-02-16 19:14:31,756] org.apache.curator.framework.imps.CuratorFrameworkImpl: Starting
INFO  [2016-02-16 19:14:31,761] org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
INFO  [2016-02-16 19:14:31,761] org.apache.zookeeper.ZooKeeper: Client environment:host.name=mesos-master-i-635669ba.example.com
INFO  [2016-02-16 19:14:31,761] org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_72
INFO  [2016-02-16 19:14:31,761] org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
INFO  [2016-02-16 19:14:31,761] org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre
INFO  [2016-02-16 19:14:31,761] org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/opt/singularity/SingularityService-0.4.9-shaded.jar
INFO  [2016-02-16 19:14:31,761] org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
INFO  [2016-02-16 19:14:31,761] org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
INFO  [2016-02-16 19:14:31,761] org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
INFO  [2016-02-16 19:14:31,761] org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
INFO  [2016-02-16 19:14:31,761] org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
INFO  [2016-02-16 19:14:31,761] org.apache.zookeeper.ZooKeeper: Client environment:os.version=3.19.0-43-generic
INFO  [2016-02-16 19:14:31,761] org.apache.zookeeper.ZooKeeper: Client environment:user.name=root
INFO  [2016-02-16 19:14:31,762] org.apache.zookeeper.ZooKeeper: Client environment:user.home=/root
INFO  [2016-02-16 19:14:31,762] org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/opt/singularity
INFO  [2016-02-16 19:14:31,762] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=10.92.41.178:2181,10.92.31.170:2181,10.92.21.247:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@45d64d27
INFO  [2016-02-16 19:14:31,775] org.apache.zookeeper.ClientCnxn: Opening socket connection to server 10.92.41.178/10.92.41.178:2181. Will not attempt to authenticate using SASL (unknown error)
INFO  [2016-02-16 19:14:31,781] org.apache.zookeeper.ClientCnxn: Socket connection established to 10.92.41.178/10.92.41.178:2181, initiating session
INFO  [2016-02-16 19:14:31,789] org.apache.zookeeper.ClientCnxn: Session establishment complete on server 10.92.41.178/10.92.41.178:2181, sessionid = 0x152c788a234004f, negotiated timeout = 40000
INFO  [2016-02-16 19:14:31,792] org.apache.curator.framework.state.ConnectionStateManager: State change: CONNECTED
INFO  [2016-02-16 19:14:32,794] com.hubspot.singularity.SingularityCuratorProvider: Connected to ZK after 00:01.022
WARN  [2016-02-16 19:14:32,803] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Not running SingularityMailRecordCleaner due to delay value of 0
INFO  [2016-02-16 19:14:32,803] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Starting a SingularityWebhookPoller with a 00:10.000 delay
INFO  [2016-02-16 19:14:32,809] com.hubspot.singularity.sentry.NotifyingExceptionMapper: Setting NotifyingUncaughtExceptionManager as the default uncaught exception provider...
INFO  [2016-02-16 19:14:32,809] com.hubspot.singularity.metrics.SingularityGraphiteReporterManaged: Not reporting data points to graphite.
INFO  [2016-02-16 19:14:32,809] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Starting a SingularityCleanupPoller with a 00:05.000 delay
INFO  [2016-02-16 19:14:32,811] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Starting a SingularityExpiringUserActionPoller with a 00:45.000 delay
INFO  [2016-02-16 19:14:32,812] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: SingularityHistoryPurger is not enabled, not starting.
INFO  [2016-02-16 19:14:32,812] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Starting a SingularityDeadSlavePoller with a 60:00.000 delay
INFO  [2016-02-16 19:14:32,812] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Starting a SingularityCooldownPoller with a 07:30.000 delay
INFO  [2016-02-16 19:14:32,813] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Starting a SingularityDeployPoller with a 00:05.000 delay
INFO  [2016-02-16 19:14:32,813] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Starting a SingularitySchedulerPoller with a 00:05.000 delay
INFO  [2016-02-16 19:14:32,813] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Starting a SingularityScheduledJobPoller with a 10:00.000 delay
INFO  [2016-02-16 19:14:32,813] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Starting a SingularityTaskShellCommandDispatchPoller with a 00:05.000 delay
INFO  [2016-02-16 19:14:32,814] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Starting a SingularityTaskReconciliationPoller with a 10:00.000 delay
INFO  [2016-02-16 19:14:32,814] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Starting a SingularityRequestHistoryPersister with a 60:00.000 delay
INFO  [2016-02-16 19:14:32,814] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Starting a SingularityDeployHistoryPersister with a 60:00.000 delay
INFO  [2016-02-16 19:14:32,814] com.hubspot.singularity.scheduler.SingularityLeaderOnlyPoller: Starting a SingularityTaskHistoryPersister with a 60:00.000 delay
INFO  [2016-02-16 19:14:32,877] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.sentry.NotifyingExceptionMapper as a provider class
INFO  [2016-02-16 19:14:32,877] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.guice.GuicePropertyFilteringMessageBodyWriter as a provider class
INFO  [2016-02-16 19:14:32,877] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.resources.DeployResource as a root resource class
INFO  [2016-02-16 19:14:32,877] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.resources.HistoryResource as a root resource class
INFO  [2016-02-16 19:14:32,877] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.resources.RackResource as a root resource class
INFO  [2016-02-16 19:14:32,877] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.resources.RequestResource as a root resource class
INFO  [2016-02-16 19:14:32,877] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.resources.S3LogResource as a root resource class
INFO  [2016-02-16 19:14:32,877] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.resources.SandboxResource as a root resource class
INFO  [2016-02-16 19:14:32,877] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.resources.SlaveResource as a root resource class
INFO  [2016-02-16 19:14:32,877] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.resources.StateResource as a root resource class
INFO  [2016-02-16 19:14:32,877] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.resources.TaskResource as a root resource class
INFO  [2016-02-16 19:14:32,878] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.resources.TestResource as a root resource class
INFO  [2016-02-16 19:14:32,878] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.resources.WebhookResource as a root resource class
INFO  [2016-02-16 19:14:32,878] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.resources.AuthResource as a root resource class
INFO  [2016-02-16 19:14:32,878] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.resources.MetricsResource as a root resource class
INFO  [2016-02-16 19:14:32,878] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Registering com.hubspot.singularity.resources.StaticCatchallResource as a root resource class
INFO  [2016-02-16 19:14:32,879] com.sun.jersey.server.impl.application.WebApplicationImpl: Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM'
INFO  [2016-02-16 19:14:32,939] io.dropwizard.jersey.DropwizardResourceConfig: The following paths were found for the configured resources:

    DELETE  /api/deploys/deploy/{deployId}/request/{requestId} (com.hubspot.singularity.resources.DeployResource)
    GET     /api/deploys/pending (com.hubspot.singularity.resources.DeployResource)
    POST    /api/deploys (com.hubspot.singularity.resources.DeployResource)
    GET     /api/history/request/{requestId}/deploy/{deployId} (com.hubspot.singularity.resources.HistoryResource)
    GET     /api/history/request/{requestId}/deploy/{deployId}/tasks/active (com.hubspot.singularity.resources.HistoryResource)
    GET     /api/history/request/{requestId}/deploy/{deployId}/tasks/inactive (com.hubspot.singularity.resources.HistoryResource)
    GET     /api/history/request/{requestId}/deploys (com.hubspot.singularity.resources.HistoryResource)
    GET     /api/history/request/{requestId}/requests (com.hubspot.singularity.resources.HistoryResource)
    GET     /api/history/request/{requestId}/run/{runId} (com.hubspot.singularity.resources.HistoryResource)
    GET     /api/history/request/{requestId}/tasks (com.hubspot.singularity.resources.HistoryResource)
    GET     /api/history/request/{requestId}/tasks/active (com.hubspot.singularity.resources.HistoryResource)
    GET     /api/history/requests/search (com.hubspot.singularity.resources.HistoryResource)
    GET     /api/history/task/{taskId} (com.hubspot.singularity.resources.HistoryResource)
    DELETE  /api/racks/rack/{rackId} (com.hubspot.singularity.resources.RackResource)
    GET     /api/racks (com.hubspot.singularity.resources.RackResource)
    GET     /api/racks/rack/{rackId} (com.hubspot.singularity.resources.RackResource)
    POST    /api/racks/rack/{rackId}/activate (com.hubspot.singularity.resources.RackResource)
    POST    /api/racks/rack/{rackId}/decommission (com.hubspot.singularity.resources.RackResource)
    POST    /api/racks/rack/{rackId}/freeze (com.hubspot.singularity.resources.RackResource)
    DELETE  /api/requests/request/{requestId} (com.hubspot.singularity.resources.RequestResource)
    DELETE  /api/requests/request/{requestId} (com.hubspot.singularity.resources.RequestResource)
    DELETE  /api/requests/request/{requestId}/bounce (com.hubspot.singularity.resources.RequestResource)
    DELETE  /api/requests/request/{requestId}/pause (com.hubspot.singularity.resources.RequestResource)
    DELETE  /api/requests/request/{requestId}/scale (com.hubspot.singularity.resources.RequestResource)
    DELETE  /api/requests/request/{requestId}/skipHealthchecks (com.hubspot.singularity.resources.RequestResource)
    GET     /api/requests (com.hubspot.singularity.resources.RequestResource)
    GET     /api/requests/active (com.hubspot.singularity.resources.RequestResource)
    GET     /api/requests/cooldown (com.hubspot.singularity.resources.RequestResource)
    GET     /api/requests/finished (com.hubspot.singularity.resources.RequestResource)
    GET     /api/requests/lbcleanup (com.hubspot.singularity.resources.RequestResource)
    GET     /api/requests/paused (com.hubspot.singularity.resources.RequestResource)
    GET     /api/requests/queued/cleanup (com.hubspot.singularity.resources.RequestResource)
    GET     /api/requests/queued/pending (com.hubspot.singularity.resources.RequestResource)
    GET     /api/requests/request/{requestId} (com.hubspot.singularity.resources.RequestResource)
    POST    /api/requests (com.hubspot.singularity.resources.RequestResource)
    POST    /api/requests/request/{requestId}/bounce (com.hubspot.singularity.resources.RequestResource)
    POST    /api/requests/request/{requestId}/bounce (com.hubspot.singularity.resources.RequestResource)
    POST    /api/requests/request/{requestId}/exit-cooldown (com.hubspot.singularity.resources.RequestResource)
    POST    /api/requests/request/{requestId}/exit-cooldown (com.hubspot.singularity.resources.RequestResource)
    POST    /api/requests/request/{requestId}/pause (com.hubspot.singularity.resources.RequestResource)
    POST    /api/requests/request/{requestId}/pause (com.hubspot.singularity.resources.RequestResource)
    POST    /api/requests/request/{requestId}/run (com.hubspot.singularity.resources.RequestResource)
    POST    /api/requests/request/{requestId}/run (com.hubspot.singularity.resources.RequestResource)
    POST    /api/requests/request/{requestId}/unpause (com.hubspot.singularity.resources.RequestResource)
    POST    /api/requests/request/{requestId}/unpause (com.hubspot.singularity.resources.RequestResource)
    PUT     /api/requests/request/{requestId}/scale (com.hubspot.singularity.resources.RequestResource)
    PUT     /api/requests/request/{requestId}/skipHealthchecks (com.hubspot.singularity.resources.RequestResource)
    GET     /api/logs/request/{requestId} (com.hubspot.singularity.resources.S3LogResource)
    GET     /api/logs/request/{requestId}/deploy/{deployId} (com.hubspot.singularity.resources.S3LogResource)
    GET     /api/logs/task/{taskId} (com.hubspot.singularity.resources.S3LogResource)
    GET     /api/sandbox/{taskId}/browse (com.hubspot.singularity.resources.SandboxResource)
    GET     /api/sandbox/{taskId}/read (com.hubspot.singularity.resources.SandboxResource)
    DELETE  /api/slaves/slave/{slaveId} (com.hubspot.singularity.resources.SlaveResource)
    GET     /api/slaves (com.hubspot.singularity.resources.SlaveResource)
    GET     /api/slaves/slave/{slaveId} (com.hubspot.singularity.resources.SlaveResource)
    POST    /api/slaves/slave/{slaveId}/activate (com.hubspot.singularity.resources.SlaveResource)
    POST    /api/slaves/slave/{slaveId}/decommission (com.hubspot.singularity.resources.SlaveResource)
    POST    /api/slaves/slave/{slaveId}/freeze (com.hubspot.singularity.resources.SlaveResource)
    GET     /api/state (com.hubspot.singularity.resources.StateResource)
    GET     /api/state/requests/over-provisioned (com.hubspot.singularity.resources.StateResource)
    GET     /api/state/requests/under-provisioned (com.hubspot.singularity.resources.StateResource)
    DELETE  /api/tasks/task/{taskId} (com.hubspot.singularity.resources.TaskResource)
    DELETE  /api/tasks/task/{taskId} (com.hubspot.singularity.resources.TaskResource)
    GET     /api/tasks/active (com.hubspot.singularity.resources.TaskResource)
    GET     /api/tasks/active/slave/{slaveId} (com.hubspot.singularity.resources.TaskResource)
    GET     /api/tasks/cleaning (com.hubspot.singularity.resources.TaskResource)
    GET     /api/tasks/killed (com.hubspot.singularity.resources.TaskResource)
    GET     /api/tasks/lbcleanup (com.hubspot.singularity.resources.TaskResource)
    GET     /api/tasks/scheduled (com.hubspot.singularity.resources.TaskResource)
    GET     /api/tasks/scheduled/ids (com.hubspot.singularity.resources.TaskResource)
    GET     /api/tasks/scheduled/request/{requestId} (com.hubspot.singularity.resources.TaskResource)
    GET     /api/tasks/scheduled/task/{pendingTaskId} (com.hubspot.singularity.resources.TaskResource)
    GET     /api/tasks/task/{taskId} (com.hubspot.singularity.resources.TaskResource)
    GET     /api/tasks/task/{taskId}/cleanup (com.hubspot.singularity.resources.TaskResource)
    GET     /api/tasks/task/{taskId}/statistics (com.hubspot.singularity.resources.TaskResource)
    POST    /api/tasks/task/{taskId}/command (com.hubspot.singularity.resources.TaskResource)
    POST    /api/test/abort (com.hubspot.singularity.resources.TestResource)
    POST    /api/test/exception (com.hubspot.singularity.resources.TestResource)
    POST    /api/test/leader (com.hubspot.singularity.resources.TestResource)
    POST    /api/test/notleader (com.hubspot.singularity.resources.TestResource)
    POST    /api/test/scheduler/statusUpdate/{taskId}/{taskState} (com.hubspot.singularity.resources.TestResource)
    POST    /api/test/start (com.hubspot.singularity.resources.TestResource)
    POST    /api/test/stop (com.hubspot.singularity.resources.TestResource)
    DELETE  /api/webhooks/{webhookId} (com.hubspot.singularity.resources.WebhookResource)
    GET     /api/webhooks (com.hubspot.singularity.resources.WebhookResource)
    GET     /api/webhooks/deploy/{webhookId} (com.hubspot.singularity.resources.WebhookResource)
    GET     /api/webhooks/request/{webhookId} (com.hubspot.singularity.resources.WebhookResource)
    GET     /api/webhooks/task/{webhookId} (com.hubspot.singularity.resources.WebhookResource)
    POST    /api/webhooks (com.hubspot.singularity.resources.WebhookResource)
    GET     /api/auth/user (com.hubspot.singularity.resources.AuthResource)
    GET     /api/metrics (com.hubspot.singularity.resources.MetricsResource)
    GET     /{uiPath:.*} (com.hubspot.singularity.resources.StaticCatchallResource)

INFO  [2016-02-16 19:14:32,952] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.sentry.NotifyingExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
INFO  [2016-02-16 19:14:32,955] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.guice.GuicePropertyFilteringMessageBodyWriter to GuiceManagedComponentProvider with the scope "Singleton"
INFO  [2016-02-16 19:14:33,064] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.resources.DeployResource to GuiceManagedComponentProvider with the scope "PerRequest"
INFO  [2016-02-16 19:14:33,069] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.resources.HistoryResource to GuiceManagedComponentProvider with the scope "PerRequest"
INFO  [2016-02-16 19:14:33,076] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.resources.RackResource to GuiceManagedComponentProvider with the scope "PerRequest"
INFO  [2016-02-16 19:14:33,082] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.resources.RequestResource to GuiceManagedComponentProvider with the scope "PerRequest"
INFO  [2016-02-16 19:14:33,083] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.resources.S3LogResource to GuiceManagedComponentProvider with the scope "PerRequest"
INFO  [2016-02-16 19:14:33,083] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.resources.SandboxResource to GuiceManagedComponentProvider with the scope "PerRequest"
INFO  [2016-02-16 19:14:33,085] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.resources.SlaveResource to GuiceManagedComponentProvider with the scope "PerRequest"
INFO  [2016-02-16 19:14:33,086] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.resources.StateResource to GuiceManagedComponentProvider with the scope "PerRequest"
INFO  [2016-02-16 19:14:33,088] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.resources.TaskResource to GuiceManagedComponentProvider with the scope "PerRequest"
INFO  [2016-02-16 19:14:33,090] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.resources.TestResource to GuiceManagedComponentProvider with the scope "PerRequest"
INFO  [2016-02-16 19:14:33,091] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.resources.WebhookResource to GuiceManagedComponentProvider with the scope "PerRequest"
INFO  [2016-02-16 19:14:33,091] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.resources.AuthResource to GuiceManagedComponentProvider with the scope "PerRequest"
INFO  [2016-02-16 19:14:33,091] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.resources.MetricsResource to GuiceManagedComponentProvider with the scope "PerRequest"
INFO  [2016-02-16 19:14:33,091] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory: Binding com.hubspot.singularity.resources.StaticCatchallResource to GuiceManagedComponentProvider with the scope "Singleton"
WARN  [2016-02-16 19:14:33,094] com.sun.jersey.spi.inject.Errors: The following warnings have been detected with resource and/or provider classes:
  WARNING: A sub-resource method, public java.util.List com.hubspot.singularity.resources.RackResource.getRacks(com.google.common.base.Optional), with URI template, "/", is treated as a resource method
  WARNING: A sub-resource method, public java.util.List com.hubspot.singularity.resources.SlaveResource.getSlaves(com.google.common.base.Optional), with URI template, "/", is treated as a resource method
INFO  [2016-02-16 19:14:33,094] org.eclipse.jetty.server.handler.ContextHandler: Started i.d.j.MutableServletContextHandler@895416d{/,null,AVAILABLE}
INFO  [2016-02-16 19:14:33,096] io.dropwizard.setup.AdminEnvironment: tasks =

    POST    /tasks/gc (io.dropwizard.servlets.tasks.GarbageCollectionTask)

INFO  [2016-02-16 19:14:33,099] org.eclipse.jetty.server.handler.ContextHandler: Started i.d.j.MutableServletContextHandler@26cb5207{/admin,null,AVAILABLE}
INFO  [2016-02-16 19:14:33,101] org.eclipse.jetty.server.ServerConnector: Started SingularityService@5e5073ab{HTTP/1.1}{0.0.0.0:8070}
INFO  [2016-02-16 19:17:53,677] com.hubspot.singularity.data.StateManager: Generated new state and saved 653 bytes in 00:00.092

Here is the config:

root@mesos-master-i-635669ba # cat /opt/singularity/singularity_config.yaml

---
enableCorsFilter: true
sandboxDefaultsToTaskId: false
server:
  type: simple
  applicationContextPath: "/"
  connector:
    type: http
    port: 8070
  requestLog:
    appenders:
    - type: file
      currentLogFilename: "/var/log/singularity-access.log"
      archivedLogFilenamePattern: "/var/log/singularity-access-%d.log.gz"
mesos:
  master: |
    zk://10.92.21.247:2181,10.92.31.170:2181,10.92.41.178:2181/mesos
  defaultCpus: '0.1'
  defaultMemory: '128'
  frameworkName: Singularity
  frameworkId: Singularity
  frameworkFailoverTimeout: 1000000
  defaultRackId: us-west-2a
zookeeper:
  quorum: 10.92.41.178:2181,10.92.31.170:2181,10.92.21.247:2181
  zkNamespace: singularity
  sessionTimeoutMillis: 60000
  connectTimeoutMillis: 5000
  retryBaseSleepTimeMilliseconds: 1000
  retryMaxTries: 3
logging:
  loggers:
    '"com.hubspot.singularity"': INFO
ui:
  title: Singularity
hostname: mesos-master-i-635669ba.example.com
scalp42 commented 8 years ago

I can connect to all 3 Singularity instances just to confirm.

ssalinas commented 8 years ago

@scalp42 The driver will only be started on the leading instance of Singularity. Only the leader instance will actually process deploys, scheduled tasks, etc, but all instances can serve api requests. Non-leading instances will take over leadership should the leading instance become unavailable

scalp42 commented 8 years ago

@ssalinas it makes much more sense now, so everything is working as expected then.

Side question but tasks scheduled through Singularity will still be deployed on Mesos slaves correct?

I want to make sure that by "driver", it doesn't mean "executor".

ssalinas commented 8 years ago

Sry, missed this last question. Yes, tasks will be scheduled on the mesos slaves. The driver refers to the scheduler driver, not executor