search

vmware-archive / pcfdev

This is the depricated version of PCF Dev - please visit the current Github repository https://github.com/cloudfoundry-incubator/cfdev for the latest updates
Apache License 2.0
273 stars 67 forks source link

cf dev start is stuck at "57 out of 58 running" #294

Closed brennerm closed 6 years ago

brennerm commented 6 years ago

Executing "cf dev start" gives the following result:

Using existing image. Allocating 4096 MB out of 15715 MB total system memory (11282 MB free). Importing VM... Starting VM... Provisioning VM... Waiting for services to start... 7 out of 58 running 7 out of 58 running 7 out of 58 running 7 out of 58 running 35 out of 58 running 39 out of 58 running 55 out of 58 running 57 out of 58 running ... 57 out of 58 running

I'm already waiting around half an hour for it to finish. It seems like one service fails to start. Please tell me how to access the logs of the pcfdev VM to find out what's the problem.

cf version: cf version 6.34.1+bbdf81482.2018-01-17 cf dev version: PCF Dev version 0.28.0 (CLI: 5cda315, OVA: 0.547.0) Virtualbox version: Version 5.1.32 r120294 (Qt5.5.1)

cf-gitbot commented 6 years ago

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

xPerdev commented 6 years ago

MacBook Pro (13-inch, Late 2016)/Intel Core i7 2,4 GHz dual-core/16Gb Memory/ 500 Gb SSD VirtualBox Version 5.2.6 r120293 (Qt5.6.3) PCF Dev version 0.28.0 (CLI: 5cda315, OVA: 0.547.0)

All was running fine last week, but since yesterday (February 18th, 2018) it fails to complete startup as only 57 of 58 services will start.

$ cf dev start
Using existing image.
Allocating 4096 MB out of 16384 MB total system memory (12274 MB free).
Importing VM...
Starting VM...
Provisioning VM...
Waiting for services to start...
7 out of 58 running
7 out of 58 running
7 out of 58 running
7 out of 58 running
35 out of 58 running
39 out of 58 running
55 out of 58 running
57 out of 58 running
…
57 out of 58 running
Timed out after 3600 seconds.
FAILED
Error: failed to provision VM: Process exited with: 1. Reason was:  ().

Gathering log files using cf dev debug command and ssh’ing into the virtual machine teaches me that it’s the service syslog_drain_binder that doesn’t start because its certificate (/var/vcap/jobs/syslog_drain_binder/config/certs/syslogdrainbinder.crt) expired at Feb 15 22:35:39 2018 GMT

As such it seems we need to update the certificate to make it work again.

The information in the logs that lead me to this conclusion are below:

/var/pcfdev/provision.log

----------- STARTING pre-start-execution at Mon Feb 19 07:58:22 UTC 2018 --------------
     2 2018-02-19 07:58:33 ----- pre-start execution script: pre-start completed with error code 0
     3 2018-02-19 07:58:33 ----- pre-start execution script: completed pre-start execution
     4 + start_services mariadb_ctrl galera-healthcheck
     5 + for service in '$@'
     6 + /var/vcap/bosh/bin/monit start mariadb_ctrl
     7 + for service in '$@'
     8 + /var/vcap/bosh/bin/monit start galera-healthcheck
     9 + for service in '$@'
    10 + grep -q running
    11 + grep mariadb_ctrl
    12 + monit_summary
    13 ++ /var/vcap/bosh/bin/monit summary
    14 + output='The Monit daemon 5.2.5 uptime: 0m
    15
    16 Process '\''pcfdev-api'\''                not monitored
    17 Process '\''pcfdev-nginx'\''              not monitored
    18 Process '\''vxlan-policy-agent'\''        not monitored
    19 Process '\''uaa'\''                       not monitored
    20 Process '\''tps_listener'\''              not monitored
    21 Process '\''tps_watcher'\''               not monitored
    22 Process '\''tcp_router'\''                not monitored
    23 Process '\''tcp_emitter'\''               not monitored
    24 Process '\''syslog_drain_binder'\''       not monitored
    25 Process '\''syslog-configurator'\''       not monitored
    26 Process '\''stager'\''                    not monitored
    27 Process '\''ssh_proxy'\''                 not monitored
    28 Process '\''silk-daemon'\''               not monitored
    29 Process '\''silk-controller'\''           not monitored
    30 Process '\''routing-api'\''               not monitored
    31 Process '\''route_registrar'\''           not monitored
    32 Process '\''route_emitter'\''             not monitored
    33 Process '\''rep'\''                       not monitored
    34 Process '\''rabbitmq-server'\''           not monitored
    35 Process '\''rabbitmq-broker'\''           not monitored
    36 Process '\''policy-server'\''             not monitored
    37 Process '\''nsync_listener'\''            not monitored
    38 Process '\''nsync_bulker'\''              not monitored
    39 Process '\''netmon'\''                    not monitored
    40 Process '\''nats'\''                      not monitored
    41 Process '\''mariadb_ctrl'\''              not monitored - start pending
    42 Process '\''galera-healthcheck'\''        not monitored - start pending
    43 Process '\''gra-log-purger-executable'\'' not monitored
    44 Process '\''cluster_health_logger'\''     not monitored
    45 Process '\''metron_agent'\''              not monitored
    46 Process '\''loggregator_trafficcontroller'\'' not monitored
    47 Process '\''localdriver'\''               not monitored
    48 Process '\''localbroker'\''               not monitored
    49 Process '\''gorouter'\''                  not monitored
    50 Process '\''garden'\''                    not monitored
    51 Process '\''file_server'\''               not monitored
    52 Process '\''etcd'\''                      not monitored
    53 Process '\''etcd_consistency_checker'\''  not monitored
    54 Process '\''doppler'\''                   not monitored
    55 Process '\''consul_agent'\''              not monitored
    56 Process '\''cloud_controller_worker_1'\'' not monitored
    57 Process '\''cloud_controller_ng'\''       not monitored
    58 Process '\''cloud_controller_worker_local_1'\'' not monitored
    59 Process '\''cloud_controller_worker_local_2'\'' not monitored
    60 Process '\''nginx_cc'\''                  not monitored
    61 Process '\''cloud_controller_clock'\''    not monitored
    62 Process '\''process-watcher'\''           not monitored
    63 Process '\''process-destroyer'\''         not monitored
    64 Process '\''cf-redis-broker'\''           not monitored
    65 Process '\''broker-nginx'\''              not monitored
    66 Process '\''cf-mysql-broker'\''           not monitored
    67 Process '\''quota-enforcer'\''            not monitored
    68 Process '\''cc_uploader'\''               not monitored
    69 Process '\''blobstore_nginx'\''           not monitored
    70 Process '\''blobstore_url_signer'\''      not monitored
    71 Process '\''bbs'\''                       not monitored
    72 Process '\''auctioneer'\''                not monitored
    73 System '\''system_localhost'\''           not monitored'
    74 + [[ The Monit daemon 5.2.5 uptime: 0m
…
+ for service in '$(stopped_services)'
 17977 + /var/vcap/bosh/bin/monit start pcfdev-api
 17978 + for service in '$(stopped_services)'
 17979 + /var/vcap/bosh/bin/monit start pcfdev-nginx
 17980 + for service in '$(stopped_services)'
 17981 + /var/vcap/bosh/bin/monit start vxlan-policy-agent
 17982 + for service in '$(stopped_services)'
 17983 + /var/vcap/bosh/bin/monit start tps_listener
 17984 + for service in '$(stopped_services)'
 17985 + /var/vcap/bosh/bin/monit start tps_watcher
 17986 + for service in '$(stopped_services)'
 17987 + /var/vcap/bosh/bin/monit start tcp_router
 17988 + for service in '$(stopped_services)'
 17989 + /var/vcap/bosh/bin/monit start tcp_emitter
 17990 + for service in '$(stopped_services)'
 17991 + /var/vcap/bosh/bin/monit start syslog_drain_binder
 17992 + for service in '$(stopped_services)'
 17993 + /var/vcap/bosh/bin/monit start syslog-configurator
 17994 + for service in '$(stopped_services)'
 17995 + /var/vcap/bosh/bin/monit start stager
 17996 + for service in '$(stopped_services)'
 17997 + /var/vcap/bosh/bin/monit start ssh_proxy
 17998 + for service in '$(stopped_services)'
 17999 + /var/vcap/bosh/bin/monit start silk-daemon
 18000 + for service in '$(stopped_services)'
 18001 + /var/vcap/bosh/bin/monit start silk-controller
 18002 + for service in '$(stopped_services)'
 18003 + /var/vcap/bosh/bin/monit start routing-api
 18004 + for service in '$(stopped_services)'
 18005 + /var/vcap/bosh/bin/monit start route_registrar
 18006 + for service in '$(stopped_services)'
 18007 + /var/vcap/bosh/bin/monit start route_emitter
 18008 + for service in '$(stopped_services)'
 18009 + /var/vcap/bosh/bin/monit start rep
 18010 + for service in '$(stopped_services)'
 18011 + /var/vcap/bosh/bin/monit start rabbitmq-server
 18012 + for service in '$(stopped_services)'
 18013 + /var/vcap/bosh/bin/monit start rabbitmq-broker
 18014 + for service in '$(stopped_services)'
 18015 + /var/vcap/bosh/bin/monit start policy-server
 18016 + for service in '$(stopped_services)'
 18017 + /var/vcap/bosh/bin/monit start nsync_listener
 18018 + for service in '$(stopped_services)'
 18019 + /var/vcap/bosh/bin/monit start nsync_bulker
 18020 + for service in '$(stopped_services)'
 18021 + /var/vcap/bosh/bin/monit start netmon
 18022 + for service in '$(stopped_services)'
 18023 + /var/vcap/bosh/bin/monit start nats
 18024 + for service in '$(stopped_services)'
 18025 + /var/vcap/bosh/bin/monit start gra-log-purger-executable
 18026 + for service in '$(stopped_services)'
 18027 + /var/vcap/bosh/bin/monit start cluster_health_logger
 18028 + for service in '$(stopped_services)'
 18029 + /var/vcap/bosh/bin/monit start metron_agent
 18030 + for service in '$(stopped_services)'
 18031 + /var/vcap/bosh/bin/monit start loggregator_trafficcontroller
 18032 + for service in '$(stopped_services)'
 18033 + /var/vcap/bosh/bin/monit start localdriver
 18034 + for service in '$(stopped_services)'
 18035 + /var/vcap/bosh/bin/monit start localbroker
 18036 + for service in '$(stopped_services)'
 18037 + /var/vcap/bosh/bin/monit start gorouter
 18038 + for service in '$(stopped_services)'
 18039 + /var/vcap/bosh/bin/monit start file_server
 18040 + for service in '$(stopped_services)'
 18041 + /var/vcap/bosh/bin/monit start etcd_consistency_checker
 18042 + for service in '$(stopped_services)'
 18043 + /var/vcap/bosh/bin/monit start doppler
 18044 + for service in '$(stopped_services)'
 18045 + /var/vcap/bosh/bin/monit start cloud_controller_worker_1
 18046 + for service in '$(stopped_services)'
 18047 + /var/vcap/bosh/bin/monit start cloud_controller_ng
 18048 + for service in '$(stopped_services)'
 18049 + /var/vcap/bosh/bin/monit start cloud_controller_worker_local_1
 18050 + for service in '$(stopped_services)'
 18051 + /var/vcap/bosh/bin/monit start cloud_controller_worker_local_2
 18052 + for service in '$(stopped_services)'
 18053 + /var/vcap/bosh/bin/monit start nginx_cc
 18054 + for service in '$(stopped_services)'
 18055 + /var/vcap/bosh/bin/monit start cloud_controller_clock
 18056 + for service in '$(stopped_services)'
 18057 + /var/vcap/bosh/bin/monit start process-watcher
 18058 + for service in '$(stopped_services)'
 18059 + /var/vcap/bosh/bin/monit start process-destroyer
 18060 + for service in '$(stopped_services)'
 18061 + /var/vcap/bosh/bin/monit start cf-redis-broker
 18062 + for service in '$(stopped_services)'
 18063 + /var/vcap/bosh/bin/monit start broker-nginx
 18064 + for service in '$(stopped_services)'
 18065 + /var/vcap/bosh/bin/monit start cf-mysql-broker
 18066 + for service in '$(stopped_services)'
 18067 + /var/vcap/bosh/bin/monit start quota-enforcer
 18068 + for service in '$(stopped_services)'
 18069 + /var/vcap/bosh/bin/monit start cc_uploader
 18070 + for service in '$(stopped_services)'
 18071 + /var/vcap/bosh/bin/monit start blobstore_nginx
 18072 + for service in '$(stopped_services)'
 18073 + /var/vcap/bosh/bin/monit start blobstore_url_signer
 18074 + for service in '$(stopped_services)'
 18075 + /var/vcap/bosh/bin/monit start auctioneer
 18076 + for service in '$(stopped_services)'
 18077 + /var/vcap/bosh/bin/monit start system_localhost
 18078 ++ total_services
 18079 ++ monit_summary
 18080 ++ grep -E '^(Process|File|System)'
 18081 ++ wc -l
 18082 +++ /var/vcap/bosh/bin/monit summary
 18083 ++ output='The Monit daemon 5.2.5 uptime: 2m
 18084
 18085 Process '\''pcfdev-api'\''                initializing
 18086 Process '\''pcfdev-nginx'\''              initializing
 18087 Process '\''vxlan-policy-agent'\''        not monitored - start pending
 18088 Process '\''uaa'\''                       running
 18089 Process '\''tps_listener'\''              not monitored - start pending
 18090 Process '\''tps_watcher'\''               not monitored - start pending
 18091 Process '\''tcp_router'\''                not monitored - start pending
 18092 Process '\''tcp_emitter'\''               not monitored - start pending
 18093 Process '\''syslog_drain_binder'\''       not monitored - start pending
 18094 Process '\''syslog-configurator'\''       not monitored - start pending
 18095 Process '\''stager'\''                    not monitored - start pending
 18096 Process '\''ssh_proxy'\''                 not monitored - start pending
 18097 Process '\''silk-daemon'\''               not monitored - start pending
 18098 Process '\''silk-controller'\''           not monitored - start pending
 18099 Process '\''routing-api'\''               not monitored - start pending
 18100 Process '\''route_registrar'\''           not monitored - start pending
 18101 Process '\''route_emitter'\''             not monitored - start pending
 18102 Process '\''rep'\''                       not monitored - start pending
 18103 Process '\''rabbitmq-server'\''           not monitored - start pending
 18104 Process '\''rabbitmq-broker'\''           not monitored - start pending
 18105 Process '\''policy-server'\''             not monitored - start pending
 18106 Process '\''nsync_listener'\''            not monitored - start pending
 18107 Process '\''nsync_bulker'\''              not monitored - start pending
 18108 Process '\''netmon'\''                    not monitored - start pending
 18109 Process '\''nats'\''                      not monitored - start pending
 18110 Process '\''mariadb_ctrl'\''              running
 18111 Process '\''galera-healthcheck'\''        running
 18112 Process '\''gra-log-purger-executable'\'' not monitored - start pending
 18113 Process '\''cluster_health_logger'\''     not monitored - start pending
 18114 Process '\''metron_agent'\''              not monitored - start pending
 18115 Process '\''loggregator_trafficcontroller'\'' not monitored - start pending
 18116 Process '\''localdriver'\''               not monitored - start pending
 18117 Process '\''localbroker'\''               not monitored - start pending
 18118 Process '\''gorouter'\''                  not monitored - start pending
 18119 Process '\''garden'\''                    running
 18120 Process '\''file_server'\''               not monitored - start pending
 18121 Process '\''etcd'\''                      running
 18122 Process '\''etcd_consistency_checker'\''  not monitored - start pending
 18123 Process '\''doppler'\''                   not monitored - start pending
 18124 Process '\''consul_agent'\''              running
 18125 Process '\''cloud_controller_worker_1'\'' not monitored - start pending
 18126 Process '\''cloud_controller_ng'\''       not monitored - start pending
 18127 Process '\''cloud_controller_worker_local_1'\'' not monitored - start pending
 18128 Process '\''cloud_controller_worker_local_2'\'' not monitored - start pending
 18129 Process '\''nginx_cc'\''                  not monitored - start pending
 18130 Process '\''cloud_controller_clock'\''    not monitored - start pending
 18131 Process '\''process-watcher'\''           not monitored - start pending
 18132 Process '\''process-destroyer'\''         not monitored - start pending
 18133 Process '\''cf-redis-broker'\''           not monitored - start pending
 18134 Process '\''broker-nginx'\''              not monitored - start pending
 18135 Process '\''cf-mysql-broker'\''           not monitored - start pending
 18136 Process '\''quota-enforcer'\''            not monitored - start pending
 18137 Process '\''cc_uploader'\''               not monitored - start pending
 18138 Process '\''blobstore_nginx'\''           not monitored - start pending
 18139 Process '\''blobstore_url_signer'\''      not monitored - start pending
 18140 Process '\''bbs'\''                       running
 18141 Process '\''auctioneer'\''                not monitored - start pending
 18142 System '\''system_localhost'\''           not monitored - start pending'
 18143 ++ [[ The Monit daemon 5.2.5 uptime: 2m
…
Process 'pcfdev-api'                initializing
 24704 Process 'pcfdev-nginx'              initializing
 24705 Process 'vxlan-policy-agent'        initializing
 24706 Process 'uaa'                       running
 24707 Process 'tps_listener'              initializing
 24708 Process 'tps_watcher'               initializing
 24709 Process 'tcp_router'                initializing
 24710 Process 'tcp_emitter'               initializing
 24711 Process 'syslog_drain_binder'       initializing
 24712 Process 'syslog-configurator'       initializing - start pending
 24713 Process 'stager'                    not monitored - start pending
 24714 Process 'ssh_proxy'                 not monitored - start pending
 24715 Process 'silk-daemon'               not monitored - start pending
 24716 Process 'silk-controller'           not monitored - start pending
 24717 Process 'routing-api'               not monitored - start pending
 24718 Process 'route_registrar'           not monitored - start pending
 24719 Process 'route_emitter'             not monitored - start pending
 24720 Process 'rep'                       not monitored - start pending
 24721 Process 'rabbitmq-server'           not monitored - start pending
 24722 Process 'rabbitmq-broker'           not monitored - start pending
 24723 Process 'policy-server'             not monitored - start pending
 24724 Process 'nsync_listener'            not monitored - start pending
 24725 Process 'nsync_bulker'              not monitored - start pending
 24726 Process 'netmon'                    not monitored - start pending
 24727 Process 'nats'                      not monitored - start pending
 24728 Process 'mariadb_ctrl'              running
 24729 Process 'galera-healthcheck'        running
 24730 Process 'gra-log-purger-executable' not monitored - start pending
 24731 Process 'cluster_health_logger'     not monitored - start pending
 24732 Process 'metron_agent'              not monitored - start pending
 24733 Process 'loggregator_trafficcontroller' not monitored - start pending
 24734 Process 'localdriver'               not monitored - start pending
 24735 Process 'localbroker'               not monitored - start pending
 24736 Process 'gorouter'                  not monitored - start pending
 24737 Process 'garden'                    running
 24738 Process 'file_server'               not monitored - start pending
 24739 Process 'etcd'                      running
 24740 Process 'etcd_consistency_checker'  not monitored - start pending
 24741 Process 'doppler'                   not monitored - start pending
 24742 Process 'consul_agent'              running
 24743 Process 'cloud_controller_worker_1' not monitored - start pending
 24744 Process 'cloud_controller_ng'       not monitored - start pending
 24745 Process 'cloud_controller_worker_local_1' not monitored - start pending
 24746 Process 'cloud_controller_worker_local_2' not monitored - start pending
 24747 Process 'nginx_cc'                  not monitored - start pending
 24748 Process 'cloud_controller_clock'    not monitored - start pending
 24749 Process 'process-watcher'           not monitored - start pending
 24750 Process 'process-destroyer'         initializing - start pending
 24751 Process 'cf-redis-broker'           not monitored - start pending
 24752 Process 'broker-nginx'              not monitored - start pending
 24753 Process 'cf-mysql-broker'           not monitored - start pending
 24754 Process 'quota-enforcer'            not monitored - start pending
 24755 Process 'cc_uploader'               not monitored - start pending
 24756 Process 'blobstore_nginx'           not monitored - start pending
 24757 Process 'blobstore_url_signer'      not monitored - start pending
 24758 Process 'bbs'                       running
 24759 Process 'auctioneer'                not monitored - start pending
 24760 System 'system_localhost'           not monitored - start pending = *\e\r\r\o\r\ \c\o\n\n\e\c\t\i\n\g\ \t\o\ \t\h\e\ \m\o\n\i\t\ \d\a\e\m\o\n* ]]
 24761 ++ echo 'The Monit daemon 5.2.5 uptime: 2m
…
74827 Process '\''pcfdev-api'\''                running
 74828 Process '\''pcfdev-nginx'\''              running
 74829 Process '\''vxlan-policy-agent'\''        Does not exist
 74830 Process '\''uaa'\''                       running
 74831 Process '\''tps_listener'\''              running
 74832 Process '\''tps_watcher'\''               running
 74833 Process '\''tcp_router'\''                running
 74834 Process '\''tcp_emitter'\''               running
 74835 Process '\''syslog_drain_binder'\''       Execution failed
 74836 Process '\''syslog-configurator'\''       running
 74837 Process '\''stager'\''                    running
 74838 Process '\''ssh_proxy'\''                 running
 74839 Process '\''silk-daemon'\''               not monitored
 74840 Process '\''silk-controller'\''           initializing
 74841 Process '\''routing-api'\''               initializing
 74842 Process '\''route_registrar'\''           initializing
 74843 Process '\''route_emitter'\''             initializing
 74844 Process '\''rep'\''                       initializing
 74845 Process '\''rabbitmq-server'\''           initializing
 74846 Process '\''rabbitmq-broker'\''           initializing
 74847 Process '\''policy-server'\''             initializing
 74848 Process '\''nsync_listener'\''            initializing
 74849 Process '\''nsync_bulker'\''              initializing
 74850 Process '\''netmon'\''                    initializing
 74851 Process '\''nats'\''                      initializing
 74852 Process '\''mariadb_ctrl'\''              running
 74853 Process '\''galera-healthcheck'\''        running
 74854 Process '\''gra-log-purger-executable'\'' initializing
 74855 Process '\''cluster_health_logger'\''     initializing
 74856 Process '\''metron_agent'\''              initializing
 74857 Process '\''loggregator_trafficcontroller'\'' initializing
 74858 Process '\''localdriver'\''               initializing
 74859 Process '\''localbroker'\''               initializing
 74860 Process '\''gorouter'\''                  initializing
 74861 Process '\''garden'\''                    running
 74862 Process '\''file_server'\''               initializing
 74863 Process '\''etcd'\''                      running
 74864 Process '\''etcd_consistency_checker'\''  initializing
 74865 Process '\''doppler'\''                   initializing
 74866 Process '\''consul_agent'\''              running
 74867 Process '\''cloud_controller_worker_1'\'' initializing
 74868 Process '\''cloud_controller_ng'\''       initializing
 74869 Process '\''cloud_controller_worker_local_1'\'' initializing
 74870 Process '\''cloud_controller_worker_local_2'\'' initializing
 74871 Process '\''nginx_cc'\''                  initializing
 74872 Process '\''cloud_controller_clock'\''    initializing
 74873 Process '\''process-watcher'\''           initializing
 74874 Process '\''process-destroyer'\''         initializing
 74875 Process '\''cf-redis-broker'\''           initializing
 74876 Process '\''broker-nginx'\''              initializing
 74877 Process '\''cf-mysql-broker'\''           initializing
 74878 Process '\''quota-enforcer'\''            initializing
 74879 Process '\''cc_uploader'\''               initializing
 74880 Process '\''blobstore_nginx'\''           initializing
 74881 Process '\''blobstore_url_signer'\''      initializing
 74882 Process '\''bbs'\''                       running
 74883 Process '\''auctioneer'\''                initializing
 74884 System '\''system_localhost'\''           initializing'
 74885 ++ [[ The Monit daemon 5.2.5 uptime: 7m
…
90848 Process '\''pcfdev-api'\''                running
 90849 Process '\''pcfdev-nginx'\''              running
 90850 Process '\''vxlan-policy-agent'\''        running
 90851 Process '\''uaa'\''                       running
 90852 Process '\''tps_listener'\''              running
 90853 Process '\''tps_watcher'\''               running
 90854 Process '\''tcp_router'\''                running
 90855 Process '\''tcp_emitter'\''               running
 90856 Process '\''syslog_drain_binder'\''       not monitored
 90857 Process '\''syslog-configurator'\''       running
 90858 Process '\''stager'\''                    running
 90859 Process '\''ssh_proxy'\''                 running
 90860 Process '\''silk-daemon'\''               Does not exist
 90861 Process '\''silk-controller'\''           running
 90862 Process '\''routing-api'\''               running
 90863 Process '\''route_registrar'\''           Does not exist
 90864 Process '\''route_emitter'\''             Does not exist
 90865 Process '\''rep'\''                       Does not exist
 90866 Process '\''rabbitmq-server'\''           running
 90867 Process '\''rabbitmq-broker'\''           running
 90868 Process '\''policy-server'\''             running
 90869 Process '\''nsync_listener'\''            running
 90870 Process '\''nsync_bulker'\''              running
 90871 Process '\''netmon'\''                    running
 90872 Process '\''nats'\''                      running
 90873 Process '\''mariadb_ctrl'\''              running
 90874 Process '\''galera-healthcheck'\''        running
 90875 Process '\''gra-log-purger-executable'\'' running
 90876 Process '\''cluster_health_logger'\''     running
 90877 Process '\''metron_agent'\''              running
 90878 Process '\''loggregator_trafficcontroller'\'' running
 90879 Process '\''localdriver'\''               running
 90880 Process '\''localbroker'\''               running
 90881 Process '\''gorouter'\''                  running
 90882 Process '\''garden'\''                    running
 90883 Process '\''file_server'\''               running
 90884 Process '\''etcd'\''                      running
 90885 Process '\''etcd_consistency_checker'\''  running
 90886 Process '\''doppler'\''                   running
 90887 Process '\''consul_agent'\''              running
 90888 Process '\''cloud_controller_worker_1'\'' Does not exist
 90889 Process '\''cloud_controller_ng'\''       Does not exist
 90890 Process '\''cloud_controller_worker_local_1'\'' Does not exist
 90891 Process '\''cloud_controller_worker_local_2'\'' Does not exist
 90892 Process '\''nginx_cc'\''                  initializing
 90893 Process '\''cloud_controller_clock'\''    running
 90894 Process '\''process-watcher'\''           running
 90895 Process '\''process-destroyer'\''         running
 90896 Process '\''cf-redis-broker'\''           running
 90897 Process '\''broker-nginx'\''              running
 90898 Process '\''cf-mysql-broker'\''           Execution failed
 90899 Process '\''quota-enforcer'\''            initializing
 90900 Process '\''cc_uploader'\''               running
 90901 Process '\''blobstore_nginx'\''           running
 90902 Process '\''blobstore_url_signer'\''      running
 90903 Process '\''bbs'\''                       running
 90904 Process '\''auctioneer'\''                running
90905 System '\''system_localhost'\''           running'
 90906 ++ [[ The Monit daemon 5.2.5 uptime: 8m
…
672913 Process '\''pcfdev-api'\''                running
672914 Process '\''pcfdev-nginx'\''              running
672915 Process '\''vxlan-policy-agent'\''        running
672916 Process '\''uaa'\''                       running
672917 Process '\''tps_listener'\''              running
672918 Process '\''tps_watcher'\''               running
672919 Process '\''tcp_router'\''                running
672920 Process '\''tcp_emitter'\''               running
672921 Process '\''syslog_drain_binder'\''       not monitored
672922 Process '\''syslog-configurator'\''       running
672923 Process '\''stager'\''                    running
672924 Process '\''ssh_proxy'\''                 running
672925 Process '\''silk-daemon'\''               running
672926 Process '\''silk-controller'\''           running
672927 Process '\''routing-api'\''               running
672928 Process '\''route_registrar'\''           running
672929 Process '\''route_emitter'\''             running
672930 Process '\''rep'\''                       running
672931 Process '\''rabbitmq-server'\''           running
672932 Process '\''rabbitmq-broker'\''           running
672933 Process '\''policy-server'\''             running
672934 Process '\''nsync_listener'\''            running
672935 Process '\''nsync_bulker'\''              running
672936 Process '\''netmon'\''                    running
672937 Process '\''nats'\''                      running
672938 Process '\''mariadb_ctrl'\''              running
672939 Process '\''galera-healthcheck'\''        running
672940 Process '\''gra-log-purger-executable'\'' running
672941 Process '\''cluster_health_logger'\''     running
672942 Process '\''metron_agent'\''              running
672943 Process '\''loggregator_trafficcontroller'\'' running
672944 Process '\''localdriver'\''               running
672945 Process '\''localbroker'\''               running
672946 Process '\''gorouter'\''                  running
672947 Process '\''garden'\''                    running
672948 Process '\''file_server'\''               running
672949 Process '\''etcd'\''                      running
672950 Process '\''etcd_consistency_checker'\''  running
672951 Process '\''doppler'\''                   running
672952 Process '\''consul_agent'\''              running
672953 Process '\''cloud_controller_worker_1'\'' running
672954 Process '\''cloud_controller_ng'\''       running
672955 Process '\''cloud_controller_worker_local_1'\'' running
672956 Process '\''cloud_controller_worker_local_2'\'' running
672957 Process '\''nginx_cc'\''                  running
672958 Process '\''cloud_controller_clock'\''    running
672959 Process '\''process-watcher'\''           running
672960 Process '\''process-destroyer'\''         running
672961 Process '\''cf-redis-broker'\''           running
672962 Process '\''broker-nginx'\''              running
672963 Process '\''cf-mysql-broker'\''           running
672964 Process '\''quota-enforcer'\''            running
672965 Process '\''cc_uploader'\''               running
672966 Process '\''blobstore_nginx'\''           running
672967 Process '\''blobstore_url_signer'\''      running
672968 Process '\''bbs'\''                       running
672969 Process '\''auctioneer'\''                running
672970 System '\''system_localhost'\''           running'
672971 + started=57
672972 + [[ 57 -lt 58 ]]
672973 + counter=3392
672974 + [[ 32 = 0 ]]
672975 + sleep 1

/var/vcap/monit/monit.log

[UTC Feb 19 08:00:33] info     : start service 'syslog_drain_binder' on user request
[UTC Feb 19 08:00:33] info     : monit daemon at 4562 awakened
…
[UTC Feb 19 08:01:06] info     : 'syslog_drain_binder' start: /var/vcap/jobs/syslog_drain_binder/bin/syslog_drain_binder_ctl
[UTC Feb 19 08:01:06] info     : 'syslog_drain_binder' start action done
…
[UTC Feb 19 08:06:53] error    : 'syslog_drain_binder' process is not running
[UTC Feb 19 08:06:53] info     : 'syslog_drain_binder' trying to restart
[UTC Feb 19 08:06:53] info     : 'syslog_drain_binder' start: /var/vcap/jobs/syslog_drain_binder/bin/syslog_drain_binder_ctl
[UTC Feb 19 08:07:23] error    : 'syslog_drain_binder' failed to start
…
UTC Feb 19 10:55:46] error    : 'syslog_drain_binder' process is not running
[UTC Feb 19 10:55:46] info     : 'syslog_drain_binder' trying to restart
[UTC Feb 19 10:55:46] info     : 'syslog_drain_binder' start: /var/vcap/jobs/syslog_drain_binder/bin/syslog_drain_binder_ctl
[UTC Feb 19 10:56:16] error    : 'syslog_drain_binder' failed to start

var/vcap/sys/log/syslog_drain_binder/syslog_drain_binder.log

2018/02/19 08:01:07 Starting pprof server on: 127.0.0.1:39516
panic: x509: certificate has expired or is not yet valid

goroutine 1 [running]:
panic(0xa34b80, 0xc4201c8190)
        /var/vcap/data/packages/golang1.7/c5de560ab1578122cd22032249fb7f1f287768d4/src/runtime/panic.go:500 +0x1a1
main.main()
        /var/vcap/data/compile/syslog_drain_binder/loggregator/src/syslog_drain_binder/main.go:55 +0xbd5
2018/02/19 08:04:59 Starting pprof server on: 127.0.0.1:43004
panic: x509: certificate has expired or is not yet valid

vcap@agent-id-pcfdev-0:/var/vcap/jobs/syslog_drain_binder/config/certs$ ls
etcd-ca.crt  etcd-client.crt  etcd-client.key  mutual_tls_ca.crt  syslogdrainbinder.crt  syslogdrainbinder.key

vcap@agent-id-pcfdev-0:/var/vcap/jobs/syslog_drain_binder/config/certs$  openssl x509 -in syslogdrainbinder.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 16412547589211907925 (0xe3c51513322b6755)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: CN=*.service.cf.internal, C=US, ST=New-York, L=New York, O=Pivotal, OU=PCF Dev/emailAddress=pcfdev-eng@pivotal.io
        Validity
            Not Before: Feb 15 22:35:39 2017 GMT
            Not After : Feb 15 22:35:39 2018 GMT
        Subject: CN=*.service.cf.internal, C=US, ST=New-York, L=New York, O=Pivotal, OU=PCF Dev/emailAddress=pcfdev-eng@pivotal.io
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c3:8f:42:50:89:9e:27:64:f1:72:c3:d1:3c:ac:
                    19:44:52:27:7e:41:04:26:8a:3c:65:0b:54:45:a4:
                    94:ee:67:cc:fe:bc:27:6d:67:63:fd:68:50:47:67:
                    03:57:79:70:98:76:ed:4d:83:ea:06:7a:3c:43:7b:
                    46:c6:53:9a:9c:a0:14:e4:10:6b:16:dc:f0:82:b1:
                    f3:a6:9c:90:8e:87:0a:ad:d6:c2:20:c0:f7:62:09:
                    30:cb:de:74:fa:c6:59:43:68:1b:a8:7f:a8:0c:22:
                    44:2a:37:4d:d5:7a:d7:37:f2:d2:6a:81:77:16:b5:
                    5f:46:a5:81:44:0e:64:25:07:93:a4:7e:0d:16:c2:
                    4e:15:d0:c9:95:bb:30:dc:9f:9c:d6:f5:b9:bc:34:
                    dd:e1:e9:79:24:f1:32:f6:26:d2:4f:fe:11:b6:2d:
                    85:e6:35:87:f3:09:92:58:9d:10:4c:13:c3:9c:a0:
                    c3:c2:9c:c6:8c:1c:08:58:dd:26:36:aa:56:03:e1:
                    35:90:8d:4e:e0:3a:19:4a:c8:40:9f:61:07:f2:87:
                    a4:79:a6:89:e8:1e:de:97:aa:e5:14:19:83:f1:1f:
                    a0:35:2d:45:b5:5d:e5:5d:73:ee:df:b6:3b:e2:20:
                    05:31:0a:05:8e:72:ce:be:12:47:12:bd:b5:d1:c4:
                    9f:e3
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                51:52:F9:01:4D:9F:A3:7F:46:A8:F3:95:6E:AC:D3:10:BF:64:09:75
            X509v3 Authority Key Identifier:
                keyid:51:52:F9:01:4D:9F:A3:7F:46:A8:F3:95:6E:AC:D3:10:BF:64:09:75
                DirName:/CN=*.service.cf.internal/C=US/ST=New-York/L=New York/O=Pivotal/OU=PCF Dev/emailAddress=pcfdev-eng@pivotal.io
                serial:E3:C5:15:13:32:2B:67:55

            X509v3 Basic Constraints:
                CA:TRUE
    Signature Algorithm: sha1WithRSAEncryption
         64:d7:d2:77:c9:f3:1b:d0:31:02:fc:10:43:94:75:32:1a:fb:
         ec:2e:bd:0f:55:b6:46:b5:84:fb:ab:11:e4:a8:e4:01:17:3d:
         64:ab:ae:5a:78:db:93:f0:3d:06:48:cc:ba:81:f1:7a:e1:c4:
         4f:a7:70:3b:6b:c3:5f:79:8b:12:a3:29:ee:7c:c1:cc:68:c2:
         84:ce:56:23:4b:23:46:c0:37:31:c9:1c:2f:79:0e:f0:e1:07:
         97:f4:49:d1:08:17:ff:25:07:42:f5:b2:08:b3:d2:28:24:08:
         0d:1e:11:19:1e:7c:dd:9e:de:fa:35:7d:61:27:ec:3c:49:54:
         e0:8b:9c:dd:42:49:eb:bf:ec:2b:a0:a6:d2:b4:ac:b6:47:9a:
         b1:9d:6d:5b:3e:0b:7a:02:d3:28:13:73:3f:65:48:fe:0c:d4:
         2f:53:7c:04:71:36:ac:5a:ec:6f:0d:43:a1:65:fa:0e:33:44:
         5d:53:1a:47:29:0a:7e:e8:f3:ee:80:4e:7d:8b:fc:60:fd:59:
         b9:dc:5a:c0:43:af:b8:4e:7e:b0:e6:38:ed:58:1b:33:1f:c3:
         b1:16:4c:83:fa:62:a1:2a:c8:7f:0e:47:d5:91:7b:e7:6e:71:
         d1:99:a1:2c:40:16:e7:5a:48:72:80:3c:2c:dd:2d:58:4f:cf:
         ef:4d:90:70
vcap@agent-id-pcfdev-0:/var/vcap/jobs/syslog_drain_binder/config/certs$
sjudeng commented 6 years ago

Looks like this is a duplicate of #293

brennerm commented 6 years ago

Yes that's true. You can close this issue if you want.

ssisil commented 6 years ago

Fixed in v0.29