Open jhiemer opened 9 years ago
Next step, looking into gateways 0 I see:
root@8d482271-7d3c-4b4c-9651-65ce5d63f6d5:/var/vcap/bosh_ssh/bosh_en9u1iqap# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 172.24.102.157:51794 0.0.0.0:* LISTEN 17353/ruby
tcp 0 0 127.0.0.1:33331 0.0.0.0:* LISTEN 782/bosh-agent
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1347/sshd
tcp 0 0 0.0.0.0:57792 0.0.0.0:* LISTEN 624/rpc.statd
tcp 0 0 127.0.0.1:2822 0.0.0.0:* LISTEN 1466/monit
tcp 0 0 172.24.102.157:59401 0.0.0.0:* LISTEN 17343/ruby
tcp 0 0 127.0.0.1:2825 0.0.0.0:* LISTEN 782/bosh-agent
tcp 0 0 172.24.102.157:43722 0.0.0.0:* LISTEN 17363/ruby
tcp 0 0 172.24.102.157:50474 0.0.0.0:* LISTEN 17333/ruby
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 611/rpcbind
tcp6 0 0 :::22 :::* LISTEN 1347/sshd
tcp6 0 0 :::36254 :::* LISTEN 624/rpc.statd
tcp6 0 0 :::111 :::* LISTEN 611/rpcbind
udp 0 0 0.0.0.0:39395 0.0.0.0:* 624/rpc.statd
udp 0 0 0.0.0.0:111 0.0.0.0:* 611/rpcbind
udp 0 0 0.0.0.0:761 0.0.0.0:* 611/rpcbind
udp 0 0 127.0.0.1:800 0.0.0.0:* 624/rpc.statd
udp6 0 0 :::35329 :::* 624/rpc.statd
udp6 0 0 :::111 :::* 611/rpcbind
udp6 0 0 :::761 :::* 611/rpcbind
Comparing to the ports in the database:
"http://172.24.102.157:43722"
"http://172.24.102.157:59401"
"http://172.24.102.157:51794"
"http://172.24.102.157:50474"
So they are completely different. What's also interesting is that, watch monit summary
does consistently show running
then Does not exist
.
The Monit daemon 5.2.4 uptime: 21h 58m
Process 'mongodb_gateway' running
Process 'rabbit_gateway' Does not exist
Process 'postgresql_gateway' running
Process 'redis_gateway' running
System 'system_8d482271-7d3c-4b4c-9651-65ce5d63f6d5' running
The question is, if the Jobs are failing? That would explain, why the ports are changing that often in the database.
I will keep debugging.
Ok, started the mongodb_gateway by hand and I could see the ruby process being started:
root@8d482271-7d3c-4b4c-9651-65ce5d63f6d5:/var/vcap/sys/log/mongodb_gateway# top | grep ruby
20046 root 20 0 89612 27332 7968 R 19.3 2.7 0:00.58 ruby
20046 root 20 0 222160 57368 11104 S 71.1 5.6 0:02.72 ruby
20046 root 20 0 223672 59204 11388 S 8.6 5.8 0:02.98 ruby
20046 root 20 0 223800 59248 11388 S 0.3 5.8 0:02.99 ruby
20046 root 20 0 223928 59288 11388 S 0.3 5.8 0:03.00 ruby
20046 root 20 0 223928 59288 11388 S 0.3 5.8 0:03.01 ruby
20046 root 20 0 223928 59288 11388 S 0.3 5.8 0:03.02 ruby
Running for a few seconds, and then they are gone.
And completely manually starting results in the same behaviour:
root@8d482271-7d3c-4b4c-9651-65ce5d63f6d5:/var/vcap/packages/mongodb_gateway/services/ng/mongodb/bin# /var/vcap/packages/mongodb_gateway/services/ng/mongodb/bin/mongodb_gateway -c /var/vcap/jobs/mongodb_gateway/config/mongodb_gateway.yml
Resque::Helpers will be gone with no replacement in Resque 2.0.0.
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:13: warning: already initialized constant HTTP_BAD_REQUEST
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:14: warning: already initialized constant HTTP_NOT_AUTHORIZED
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:15: warning: already initialized constant HTTP_FORBIDDEN
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:16: warning: already initialized constant HTTP_NOT_FOUND
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:17: warning: already initialized constant HTTP_INTERNAL
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:18: warning: already initialized constant HTTP_NOT_IMPLEMENTED
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:19: warning: already initialized constant HTTP_SERVICE_UNAVAIL
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:20: warning: already initialized constant HTTP_GATEWAY_TIMEOUT
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:29: warning: already initialized constant INVALID_CONTENT
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:30: warning: already initialized constant MALFORMATTED_REQ
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:31: warning: already initialized constant UNKNOWN_PLAN_UNIQUE_ID
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:32: warning: already initialized constant UNKNOWN_PLAN
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:33: warning: already initialized constant UNSUPPORTED_VERSION
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:36: warning: already initialized constant NOT_AUTHORIZED
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:41: warning: already initialized constant NOT_FOUND
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:44: warning: already initialized constant INTERNAL_ERROR
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:45: warning: already initialized constant EXTENSION_NOT_IMPL
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:46: warning: already initialized constant NODE_OPERATION_TIMEOUT
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:47: warning: already initialized constant SERVICE_START_TIMEOUT
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:48: warning: already initialized constant WARDEN_RUN_COMMAND_FAILURE
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:51: warning: already initialized constant SERVICE_UNAVAILABLE
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:54: warning: already initialized constant GATEWAY_TIMEOUT
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:57: warning: already initialized constant OVER_QUOTA
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:58: warning: already initialized constant JOB_QUEUE_TIMEOUT
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:59: warning: already initialized constant JOB_TIMEOUT
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:60: warning: already initialized constant BAD_SERIALIZED_DATAFILE
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:61: warning: already initialized constant FILESIZE_TOO_LARGE
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:62: warning: already initialized constant TOO_MANY_REDIRECTS
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:63: warning: already initialized constant FILE_CORRUPTED
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:64: warning: already initialized constant REDIS_CONCURRENT_UPDATE
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:65: warning: already initialized constant INVALID_SNAPSHOT_NAME
Thin web server (v1.6.1 codename Death Proof)
Maximum connections set to 1024
Listening on 172.24.102.157:33414, CTRL+C to stop
root@8d482271-7d3c-4b4c-9651-65ce5d63f6d5:/var/vcap/packages/mongodb_gateway/services/ng/mongodb/bin#
Now I need to think about the next steps. :-)
So I can still see the service brokers stopping continuously, while the services itself a running without any problems. Is there any crash log, where I can see, why the service stopped working?
mmm. We also met some instability with this release. No community effort on cf-services-contrib anyway. We had good success with https://github.com/cf-platform-eng/cf-containers-broker, as an alternative, provisionning mongo database as docker container.
Yeah the docker-boshrelease or its spinoffs like postgresql-docker-boshrelease and logstash-docker-boshrelease have been my go-to for some quick primitive services.
On Thu, Sep 10, 2015 at 11:25 AM, Pierre Oblin notifications@github.com wrote:
mmm. We also met some instability with this release. No community effort on cf-services-contrib anyway.
We had good success with https://github.com/cf-platform-eng/cf-containers-broker, as an alternative, provisionning mongo database as docker container.
Reply to this email directly or view it on GitHub: https://github.com/cloudfoundry-community/cf-services-contrib-release/issues/156#issuecomment-139335454
I have v212 deployed with the specified diego release and with service-contrib-release 6. The deployment runs through smoothly, but when I try to create a service I get:
Looking into the api_z1 I can see:
On the gateway I don't see the ports mapped. Do they change regularly? How is the updating of the ports done?