cloudfoundry-attic / bosh-init

bosh-init is a tool used to create and update the Director VM
Apache License 2.0
31 stars 33 forks source link

Compatibility with bosh-agent interface. #34

Closed allomov closed 9 years ago

allomov commented 9 years ago

Hello!

I've just had a following error:

...
[unlimitedRetryStrategy] 2015/06/04 11:39:23 DEBUG - Making attempt #131
[httpClient] 2015/06/04 11:39:23 DEBUG - Sending POST request with body {"method":"get_task","arguments":["4c1672e6-83b6-483a-67fa-646c253587c8"],"reply_to":"9552ca76-f43d-42c5-5441-afc7d05fd874"}, endpoint https://mbus:mbus-password@140.211.168.54:6868/agent
[httpAgentClient] 2015/06/04 11:39:23 DEBUG - get_task response value: map[string]interface {}{"agent_task_id":"4c1672e6-83b6-483a-67fa-646c253587c8", "state":"running"}
[unlimitedRetryStrategy] 2015/06/04 11:39:24 DEBUG - Making attempt #132
[httpClient] 2015/06/04 11:39:24 DEBUG - Sending POST request with body {"method":"get_task","arguments":["4c1672e6-83b6-483a-67fa-646c253587c8"],"reply_to":"9552ca76-f43d-42c5-5441-afc7d05fd874"}, endpoint https://mbus:mbus-password@140.211.168.54:6868/agent
[httpAgentClient] 2015/06/04 11:39:24 DEBUG - get_task response value: map[string]interface {}{"result":map[string]interface {}{"sha1":"7338352c8d3ceb1c53e369fac8cfebcf0a604ccd", "blobstore_id":"9fee0327-b6cc-45ae-73d1-e51856afd56f"}}
 Failed (00:02:14)
Failed deploying (00:04:43)

Stopping registry...[registryServer] 2015/06/04 11:39:24 DEBUG - Stopping registry server
 Finished (00:00:00)
[File System] 2015/06/04 11:39:24 DEBUG - Remove all /tmp/stemcell-manager032002003
[registryServer] 2015/06/04 11:39:24 DEBUG - Registry error occurred: accept tcp 127.0.0.1:6901: use of closed network connection
[File System] 2015/06/04 11:39:24 DEBUG - Remove all /tmp/bosh-init-release988459833
[File System] 2015/06/04 11:39:25 DEBUG - Remove all /tmp/bosh-init-release880177476
[main] 2015/06/04 11:39:25 ERROR - Command 'deploy' failed: Deploying: Building state for instance 'bosh/0': Compiling job package dependencies for instance 'bosh/0': Compiling job package dependencies: Remotely compiling package 'ruby_openstack_cpi' with the agent: Unable to parse 'compile_package' response from the agent: map[string]interface {}(nil)

Command 'deploy' failed:
  Deploying:
    Building state for instance 'bosh/0':
      Compiling job package dependencies for instance 'bosh/0':
        Compiling job package dependencies:
          Remotely compiling package 'ruby_openstack_cpi' with the agent:
            Unable to parse 'compile_package' response from the agent: map[string]interface {}(nil)

As far as I can see there was no errors on bosh-agent side and ruby_openstack_cpi was successfully compiled. I can find a blob with compiled ruby_openstack_cpi package on the following path: /var/vcap/micro_bosh/data/cache/9fee0327-b6cc-45ae-73d1-e51856afd56f. Still this error means that bosh-init is not satisfied with a response from bosh-agent. I use bosh-init with v0.0.43 and bosh-agent from stemcell 2905. I have another project where bosh-init of an older version works correctly with stemcell version 2826.

Is it possible that interface of bosh-agent is incompatible with some versions of bosh-init? Do you have a track of such dependencies.

Thank you, Alex L.

ghost commented 9 years ago

We are fully backwards compatible in all cases. I am going to try to reproduce this.

On Thu, Jun 4, 2015 at 5:57 AM, Alexander Lomov notifications@github.com wrote:

Hello!

I've just had a following error:

... [unlimitedRetryStrategy] 2015/06/04 11:39:23 DEBUG - Making attempt #131 [httpClient] 2015/06/04 11:39:23 DEBUG - Sending POST request with body {"method":"get_task","arguments":["4c1672e6-83b6-483a-67fa-646c253587c8"],"reply_to":"9552ca76-f43d-42c5-5441-afc7d05fd874"}, endpoint https://mbus:mbus-password@140.211.168.54:6868/agent [httpAgentClient] 2015/06/04 11:39:23 DEBUG - get_task response value: map[string]interface {}{"agent_task_id":"4c1672e6-83b6-483a-67fa-646c253587c8", "state":"running"} [unlimitedRetryStrategy] 2015/06/04 11:39:24 DEBUG - Making attempt #132 [httpClient] 2015/06/04 11:39:24 DEBUG - Sending POST request with body {"method":"get_task","arguments":["4c1672e6-83b6-483a-67fa-646c253587c8"],"reply_to":"9552ca76-f43d-42c5-5441-afc7d05fd874"}, endpoint https://mbus:mbus-password@140.211.168.54:6868/agent [httpAgentClient] 2015/06/04 11:39:24 DEBUG - get_task response value: map[string]interface {}{"result":map[string]interface {}{"sha1":"7338352c8d3ceb1c53e369fac8cfebcf0a604ccd", "blobstore_id":"9fee0327-b6cc-45ae-73d1-e51856afd56f"}} Failed (00:02:14) Failed deploying (00:04:43)

Stopping registry...[registryServer] 2015/06/04 11:39:24 DEBUG - Stopping registry server Finished (00:00:00) [File System] 2015/06/04 11:39:24 DEBUG - Remove all /tmp/stemcell-manager032002003 [registryServer] 2015/06/04 11:39:24 DEBUG - Registry error occurred: accept tcp 127.0.0.1:6901: use of closed network connection [File System] 2015/06/04 11:39:24 DEBUG - Remove all /tmp/bosh-init-release988459833 [File System] 2015/06/04 11:39:25 DEBUG - Remove all /tmp/bosh-init-release880177476 [main] 2015/06/04 11:39:25 ERROR - Command 'deploy' failed: Deploying: Building state for instance 'bosh/0': Compiling job package dependencies for instance 'bosh/0': Compiling job package dependencies: Remotely compiling package 'ruby_openstack_cpi' with the agent: Unable to parse 'compile_package' response from the agent: map[string]interface {}(nil)

Command 'deploy' failed: Deploying: Building state for instance 'bosh/0': Compiling job package dependencies for instance 'bosh/0': Compiling job package dependencies: Remotely compiling package 'ruby_openstack_cpi' with the agent: Unable to parse 'compile_package' response from the agent: map[string]interface {}(nil)

As far as I can see there was no errors on bosh-agent side and ruby_openstack_cpi was successfully compiled. I can find a blob with compiled ruby_openstack_cpi package on the following path: /var/vcap/micro_bosh/data/cache/9fee0327-b6cc-45ae-73d1-e51856afd56f. Still this error means that bosh-init is not satisfied with a response from bosh-agent. I use bosh-init with v0.0.43 and bosh-agent from stemcell 2905. I have another project where bosh-init of an older version works correctly with stemcell version 2826.

Is it possible that interface of bosh-agent is incompatible with some versions of bosh-init? Do you have a track of such dependencies.

Thank you, Alex L.

— Reply to this email directly or view it on GitHub https://github.com/cloudfoundry/bosh-init/issues/34.

allomov commented 9 years ago

Possibly error is in another place. I've just tried to run bosh-init with v2915 and had the same result.

allomov commented 9 years ago

When I tried bosh-init v0.0.50 the output is slightly different:

[unlimitedRetryStrategy] 2015/06/04 18:34:44 DEBUG - Making attempt #14
[httpClient] 2015/06/04 18:34:44 DEBUG - Sending POST request with body {"method":"get_task","arguments":["1d43fa11-e930-4d30-40d1-44cd87a9c22f"],"reply_to":"48883f78-d089-458c-5395-ec6bcb568f43"}, endpoint https://mbus:mbus-password@x.x.x.x:6868/agent
 Failed (00:00:16)
Failed deploying (00:03:19)

Stopping registry...[registryServer] 2015/06/04 18:34:44 DEBUG - Stopping registry server
 Finished (00:00:00)
Cleaning up rendered CPI jobs...[File System] 2015/06/04 18:34:44 DEBUG - Remove all /home/ubuntu/.bosh_init/installations/4374e931-c8c0-4319-5ee7-bb9d6e8c2175/jobs/cpi
 Finished (00:00:00)
[File System] 2015/06/04 18:34:44 DEBUG - Remove all /tmp/stemcell-manager683436338
[registryServer] 2015/06/04 18:34:44 DEBUG - Registry error occurred: accept tcp 127.0.0.1:6901: use of closed network connection
[File System] 2015/06/04 18:34:44 DEBUG - Remove all /tmp/bosh-init-release943500288
[File System] 2015/06/04 18:34:44 DEBUG - Remove all /tmp/bosh-init-release938202463
[main] 2015/06/04 18:34:44 ERROR - Command 'deploy' failed: Deploying: Building state for instance 'bosh/0': Compiling job package dependencies for instance 'bosh/0': Compiling job package dependencies: Remotely compiling package 'ruby_openstack_cpi' with the agent: Sending 'compile_package' to the agent: Sending 'get_task' to the agent: Performing request to agent endpoint 'https://mbus:mbus-password@x.x.x.x:6868/agent': Performing POST request: Post https://mbus:mbus-password@x.x.x.x:6868/agent: read tcp x.x.x.x:6868: connection reset by peer

Command 'deploy' failed:
  Deploying:
    Building state for instance 'bosh/0':
      Compiling job package dependencies for instance 'bosh/0':
        Compiling job package dependencies:
          Remotely compiling package 'ruby_openstack_cpi' with the agent:
            Sending 'compile_package' to the agent:
              Sending 'get_task' to the agent:
                Performing request to agent endpoint 'https://mbus:mbus-password@x.x.x.x:6868/agent':
                  Performing POST request:
                    Post https://mbus:mbus-password@x.x.x.x:6868/agent: read tcp x.x.x.x:6868: connection reset by peer
cppforlife commented 9 years ago

How are you using 2905? It had the problems with ssh (ssh: handshake failed: ssh: no common algorithms). Are you building your own stemcells? Which sha of bosh-agent are you using?

allomov commented 9 years ago

@cppforlife I just needed to update sshd_config for 2905 stemcell. We build our own stemcells to use them with power architecture.

At this moment we use this version of bosh-agent https://github.com/cloudfoundry/bosh-agent/tree/348a0149c1ee34988f25676ce17f07911b970320, but I'm going to update it and test with a newer one.

allomov commented 9 years ago

I also needed to update bosh-openstack-cpi-release to use ignore_server_availability_zone option. Here is a PR with this changes https://github.com/cloudfoundry-incubator/bosh-openstack-cpi-release/pull/7. But this shouldn't impact bosh-openstack-cpi-release compilation, because bosh-init already compiled it on ubuntu trusty.

allomov commented 9 years ago

https://github.com/cloudfoundry/bosh-init/pull/35 fixes this issue.