Fail to install any components on client node

[JianlanWang]

The issue just seems like Issue #20. I have two nodes: 192.168.3.1 (puppetserver.dodai.com) for dodai server and 192.168.3.2(clientnode.dodai.com) for dodai client node. I saw the proposal's status has been changed to "installed". But nothing happened on client node. Here's some info for your reference. Thanks a lot.

dodai-deploy/log/job_server.log:

[#<MCollective::RPC::Result:0x7fc47449f3f0 @agent="puppetd", @action="runonce", @results={:sender=>"clientnode.dodai.com", :data=>{:output=>"\e[0;32minfo: Caching catalog for clientnode.dodai.com\e[0m\n\e[0;32minfo: Applying configuration version '1346037947'\e[0m\n\e[0;36mnotice: Finished catalog run in 0.02 seconds\e[0m\n"}, :statuscode=>0, :statusmsg=>"OK"}>] --- !ruby/object:MCollective::RPC::Result action: runonce agent: puppetd results: :sender: clientnode.dodai.com :data: :output: "\e[0;32minfo: Caching catalog for clientnode.dodai.com\e[0m\n\ \e[0;32minfo: Applying configuration version '1346037947'\e[0m\n\ \e[0;36mnotice: Finished catalog run in 0.02 seconds\e[0m\n" :statuscode: 0 :statusmsg: OK install[proposal - 2] finished

dodai-deploy/log/development.log:

Started GET "/proposals" for 10.239.37.23 at Mon Aug 27 01:56:05 -0400 2012 Processing by ProposalsController#index as ^[[1m^[[36mProposal Load (0.5ms)^[[0m ^[[1mSELECT "proposals".* FROM "proposals"^[[0m ^[[1m^[[35mSoftware Load (0.2ms)^[[0m SELECT "softwares".* FROM "softwares" WHERE "softwares"."id" = 4 LIMIT 1 Rendered layouts/_menu.html.erb (2.1ms) Rendered proposals/index.html.erb within layouts/application (33.1ms) Completed 200 OK in 120ms (Views: 98.5ms | ActiveRecord: 0.7ms) ^[[1m^[[35mNode Load (0.3ms)^[[0m SELECT "nodes".* FROM "nodes" WHERE "nodes"."name" = 'clientnode.dodai.com' LIMIT 1 ^[[1m^[[36mProposal Load (0.2ms)^[[0m ^[[1mSELECT "proposals".* FROM "proposals" WHERE "proposals"."id" = 2 LIMIT 1^[[0m ^[[1m^[[35mNode Load (0.2ms)^[[0m SELECT "nodes".* FROM "nodes" WHERE "nodes"."id" = 1 LIMIT 1 ^[[1m^[[36mAREL (0.2ms)^[[0m ^[[1mINSERT INTO "logs" ("content", "node_id", "updated_at", "operation", "created_at", "proposal_id") VALUES ('info: Caching catalog for clientnode.dodai.com info: Applying configuration version ''1346037947'' notice: Finished catalog run in 0.02 seconds ', 1, '2012-08-27 05:56:06.235928', 'install', '2012-08-27 05:56:06.235928', 2)^[[0m ^[[1m^[[35mAREL (0.2ms)^[[0m UPDATE "node_configs" SET "updated_at" = '2012-08-27 05:56:06.362063', "state" = 'installed' WHERE "node_configs"."id" = 2 ^[[1m^[[36mSQL (0.1ms)^[[0m ^[[1mSELECT 1 FROM "proposals" WHERE ("proposals"."name" = 'Glance 2') AND ("proposals".id <> 2) LIMIT 1^[[0m ^[[1m^[[35mAREL (0.2ms)^[[0m UPDATE "proposals" SET "updated_at" = '2012-08-27 05:56:06.482153', "state" = 'installed' WHERE "proposals"."id" = 2

Started GET "/proposals" for 10.239.37.23 at Mon Aug 27 01:56:06 -0400 2012 Processing by ProposalsController#index as ^[[1m^[[36mProposal Load (0.5ms)^[[0m ^[[1mSELECT "proposals".* FROM "proposals"^[[0m ^[[1m^[[35mSoftware Load (0.2ms)^[[0m SELECT "softwares".* FROM "softwares" WHERE "softwares"."id" = 4 LIMIT 1 Rendered layouts/_menu.html.erb (2.1ms) Rendered proposals/index.html.erb within layouts/application (32.6ms) Completed 200 OK in 119ms (Views: 35.1ms | ActiveRecord: 0.7ms)

[guanxiaohua2k6]

Could you paste the contents of /var/log/puppet/*.yml and the contents of /etc/hosts in both nodes?

[JianlanWang]

Actually there's no *yml in /var/log/puppet. I commented out the line of "node_terminus=exec" in /etc/puppet/puppet.conf. Because puppet client would report one error of "Could not find node "clientnode.dodai.com", cannot compile" while I enabled "node_terminux=exec". I don't know why. And also is it OK that I used root user to complete all operations? Here's /etc/hosts for your reference:

For puppetserver.dodai.com: 127.0.0.1 puppetserver.dodai.com puppetserver localhost 127.0.1.1 puppetserver.dodai.com puppetserver 192.168.3.1 puppetserver.dodai.com puppetserver 192.168.3.2 clientnode.dodai.com clientnode

For clientnode.dodai.com: 127.0.0.1 localhost 127.0.1.1 clientnode.dodai.com clientnode 192.168.3.2 clientnode.dodai.com clientnode 192.168.3.1 puppetserver.dodai.com puppetserver

[guanxiaohua2k6]

Hi, Jianlan,

Firstly you should enable "node_terminux=exec". It is necessary for dodai-deploy.

As to the error message, "Could not find node "clientnode.dodai.com", cannot compile", it is possibly caused by the error on dodai-deploy server when processing request from puppet server's external node script. So could you try again, and paste the log in file dodai-deploy/log/development.log?

[JianlanWang]

Here're some info in /dodai-deploy/log/development.log, but which was generated while starting job server not for enablling "node_terminux=exec".

DEPRECATION WARNING: config.action_view.debug_rjs will be removed in 3.1, from 3.1 onwards you will need to install prototype-rails to continue to use RJS templates . (called from /root/dodai-deploy/config/environment.rb:5) DEPRECATION WARNING: config.action_view.debug_rjs will be removed in 3.1, from 3.1 onwards you will need to install prototype-rails to continue to use RJS templates . (called from /root/dodai-deploy/config/environment.rb:5) DEPRECATION WARNING: config.action_view.debug_rjs will be removed in 3.1, from 3.1 onwards you will need to install prototype-rails to continue to use RJS templates . (called from /root/dodai-deploy/config/environment.rb:5) DEPRECATION WARNING: config.action_view.debug_rjs will be removed in 3.1, from 3.1 onwards you will need to install prototype-rails to continue to use RJS templates . (called from /root/dodai-deploy/config/environment.rb:5) DEPRECATION WARNING: config.action_view.debug_rjs will be removed in 3.1, from 3.1 onwards you will need to install prototype-rails to continue to use RJS templates . (called from /root/dodai-deploy/config/environment.rb:5)

[guanxiaohua2k6]

The messages are OK.

[JianlanWang]

More info in /dodai-deploy/log/development.log when I was trying "puppetd --test --server puppetserver.dodai.com" with "node_terminux=exec" enabling.

Started GET "/node_configs/clientnode.dodai.com/puppet.json?operation=install&proposal_id=1" for 127.0.0.1 at Tue Aug 28 03:52:31 -0400 2012 Processing by NodeConfigsController#puppet as JSON Parameters: {"operation"=>"install", "node_name"=>"clientnode.dodai.com", "proposal_id"=>"1"} Proposal Load (0.1ms) SELECT "proposals".* FROM "proposals" WHERE "proposals"."id" = 1 LIMIT 1 Node Load (0.1ms) SELECT "nodes".* FROM "nodes" WHERE "nodes"."name" = 'clientnode.dodai.com' LIMIT 1 Software Load (0.2ms) SELECT "softwares".* FROM "softwares" WHERE "softwares"."id" = 4 LIMIT 1 Completed 500 Internal Server Error in 124ms

NoMethodError (You have a nil object when you didn't expect it! You might have expected an instance of Array. The error occurred while evaluating nil.each): app/controllers/node_configs_controller.rb:47:in puppet' app/controllers/node_configs_controller.rb:42:inpuppet'

Rendered /var/lib/gems/1.8/gems/actionpack-3.0.9/lib/action_dispatch/middleware/templates/rescues/_trace.erb (1.0ms) Rendered /var/lib/gems/1.8/gems/actionpack-3.0.9/lib/action_dispatch/middleware/templates/rescues/_request_and_response.erb (4.0ms) Rendered /var/lib/gems/1.8/gems/actionpack-3.0.9/lib/action_dispatch/middleware/templates/rescues/diagnostics.erb within rescues/layout (7.6ms)

[guanxiaohua2k6]

Maybe memcached service is not running. Please check it with the command below.

/etc/init.d/memcached status

[JianlanWang]

Just check the status of memcached. It's running.

[guanxiaohua2k6]

When you tried command "puppetd --test --server puppetserver.dodai.com", the error is ok because the objects needed was not put on memcached.

Therefore you should try by clicking on dodai-deploy web UI.

[guanxiaohua2k6]

Please tell me if there are errors in development.log when installing glance via web UI.

[guanxiaohua2k6]

Could you restart the service puppetmaster on dodai-deploy server with the steps below?

ps aux | grep "puppet master"

Find the PID of puppetmaster, then kill it.

kill -9 PID

Start puppetmaster.

service puppetmaster start

Finally please try to install glance or keystone again.

[JianlanWang]

Hey, Guan. Thanks for your rapid reply. The issue was disapeared when the job server only ran in development node. I think this may be my mistake that I mixed starting job server in production and development node. But could you tell me what's the difference between production mode and development mode? Because sometimes when I ran " puppetd --test --server puppetserver.dodai.com" on clientnode, that would generated different result in production mode and development node, which made me misunderstand that puppet didn't work well. Please refer to the following info.

In production node: root@clientnode:~# puppetd --test --server puppetserver.dodai.com err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find node 'clientnode.dodai.com'; cannot compile warning: Not using cache on failed catalog err: Could not retrieve catalog; skipping run

In development mode:

root@clientnode:~# puppetd --test --server puppetserver.dodai.com info: Caching catalog for clientnode.dodai.com info: Applying configuration version '1346202633' notice: /Stage[main]/Glance_e::Glance::Uninstall/Exec[rm -rf /var/lib/glance/*]/returns: executed successfully notice: Finished catalog run in 0.41 seconds

[guanxiaohua2k6]

In different mode, job server will use different db. If the proposal is created in development mode, it will be saved in the db for development mode, so can be only used in development mode.

[JianlanWang]

Got it. Thanks a lot:) Now my issue was resolved.