cannot reconfigure

[veilig2000]

I recently grew my log server disk on EC2. when I booted back up, I went to reconfigure graylog so it would grab my updated information

it now bombs out on me everytime w/ elastisearch

jkahgee@logs:~$ sudo graylog2-ctl reconfigure
Starting Chef Client, version 12.0.3
Compiling Cookbooks...
Recipe: graylog2::default
  * directory[/etc/graylog2] action create (up to date)
  Converging 119 resources
  * directory[/etc/graylog2] action nothing (skipped due to action :nothing)
Recipe: graylog2::users
  * group[graylog2] action create (up to date)
  * user[graylog2] action create (up to date)
  * directory[/var/opt/graylog2] action create (up to date)
  * template[/var/opt/graylog2/.gitconfig] action create (up to date)
  * directory[/var/opt/graylog2/data] action create (up to date)
Recipe: graylog2::authbind
  * directory[/etc/authbind/byport] action create (up to date)
  * file[/etc/authbind/byport/!514] action create (up to date)
Recipe: runit::upstart
  * cookbook_file[/etc/init/graylog2-runsvdir.conf] action create (up to date)
  * execute[initctl status graylog2-runsvdir] action run
    - execute initctl status graylog2-runsvdir
  * execute[initctl start graylog2-runsvdir] action run (skipped due to only_if)
Recipe: timezone-ii::default
  * apt_package[tzdata] action install (up to date)
Recipe: timezone-ii::debian
  * template[/etc/timezone] action create (up to date)
  * bash[dpkg-reconfigure tzdata] action nothing (skipped due to action :nothing)
Recipe: graylog2::etcd
  * directory[/var/log/graylog2/etcd] action create (up to date)
  * directory[/var/opt/graylog2/data/etcd] action create (up to date)
  * directory[/opt/graylog2/sv/etcd] action create (up to date)
  * directory[/opt/graylog2/sv/etcd/log] action create (up to date)
  * directory[/opt/graylog2/sv/etcd/log/main] action create (up to date)
  * template[/opt/graylog2/sv/etcd/run] action create (up to date)
  * template[/opt/graylog2/sv/etcd/log/run] action create (up to date)
  * template[/var/log/graylog2/etcd/config] action create (up to date)
  * ruby_block[reload etcd svlogd configuration] action nothing (skipped due to action :nothing)
  * file[/opt/graylog2/sv/etcd/down] action delete (up to date)
  * link[/opt/graylog2/init/etcd] action create (up to date)
  * link[/opt/graylog2/service/etcd] action create (up to date)
  * ruby_block[supervise_etcd_sleep] action run (skipped due to not_if)
  * service[etcd] action nothing (skipped due to action :nothing)
  * execute[/opt/graylog2/embedded/bin/graylog2-ctl start etcd] action run
    - execute /opt/graylog2/embedded/bin/graylog2-ctl start etcd
Recipe: graylog2::elasticsearch
  * directory[/var/log/graylog2/elasticsearch] action create (up to date)
  * directory[/var/opt/graylog2/data/elasticsearch] action create (up to date)
  * template[/opt/graylog2/conf/elasticsearch.yml] action create (up to date)
  * directory[/opt/graylog2/sv/elasticsearch] action create (up to date)
  * directory[/opt/graylog2/sv/elasticsearch/log] action create (up to date)
  * directory[/opt/graylog2/sv/elasticsearch/log/main] action create (up to date)
  * template[/opt/graylog2/sv/elasticsearch/run] action create (up to date)
  * template[/opt/graylog2/sv/elasticsearch/log/run] action create (up to date)
  * template[/var/log/graylog2/elasticsearch/config] action create (up to date)
  * ruby_block[reload elasticsearch svlogd configuration] action nothing (skipped due to action :nothing)
  * file[/opt/graylog2/sv/elasticsearch/down] action delete (up to date)
  * link[/opt/graylog2/init/elasticsearch] action create (up to date)
  * link[/opt/graylog2/service/elasticsearch] action create (up to date)
  * ruby_block[supervise_elasticsearch_sleep] action run (skipped due to not_if)
  * service[elasticsearch] action nothing (skipped due to action :nothing)
  * execute[/opt/graylog2/embedded/bin/graylog2-ctl start elasticsearch] action run
    - execute /opt/graylog2/embedded/bin/graylog2-ctl start elasticsearch
  * ruby_block[add node to cluster list] action run

    ================================================================================
    Error executing action `run` on resource 'ruby_block[add node to cluster list]'
    ================================================================================

    NameError
    ---------
    undefined local variable or method `name' for #<Graylog2Registry:0x00000003c870f0>

    Cookbook Trace:
    ---------------
    /opt/graylog2/embedded/cookbooks/graylog2/libraries/registry.rb:94:in `rescue in add_node'
    /opt/graylog2/embedded/cookbooks/graylog2/libraries/registry.rb:91:in `add_node'
    /opt/graylog2/embedded/cookbooks/graylog2/libraries/registry.rb:44:in `add_es_node'
    /opt/graylog2/embedded/cookbooks/graylog2/recipes/elasticsearch.rb:48:in `block (2 levels) in from_file'

    Resource Declaration:
    ---------------------
    # In /opt/graylog2/embedded/cookbooks/graylog2/recipes/elasticsearch.rb

     46: ruby_block "add node to cluster list" do
     47:   block do
     48:     $registry.add_es_node(node['ipaddress'])
     49:   end
     50: end

    Compiled Resource:
    ------------------
    # Declared in /opt/graylog2/embedded/cookbooks/graylog2/recipes/elasticsearch.rb:46:in `from_file'

    ruby_block("add node to cluster list") do
      action "run"
      retries 0
      retry_delay 2
      default_guard_interpreter :default
      block_name "add node to cluster list"
      declared_type :ruby_block
      cookbook_name :graylog2
      recipe_name "elasticsearch"
      block #<Proc:0x00000003ee9910@/opt/graylog2/embedded/cookbooks/graylog2/recipes/elasticsearch.rb:47>
    end

Running handlers:
[2015-03-25T16:34:04-04:00] ERROR: Running exception handlers
Running handlers complete
[2015-03-25T16:34:04-04:00] ERROR: Exception handlers complete
[2015-03-25T16:34:04-04:00] FATAL: Stacktrace dumped to /opt/graylog2/embedded/cookbooks/cache/chef-stacktrace.out
Chef Client failed. 3 resources updated in 302.259348782 seconds
[2015-03-25T16:34:04-04:00] ERROR: ruby_block[add node to cluster list] (graylog2::elasticsearch line 46) had an error: NameError: undefined local variable or method `name' for #<Graylog2Registry:0x00000003c870f0>
[2015-03-25T16:34:04-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

[mariussturm]

Hi, could you please provide the content of /etc/graylog/graylog-settings.json ?

On 25 March 2015 at 21:36, Jamie Kahgee notifications@github.com wrote:

I recently grew my log server disk on EC2. when I booted back up, I went to reconfigure graylog so it would grab my updated information

it now bombs out on me everytime w/ elasticache

jkahgee@logs:~$ sudo graylog2-ctl reconfigure Starting Chef Client, version 12.0.3 Compiling Cookbooks... Recipe: graylog2::default

• directory[/etc/graylog2] action create (up to date) Converging 119 resources
• directory[/etc/graylog2] action nothing (skipped due to action :nothing) Recipe: graylog2::users
• group[graylog2] action create (up to date)
• user[graylog2] action create (up to date)
• directory[/var/opt/graylog2] action create (up to date)
• template[/var/opt/graylog2/.gitconfig] action create (up to date)
• directory[/var/opt/graylog2/data] action create (up to date) Recipe: graylog2::authbind
• directory[/etc/authbind/byport] action create (up to date)
• file[/etc/authbind/byport/!514] action create (up to date) Recipe: runit::upstart
• cookbook_file[/etc/init/graylog2-runsvdir.conf] action create (up to date)
• execute[initctl status graylog2-runsvdir] action run
  • execute initctl status graylog2-runsvdir
• execute[initctl start graylog2-runsvdir] action run (skipped due to only_if) Recipe: timezone-ii::default
• apt_package[tzdata] action install (up to date) Recipe: timezone-ii::debian
• template[/etc/timezone] action create (up to date)
• bash[dpkg-reconfigure tzdata] action nothing (skipped due to action :nothing) Recipe: graylog2::etcd
• directory[/var/log/graylog2/etcd] action create (up to date)
• directory[/var/opt/graylog2/data/etcd] action create (up to date)
• directory[/opt/graylog2/sv/etcd] action create (up to date)
• directory[/opt/graylog2/sv/etcd/log] action create (up to date)
• directory[/opt/graylog2/sv/etcd/log/main] action create (up to date)
• template[/opt/graylog2/sv/etcd/run] action create (up to date)
• template[/opt/graylog2/sv/etcd/log/run] action create (up to date)
• template[/var/log/graylog2/etcd/config] action create (up to date)
• ruby_block[reload etcd svlogd configuration] action nothing (skipped due to action :nothing)
• file[/opt/graylog2/sv/etcd/down] action delete (up to date)
• link[/opt/graylog2/init/etcd] action create (up to date)
• link[/opt/graylog2/service/etcd] action create (up to date)
• ruby_block[supervise_etcd_sleep] action run (skipped due to not_if)
• service[etcd] action nothing (skipped due to action :nothing)
• execute[/opt/graylog2/embedded/bin/graylog2-ctl start etcd] action run
  • execute /opt/graylog2/embedded/bin/graylog2-ctl start etcd Recipe: graylog2::elasticsearch
• directory[/var/log/graylog2/elasticsearch] action create (up to date)
• directory[/var/opt/graylog2/data/elasticsearch] action create (up to date)
• template[/opt/graylog2/conf/elasticsearch.yml] action create (up to date)
• directory[/opt/graylog2/sv/elasticsearch] action create (up to date)
• directory[/opt/graylog2/sv/elasticsearch/log] action create (up to date)
• directory[/opt/graylog2/sv/elasticsearch/log/main] action create (up to date)
• template[/opt/graylog2/sv/elasticsearch/run] action create (up to date)
• template[/opt/graylog2/sv/elasticsearch/log/run] action create (up to date)
• template[/var/log/graylog2/elasticsearch/config] action create (up to date)
• ruby_block[reload elasticsearch svlogd configuration] action nothing (skipped due to action :nothing)
• file[/opt/graylog2/sv/elasticsearch/down] action delete (up to date)
• link[/opt/graylog2/init/elasticsearch] action create (up to date)
• link[/opt/graylog2/service/elasticsearch] action create (up to date)
• ruby_block[supervise_elasticsearch_sleep] action run (skipped due to not_if)
• service[elasticsearch] action nothing (skipped due to action :nothing)
• execute[/opt/graylog2/embedded/bin/graylog2-ctl start elasticsearch] action run
  • execute /opt/graylog2/embedded/bin/graylog2-ctl start elasticsearch

• ruby_block[add node to cluster list] action run

  Error executing action run on resource 'ruby_block[add node to cluster list]'

  NameError

  undefined local variable or method `name' for #Graylog2Registry:0x00000003c870f0

  Cookbook Trace:

  /opt/graylog2/embedded/cookbooks/graylog2/libraries/registry.rb:94:in rescue in add_node' /opt/graylog2/embedded/cookbooks/graylog2/libraries/registry.rb:91:inadd_node' /opt/graylog2/embedded/cookbooks/graylog2/libraries/registry.rb:44:in add_es_node' /opt/graylog2/embedded/cookbooks/graylog2/recipes/elasticsearch.rb:48:inblock (2 levels) in from_file'

  Resource Declaration:

  In /opt/graylog2/embedded/cookbooks/graylog2/recipes/elasticsearch.rb

  46: ruby_block "add node to cluster list" do 47: block do 48: $registry.add_es_node(node['ipaddress']) 49: end 50: end

  Compiled Resource:

  Declared in /opt/graylog2/embedded/cookbooks/graylog2/recipes/elasticsearch.rb:46:in `from_file'

  ruby_block("add node to cluster list") do action "run" retries 0 retry_delay 2 default_guard_interpreter :default block_name "add node to cluster list" declared_type :ruby_block cookbook_name :graylog2 recipe_name "elasticsearch" block #Proc:0x00000003ee9910@/opt/graylog2/embedded/cookbooks/graylog2/recipes/elasticsearch.rb:47 end

Running handlers: [2015-03-25T16:34:04-04:00] ERROR: Running exception handlers Running handlers complete [2015-03-25T16:34:04-04:00] ERROR: Exception handlers complete [2015-03-25T16:34:04-04:00] FATAL: Stacktrace dumped to /opt/graylog2/embedded/cookbooks/cache/chef-stacktrace.out Chef Client failed. 3 resources updated in 302.259348782 seconds [2015-03-25T16:34:04-04:00] ERROR: ruby_block[add node to cluster list](graylog2::elasticsearch line 46) had an error: NameError: undefined local variable or method `name' for #Graylog2Registry:0x00000003c870f0 [2015-03-25T16:34:04-04:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

— Reply to this email directly or view it on GitHub https://github.com/Graylog2/graylog2-server/issues/1079.

Developer

Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078

TORCH GmbH - A Graylog Company Steckelhörn 11 20457 Hamburg Germany

https://www.graylog.com https://www.torch.sh/

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO)

[veilig2000]

{
  "timezone": "America/New_York",
  "smtp_server": "",
  "smtp_port": 587,
  "smtp_user": "",
  "smtp_password": "",
  "smtp_no_tls": false,
  "smtp_no_ssl": false,
  "master_node": "127.0.0.1",
  "local_connect": true,
  "current_address": "10.158.34.234",
  "last_address": "10.158.34.234",
  "enforce_ssl": false,
  "rotation_size": 1073741824,
  "rotation_time": 0,
  "indices": 10,
  "journal_size": 1
}

I've tried adjusting a few of the settings, ie: graylog2-ctl local-connect which adjusted the current_address and last_address indexes. I've since changed them back. not sure what else it changed.

[veilig2000]

actually - after looking around. I seem to have two graylogs? graylog and graylog2?

not sure how or why. but I have a /etc/graylog2 directory as well. this is the graylog-settings.json file in that directory.

{
  "timezone": "America/New_York",
  "smtp_server": "email-smtp.us-east-1.amazonaws.com",
  "smtp_port": 25,
  "smtp_user": "xxxxxx",
  "smtp_password": "xxxxxxx",
  "master_node": "127.0.0.1",
  "local_connect": false
}

it doesn't matter if I graylog-ctl reconfigure or graylog2-ctl reconfigure` both seem to fail at elasticsearch

[mariussturm]

I guess you updated the appliance at some point. There was a rebrand from Graylog2 to Graylog. That makes the situation harder to debug. You could create a snapshot of the vm and try to uninstall the graylog2 package. In this mixed state you never know which version is actually running. After the deinstall restart the vm.

On 26 March 2015 at 14:40, Jamie Kahgee notifications@github.com wrote:

actually - after looking around. I seem to have two graylogs? graylog and graylog2?

not sure how or why. but I have a /etc/graylog2 directory as well. this is the graylog-settings.json file in that directory.

{ "timezone": "America/New_York", "smtp_server": "email-smtp.us-east-1.amazonaws.com", "smtp_port": 25, "smtp_user": "xxxxxx", "smtp_password": "xxxxxxx", "master_node": "127.0.0.1", "local_connect": false }

it doesn't matter if I graylog-ctl reconfigure or graylog2-ctl reconfigure` both seem to fail at elasticsearch

— Reply to this email directly or view it on GitHub https://github.com/Graylog2/graylog2-server/issues/1079#issuecomment-86517129 .

Developer

Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078

TORCH GmbH - A Graylog Company Steckelhörn 11 20457 Hamburg Germany

https://www.graylog.com https://www.torch.sh/

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO)

[veilig2000]

I created a fresh volume from my original snapshot which removed the graylog and kept the graylog2 files. so I must have updated/added that in one of my attempts to resolve this issue

However, I think I might have found the cause - looking over the debug output and backtrace.

    undefined local variable or method `name' for #<Graylog2Registry:0x00000003c870f0>

after walking through the cookbook trace I found this line in my slightly older version

Chef::Log.debug("Can not add node #{name} to directory #{context}") in file

/opt/graylog2/embedded/cookbooks/graylog2/libraries/registry.rb:94:in `rescue in add_node'

and after poking around and looking at the latest release, I see that line has since been changed https://github.com/Graylog2/omnibus-graylog2/blob/master/files/graylog-cookbooks/graylog/libraries/registry.rb#L94

When I updated that undefined variable, it appears to have allowed me to move forward from that point. still struggling w/ a few issues, but slowly making fwd progress.

[bernd]

Closing this now, please re-open if you still have issues. Thank you!