Adding custom fields to the log is not working with rails 3.2.13

[sumanranjanpanda]

Adding custom fields to the log is not working with rails 3.2.13 and logstasher 0.2.5. I have followed all the step and created a file as config/initializers/logstasher.rb. Following are the code inside the file "logstasher.rb"

if LogStasher.enabled
 LogStasher.add_custom_fields do |fields|
    fields[:country_code] = "US"
 end
end

After that I have restarted the webrick server, but the LogStasher.enaabled returns nil value and not storing the country code in the log file.

My requirement is like, I need to save the location information to the log file with the other information to display the kibana map.

Can you plz help me?

[shadabahmed]

Hi Suman,

You have to enable logstaher in your environment config file for e.g. here is what I have in my production.rb:

config.logstasher.enabled = true

[sumanranjanpanda]

Thanks for your quick reply. But, I have already added these two lines in the development.rb though I am testing it with the development environment.

config.logstasher.enabled = true
config.logstasher.supress_app_log = false

[shadabahmed]

Can you see a file called logstasher_development.log ?

On Wed, Jul 31, 2013 at 3:32 PM, suman notifications@github.com wrote:

Thanks for your quick reply. But, I have already added these two lines in the development.rb though I am testing it with the development environment.

config.logstasher.enabled = true config.logstasher.supress_app_log = false

— Reply to this email directly or view it on GitHubhttps://github.com/shadabahmed/logstasher/issues/2#issuecomment-21851891 .

[sumanranjanpanda]

Yes Shadab, the "logstasher_development.log" has created in the log folder and also the log has written in json format. But I want request location information like country code, city name with the json log output.

[shadabahmed]

The code you've posted should work. try adding some constants as well like -> fields[:user] = 'I am a user'

[sumanranjanpanda]

Ok, But I have already tried with some constant.

#config/initializers/logstasher.rb
if LogStasher.enabled
    LogStasher.add_custom_fields do |fields|
      fields[:country_code] = "US"
    end
end

But it didn't work for me. Is there anything I am missing? I have followed all the steps mentioned in the readme file. I have also tried with the " lograge " gem (https://github.com/roidrage/lograge). Here, I am able to add the custom fields to the log file. But facing difficulty to filter the log information from the development.log file by the logstash "1.1.13". Since, I am new to logstash.

[shadabahmed]

there is nothing special to do .. try creating a brand new app and testing it in that app..

On Wed, Jul 31, 2013 at 4:00 PM, suman notifications@github.com wrote:

Ok, But I have already tried with some constant. But it didn't work for me. Is there anything I am missing? I have followed all the steps mentioned in the readme file.

config/initializers/logstasher.rb

if LogStasher.enabled LogStasher.add_custom_fields do |fields| fields[:country_code] = "US" end end

I have also tried with the " lograge " gem ( https://github.com/roidrage/lograge). Here, I am able to add the custom fields to the log file. But facing difficulty to filter the log information from the development.log file by the logstash "1.1.13". Since, I am new to logstash.

— Reply to this email directly or view it on GitHubhttps://github.com/shadabahmed/logstasher/issues/2#issuecomment-21853111 .

[sumanranjanpanda]

Ok, I'll try it with a new application. Thanks for your valuable help.

[shadabahmed]

@sumanranjanpanda . .were you able to solve the issue ?

[sumanranjanpanda]

Hi Shadab, thanks for asking me. But sorry to say that, I am not able to add custom fields. I have tried it with new applications but still unable to add.

Can you plz send me a sample application, where you have added the custom fields to the log? It will really help me a lot.

Thanks in advance.

[shadabahmed]

I created a sample application https://github.com/shadabahmed/logstasher_demo

It works absolutely fine. I added a custom field and this is the output I am getting in log/logstash_development.log

{"@source":"unknown","@tags":["request"],"@fields":     
{"method":"GET","path":"/","format":"html","controller":"logstashes","action":"index","status":200,
"duration":370.63,"view":338.55,"db":2.21,"ip":"127.0.0.1","route":"logstashes#index","parameters":"",
"user":"Logstasher"},"@timestamp":"2013-08-18T03:46:21.416014+00:00"}

You can clearly see the user field

[sumanranjanpanda]

Thank you very much Shadab for your valuable time.. I will definitely try this with another new application.

[hackhowtofaq]

Same problem here with rails 3.2.13...

There is probably a conflict with another gem or something.....

[shadabahmed]

Remove lograge and then try

On Thu, Aug 22, 2013 at 7:04 PM, HackHowToFaq notifications@github.comwrote:

Same problem here with rails 3.2.13...

There is probably a conflict with another gem or something.....

— Reply to this email directly or view it on GitHubhttps://github.com/shadabahmed/logstasher/issues/2#issuecomment-23089959 .

[hackhowtofaq]

I don't have lograge in my Gemfile....

[hackhowtofaq]

LogStasher is not enabled in the initializer even though it is enabled

config.logstasher.enabled = true

If you comment the if statement it works!

#config/initializers/logstasher.rb
#if LogStasher.enabled
 LogStasher.add_custom_fields do |fields|
    fields[:country_code] = "US"
 end
#end

[shadabahmed]

this is quite odd .. stupid question - is the logstasher.enabled set in the right environment file ?

[hackhowtofaq]

Yes it is enabled in the right environment file but the if statement in the initializer file is false....

[tomtaylor]

Just to check in and say I am also having this issue on 3.2.12 and 3.2.14. I'll take a closer look at it and see if I can submit a patch now.

[shadabahmed]

@tomtaylor Thanks tom .. if you can submit a patch that'll be awesome.. I am not able to even reproduce this issue

[tomtaylor]

Out of interest, I see that you're using Thread.current to store the custom values on. Is there a reason for that? Could we make it a class variable wrapped in a mutex? I wonder if this is causing issues with preforking servers, like unicorn, which spin up a master process and then fork child processes.

[shadabahmed]

@tomtaylor Thread.current works well in both multi-threaded (Passenger 4) and preforking servers. In prefork servers .. there is only 1 thread and hence this is non issue.

[tomtaylor]

Hmm... so I've got a little closer to reproducing this. I have a config/initializer/logstasher.rb, which contains:

if LogStasher.enabled
  LogStasher.add_custom_fields do |fields|
    if current_user
      fields[:user_email] = current_user.email
      fields[:user_id] = current_user.id.to_s
    end
  end
end

And in production.rb:

config.logstasher.enabled = true
config.logstasher.suppress_app_log = true

If I change the initializer to:

if LogStasher.enabled
  LogStasher.add_custom_fields do |fields|
    fields[:user_email] = current_user.try(:email)
    fields[:user_id] = current_user.try(:id).try(:to_s)
  end
end

It includes null user_email and user_id fields, even though there's a current_user present on that request. So, it looks like current_user is not present inside the block, even though there's a current_user method in ApplicationController (I can see it in the view).

[msaffitz]

+1 seeing this as well. We're adding the environment as a constant to custom fields and we're not seeing it appear:

if LogStasher.enabled
  LogStasher.add_custom_fields do |fields|
    # This block is run in application_controller context,
    # so you have access to all controller methods
    fields[:user] = current_user.try(:id).try(:to_s)
    fields[:site] = request.path =~ /^\/api/ ? 'api' : 'www'
    fields[:env] = Rails.env
  end
end

We're using LogStashLogger to ship directly to logstash:

  # Use a different logger for distributed setups
  config.logstasher.enabled = true

  config.logger =
    if ENV['LOG_STDOUT'].to_i == 1
      ActiveSupport::BufferedLogger.new($stdout)
    else
      config.logger = ActiveSupport::TaggedLogging.new(LogStashLogger.new('HOSTNAME', 5960, :udp))
    end
   config.logger.level = ENV['LOG_LEVEL'].present? ? Logger.const_get(ENV['LOG_LEVEL'].upcase) : Logger::INFO

[shadabahmed]

I think the issue is fixed with #17 . See the description to know why exactly this was happening

[mdarby]

Sorry to resurrect a closed issue but I'm seeing exactly the same thing and moving the gem/initializer up the load order doesn't seem to work. Getting logs, parsing just fine but no custom fields even when the normal wrapping if is removed.

[mdarby]

LogStasher.add_custom_fields do |fields|
  ip = request.env['HTTP_X_FORWARDED_FOR'] || request.remote_ip

  puts "-" * 10
  puts ip
  puts "-" * 10

  fields[:ipaddr] = ip
end

This produces the IP in the logs but no custom field.