Normal logs should not go to stderr

[chanzxxx]

Summary

Normal logs should not go to stderr.

Motivation

Some people including me detect critical issues on k8s by watching pods' stderr logs. But argocd applications are printing their all logs to stderr, and I think this is confused.

Proposal

Basically we should respect the purpose of each stream. Errors go stderr, and normal activity logs go stdout.

[alexmt]

This is not intended behavior. If some CLI commands are using stderr it is a bug.

[chanzxxx]

@alexmt I think this is related to logrus package. Its default output stream is stderr.

[l15k4]

Some of argo-events components also logs to stderr. This should be revisited across the whole argoproj. https://github.com/argoproj/argo-events/issues/884

[mrfelton]

Would love to see this resolved. My stackdriver logs are cluttered with inordinate amounts of error messages coming from argocd, which aren't actually errors - just that info level messages are being output to stderr. And since the messages themselves don't include the severity in a format that stackdriver recognises they default to being logged as errors since they went to stderr. If they go to stdout as they should they would have been logged as info rather than error.

[iam-veeramalla]

@jannfis , I am happy to contribute to this. Can you assign this to me ?

[jannfis]

Thanks @iam-veeramalla, much appreciated! I have assigned issue to you :)

[iam-veeramalla]

If I am understanding this issue correctly, It can be fairly easy fix by changing the default log from stderr to stdout. The default logging standard of logrus is stderr and can be modified to stdout by setting log.SetOutput(os.Stdout)

The real difficulty may be in testing this. Identify the different places where it outputs to stderr and change it to stdout.

[edgarkz]

same issue here with argocd: v2.0.0+f5119c0 BuildDate: 2021-04-07T06:00:33Z

Running with the following params "app patch-resource --kind Deployment --patch" logs to sdterr instead of stdout

[phelinor]

Same issue

[olvesh]

Also just ran into this, been open a while... Any known simple workarounds?

[iam-veeramalla]

Also just ran into this, been open a while... Any known simple workarounds?

@olvesh not yet, we are exploring options for a different logging package as the current project logrus is no longer active. It would continue as a maintenance project.

[olvesh]

Ok. I will try to upgrade to 2.1 tomorrow. As I understand I can set the log-format to json in that version. Perhaps that will remedy some of the logging issues in GKE (the other ones being that GKE fluentbit does not ingest the key=values as separate fields)🤞

On the other hand - why not then log structured logging to stdout and unstructured errors unhandled exeptions to stderr? Surely that can be done even if not continuing to use logrus, as most apps I have come across logs all structured logging to stdout? And I know that logrus can be configured to use either stdout or stderr. (log.SetOutput(os.Stdout))

Guess that this surely has been discussed somewhere, I just havent stumbled over it yet. 😄

[chanzxxx]

I'm willing to contribute to addressing this issue. Replacing logrus to another looks like best solution. What alternative packages do we have?

[iam-veeramalla]

@whdckszxxx https://github.com/golang/glog https://github.com/kubernetes/klog (fork of glog)

definitely many more. https://awesome-go.com/#logging

[chanzxxx]

@iam-veeramalla I was asking for packages under your consideration. If you had your research, plz share with me :)

[iam-veeramalla]

@iam-veeramalla I was asking for packages under your consideration. If you had your research, plz share with me :)

Yes, https://github.com/golang/glog https://github.com/kubernetes/klog (fork of glog)

we also had a discussion to explore the above packages in the contributors meeting

[chanzxxx]

Thanks, klog looks good to me. I will post a PR soon

[olvesh]

It has been a while since I programmed in Golang... But - does it exists a logging lib that can log sympathetic to the cloud it runs in? I.e. so that running in GCE will log in a format native to Google Logging (https://cloud.google.com/logging/docs/setup/go) and similar for AWS, Azure etc? At least so that it can be configured to tell the library that it is logging in a such-and-such environment?

[Piroddi]

Any update on this issue? Thanks

[mbhnyc]

Ping, this is definitely a weird thing.

[anjuls]

any idea when this is going to be fixed?

[crenshaw-dev]

@iam-veeramalla did some research and put up a draft PR. The conclusion of the PR review seems to have been that the performance hit associated with getting logrus to split output between stdout and stderr was not worth it and that we should just change loggers (which needs to happen anyway).

I think all that's reflected above, but hasn't been summarized here yet. :-)

If anyone has time to work on swapping out the logger, I'd be happy to review a PR! Also, please 👍 the issue so it's prioritized.

[iam-veeramalla]

@iam-veeramalla did some research and put up a draft PR. The conclusion of the PR review seems to have been that the performance hit associated with getting logrus to split output between stdout and stderr was not worth it and that we should just change loggers (which needs to happen anyway).

I think all that's reflected above, but hasn't been summarized here yet. :-)

If anyone has time to work on swapping out the logger, I'd be happy to review a PR! Also, please 👍 the issue so it's prioritized.

True, we did not evaluate the modern day loggers and choose one which is a best fit.

[crenshaw-dev]

zap and zerolog look like strong candidates.

[edoger]

@iam-veeramalla logrus can't solve this problem, maintainers don't seem interested in fixing this problem, I built a new log library for this: zkits-logger , the library is now stable, and the API style is consistent with logrus.

Logger.SetLevelOutput() // Set different output writers for different log levels.

I'm willing to provide a patch if everyone agrees.

[iam-veeramalla]

@edoger true, I was evaluating zap logger. I found it good and also recommended by others in the community. I have started migrating in steps, I can collaborate if you are interested.

[edoger]

@iam-veeramalla Is there a fixed plan? If not I will start devising a solution and I need to fix this asap.

[iam-veeramalla]

@edoger I already started migrating to the new log system. Please give me a couple of days time and I will let you know the progress. I am looking at getting the PR asap.

[edoger]

@iam-veeramalla Thanks, let us know if you need assistance, we have many full-time development engineers available to help.

[iam-veeramalla]

Sure, Thanks

[iam-veeramalla]

@edoger did you also evaluate zap logger or zero logger ? what is your thought on them ?

[crenshaw-dev]

Aside: looks like structured logging is coming to the standard library, but probably not for a while. https://github.com/golang/go/discussions/54763

[edoger]

@edoger did you also evaluate zap logger or zero logger ? what is your thought on them ?

@iam-veeramalla Zap is very powerful and adaptable to a variety of application scenarios. It is very suitable for large business applications. The disadvantage is that it must be understood how to use it correctly, and it is not suitable for small applications and libraries (because most of the functions are not used). The performance of zerolog is very good. It uses many techniques to optimize memory allocation. Before using it, you need to make a clear trade-off between ease of use and performance.

We use zap a lot in large-scale applications, but we find that it is difficult to replace the log library once it is used, because zap will explicitly enter various places in the application, and the reconstruction cost is very high. Engineers prefer to use logrus, a logging library with very low entry cost, but since the logrus library is no longer updated a few years ago, we built zkits-logger to replace logrus and zap. The reason is: For applications, the log library The performance of the log library is not an important issue, the customizability and ease of use of the logging library are very important indicators.

[iam-veeramalla]

@edgarkz Thanks for the very detailed information.

I just started implementing zap yesterday but because you have already worked with zap and zero log. I think it will be better if you or your team member picks this up. I will be happy to collaborate and be a co-author if required.

Does that work ?

[edoger]

@iam-veeramalla Okay, I first need to spend some time understanding the logging module of this library, and then replace the logging library without breaking the existing usage style as much as possible, what do you think of this attention? I don't mind which logging library to use, if you guys have come to a conclusion, just let me know and I'll decide when it's done.

[iam-veeramalla]

@edoger I think the most popular suggestion here is Zap , so we can start with it. We can also sync up over a call or bring this up in the next community meeting which takes place on Thursday.

[edoger]

@iam-veeramalla OK, I'll provide a patch asap.

[edoger]

@iam-veeramalla I will finish this this weekend.

[edoger]

@iam-veeramalla I've created a corresponding patch, which requires further discussion before a global replacement. Unit tests I will complete after the discussion.

[iam-veeramalla]

ore a global

Hi @edoger, Do you want to bring this discussion in our next community meeting which is on the thursday ?

[edoger]

Hi @edoger, Do you want to bring this discussion in our next community meeting which is on the thursday ?

@iam-veeramalla OK!

• Whether the new application logging subpackage is approved.
• Use the new Logger interface to decouple the log driver library and the application.
• Wraps zap into existing logging styles to reduce refactoring costs.
• Whether to provide configuration items to split log output to standard error and standard output.
• Whether the log format and log level need to be modified while the application is running.

Do you have any other discussion points?

[iam-veeramalla]

Hi @edoger, Do you want to bring this discussion in our next community meeting which is on the thursday ?

@iam-veeramalla OK!

* Whether the new application logging subpackage is approved.

* Use the new Logger interface to decouple the log driver library and the application.

* Wraps zap into existing logging styles to reduce refactoring costs.

* Whether to provide configuration items to split log output to standard error and standard output.

* Whether the log format and log level need to be modified while the application is running.

Do you have any other discussion points?

Sounds Great :) .. Do you have the link for the contributors meeting ? Please let me know, I can share the invite.

[edoger]

Sounds Great :) .. Do you have the link for the contributors meeting ? Please let me know, I can share the invite.

@iam-veeramalla Please email to share with me. O(∩_∩)O~~~

[iam-veeramalla]

@edoger you can find it here -> Contributors Office Hours: Every Thursday | Agenda

[edoger]

@edoger you can find it here -> Contributors Office Hours: Every Thursday | Agenda

@iam-veeramalla I don't have access to Google 😞 and may need your help with the results of the discussion.

[iam-veeramalla]

@edoger you can find it here -> Contributors Office Hours: Every Thursday | Agenda

@iam-veeramalla I don't have access to Google 😞 and may need your help with the results of the discussion.

I can add this to the agenda .. no problem. You will lead the discussion at the contributors meeting right ?

[edoger]

@edoger you can find it here -> Contributors Office Hours: Every Thursday | Agenda

@iam-veeramalla I don't have access to Google 😞 and may need your help with the results of the discussion.

I can add this to the agenda .. no problem. You will lead the discussion at the contributors meeting right ?

@iam-veeramalla I don't have the right amount of free time to attend and I support the outcome of any of your discussions. I can't understand English conversations and can only read English documents.

[iam-veeramalla]

@edoger you can find it here -> Contributors Office Hours: Every Thursday | Agenda

@iam-veeramalla I don't have access to Google 😞 and may need your help with the results of the discussion.

I can add this to the agenda .. no problem. You will lead the discussion at the contributors meeting right ?

@iam-veeramalla I don't have the right amount of free time to attend and I support the outcome of any of your discussions. I can't understand English conversations and can only read English documents.

Ok, I will see if I can get this into today's agenda and get back to you with the outcome of the meeting:)

[iam-veeramalla]

Hi @edoger, Please find the response from the community meeting.

Whether the new application logging subpackage is approved.

Yes. Do we also want to support Sugar logging with Zap ? Do you see any advantages ?

Use the new Logger interface to decouple the log driver library and the application.

Yes. There is already a log package, if possible we can leverage that.

Wraps zap into existing logging styles to reduce refactoring costs.

Yes

Whether to provide configuration items to split log output to standard error and standard output.

Yes

Whether the log format and log level need to be modified while the application is running.

No opinion, You can take this decision.

[edoger]

@iam-veeramalla thank you very much! I have seen the feedback and understand it. I'll make all the changes this weekend and can do a code review next Monday. If you have any other questions or comments, feel free to email me at any time.