Operator keeps writing log files

[skdltmxn]

What happened?

After upgrading the operator to 1.11.2, I see many log files are generated in /tmp.
It seems like there are three types of log: INFO, WARNING and ERROR.
Unlike https://github.com/pulumi/pulumi/issues/12263#issue-1597281168, where pipes didn't actually take any space, these log files do take volume spaces.
I checked what was actually written to the files and found that all logs have same content.

Log file created at: 2023/03/11 06:36:31
Running on machine: pulumi-kubernetes-operator-756d7b6c98-znfp7
Binary: Built with gc go1.20.1 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0311 06:36:31.070780    6999 log.go:84] GitHub rate limit exceeded, try again in 23m37.929230884s. You can set GITHUB_TOKEN to make an authenticated request with a higher rate limit.
E0311 06:36:31.080911    6999 log.go:84] GitHub rate limit exceeded, try again in 23m37.919093344s. You can set GITHUB_TOKEN to make an authenticated request with a higher rate limit.

Here are two questions;

1. Why do I see GitHub rate limit error from 1.11.2? There were no such errors in previous version and I think those errors actually don't affect my stack reconcilation.
2. I see the same errors for all types of log files: INFO, WARNING and ERROR. Why INFO and WARNING log files also contain error logs?

Following is screen capture of my /tmp directory.

Expected Behavior

There should be no redundant log files

Steps to reproduce

Run pulumi-kubernetes-operator v1.11.2 with constant stack reconcilation enabled.

Output of pulumi about

CLI
Version      3.57.1
Go Version   go1.20.1
Go Compiler  gc

Host
OS       debian
Version  11.6
Arch     x86_64

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[skdltmxn]

Tried --logtostderr option but doesn't work...

[rquitales]

Assigning to @squaremo for further triage who has more context about the v1.11.2 release.

[squaremo]

These logs are produced by pulumi, which is (via the "Automation API") exec'ed by the operator. I updated both the API and the pulumi executable in the image, but didn't change how they are used, which makes me think it's a change in pulumi that causes this. I'll go looking.

Incidental question: are you using the GitHub provider in your program?

[skdltmxn]

These logs are produced by pulumi, which is (via the "Automation API") exec'ed by the operator. I updated both the API and the pulumi executable in the image, but didn't change how they are used, which makes me think it's a change in pulumi that causes this. I'll go looking.

Incidental question: are you using the GitHub provider in your program?

If you are asking I'm using Github for my program repo, then yes, I'm using Github enterprise edition.

[squaremo]

Sorry, I should have been clearer: I meant are you using https://www.pulumi.com/registry/packages/github/ in your Pulumi stack, e.g.,

import * as github from "@pulumi/github";

const repo = new github.Repository("demo-repo", {
  description: "Generated from automated test",
  visibility: "private",
});

// etc.

I can think of three things that might be provoking a rate-limiting error from GitHub:

• git operations, like cloning
• using the GitHub provider programmatically in a pulumi stack
• pulumi downloading a plugin, which accesses the GitHub API.

I'm just trying to figure out which of these it might be that's causing the message. I see at least one change relating to GitHub authorisation, in pulumi/pulumi: https://github.com/pulumi/pulumi/pull/12392 for one.

[skdltmxn]

No, I'm not using github provider.
Since there is no problem on syncing my infra, I guess it's due to plugin download?