Better error handling for connectivity issues to github remote sources

sharpyy commented 11 months ago

Eschewed features

[X] This issue is not requesting templating, unstuctured edits, build-time side-effects from args or env vars, or any other eschewed feature.

What would you like to have added?

Currently, kustomize build running with github remote sources is not providing clear information when the git commands fails or did not get the expected content. We end up with error message like this one: MalformedYAMLError: yaml: line 30: mapping values are not allowed in this context in File which are not easy to understand. I would also suggest adding some retry mecanism depending on the type of error we get from git.

Why is this needed?

Having proper git command handling for all types of failures will help end user to troubleshoot issues.

Can you accomplish the motivating task without this feature, and if so, how?

We could probably enable git debugging locally but those logs can be huge and would collect all the git commands on the system. not just the one from kustomize build.

What other solutions have you considered?

N/A

Anything else we should know?

N/A

Feature ownership

[ ] I am interested in contributing this feature myself! 🎉

k8s-ci-robot commented 11 months ago

This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

robbiemcmichael commented 10 months ago

I ran into the same problem and have been digging into the code to see how best to fix it. The code around loading resources is a bit of a mess and it doesn't look like it's going to be an easy fix.

The problem comes from this section of the code:

https://github.com/kubernetes-sigs/kustomize/blob/0122aa82ef953436d0709365d55f0301e838489b/api/internal/target/kusttarget.go#L420-L433

My interpretation of this logic is:

It will try load resources by doing an HTTP GET on this URL and it stores any errors in errF
- If the HTTP GET was unsuccessful:
- If there was an HTTP error (interpreted as being outside the range 200 to 299) then it bails out and doesn't even try to fetch it with git (this is a separate problem if it's a private git repository, but that's separate from the poor error messages talked about in this issue)
- If it wasn't specifically an HTTP error then it tries to load the resources with git and stores any errors in err
  - If the git fetch fails:
  - It the HTTP resource loading error errF relates to invalid YAML then it returns only that error message and drops the error message relating to the git fetch in err
  - If the HTTP resource loading error errF is anything else then it returns only git fetch error message in err

The problem here is that the HTTP GET failed returned invalid YAML (because it's probably returning a webpage from a host like GitHub) so when the intended git fetch also fails it doesn't show the real error message.

Summary

All of the problems seem to come down to trying to load resources by optimistically fetching both the HTTP content and trying to fetch it with git. That may kind of work most of the time for the happy path, but what do you do if both of these fail?

When the user specified a resource starting with https:// did they intend for that to be an HTTP GET or a git fetch? Which error message should you present to the user to help them debug the problem?

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

robbiemcmichael commented 6 months ago

/remove-lifecycle stale

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

sharpyy commented 3 months ago

/remove-lifecycle stale

k8s-triage-robot commented 2 weeks ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

sharpyy commented 2 weeks ago

/remove-lifecycle stale

kubernetes-sigs / kustomize