Closed rocktavious closed 3 years ago
I realize this is not the same case and don't want to hijack the thread, but perhaps this may be useful to someone.
My usecase:
AWS codecommit repository, EKS cluster stood up with eksctl.
Flux installed with https://github.com/fluxcd/flux/blob/master/chart/flux/README.md#flux-with-git-over-https
Due to policy restrictions at IAM level, I can only use https credentials for codecommit. A dedicated user bearing above credentials has full write permissions to the config repository.
Received the https not supported error.
Flux versions tried: 1.18 and 1.17.1
What helped me was simply removing single quotes from my https git url in the deployment.
From:
--git-url='http://$(GIT_AUTHUSER):$(GIT_AUTHKEY)@git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/some-repo'
To:
--git-url=http://$(GIT_AUTHUSER):$(GIT_AUTHKEY)@git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/some-repo
I can't reproduce it, here is what I did:
Create secret with both username and token:
kubectl -n flux create secret generic flux-git-auth --from-literal=GIT_AUTHUSER=stefanprodan --from-literal=GIT_AUTHKEY=secret
Deploy Flux with:
envFrom:
- secretRef:
name: flux-git-auth
args:
- --git-url=https://$(GIT_AUTHUSER):$(GIT_AUTHKEY)@github.com/stefanprodan/gitops-auth.git
Logs:
ts=2020-04-02T09:47:54.425618827Z caller=loop.go:133 component=sync-loop event=refreshed url=https://stefanprodan@github.com/stefanprodan/gitops-auth.git branch=master HEAD=bd302aa796748f4952d1840da8689e8d5df7f41e
ts=2020-04-02T09:47:55.486887834Z caller=sync.go:528 method=Sync cmd=apply args= count=1
ts=2020-04-02T09:47:55.921364515Z caller=sync.go:594 method=Sync cmd="kubectl apply -f -" took=434.412927ms err=null output="namespace/test-auth created"
ts=2020-04-02T09:47:55.923003228Z caller=daemon.go:701 component=daemon event="Sync: bd302aa, <cluster>:namespace/test-auth" logupstream=false
ts=2020-04-02T09:47:55.929022962Z caller=loop.go:226 component=sync-loop state="kubernetes flux:secret/flux-git-deploy" old= new=bd302aa796748f4952d1840da8689e8d5df7f41e
ts=2020-04-02T09:47:56.651100463Z caller=loop.go:133 component=sync-loop event=refreshed url=https://stefanprodan@github.com/stefanprodan/gitops-auth.git branch=master HEAD=bd302aa796748f4952d1840da8689e8d5df7f41e
Can confirm the issue. As far as I've figured out a specific GIT_AUTHKEY
may break the http url check url.Parse(*gitURL)
I had a key like MjA1MzE1NjIwMzA0OitF9mqpJ/iulOBd9NzX36LyVZ9C
generated by Bitbucket and in this case /
inside the string rise an error for url.Parse
I followed instruction on setting up Flux with Auth Key with HTTPS, I setup a private repo with the Auth Key, but could not get it to clone the repo, here is the error message:
❯ fluxctl sync --k8s-fwd-ns flux
Error: git repository file://@https://seizadi%0A:43,,,,,,,,,,%0A@github.com/seizadi/private-flux.git is not ready to sync
Full error message: git clone --mirror: fatal: unable to access 'https://github.com/seizadi/private-flux.git/': URL using bad/illegal format or missing URL, full output:
Cloning into bare repository '/tmp/flux-gitclone152170752'...
fatal: unable to access 'https://github.com/seizadi/private-flux.git/': URL using bad/illegal format or missing URL
I am not sure the the additional character 'seizadi%0A' is getting in there, the GIT_AUTHUSER from the environment variable is 'seizadi'
> kubectl -n flux get secret flux-git-auth -o yaml
apiVersion: v1
data:
GIT_AUTHKEY: .......
GIT_AUTHUSER: c2VpemFkaQo=
kind: Secret
metadata:
name: flux-git-auth
namespace: flux
type: Opaque
❯ echo c2VpemFkaQo= | base64 --decode
seizadi
I was able to reproduce this issue when '/' character is in the the password.
Is there a way to escape that?
The GIT_AUTHKEY is what I get from GithHub for token form this URL
https://github.com/settings/tokens/new
It look like this:
e3a7cd19380c67c06fa24a898c89d34a924e2a1e
Were you able to see the extra character in the GIT_AUTHUSER in my case it was 'seizadi%0A' versus 'seizadi'?
I did not have any issue with characters not being passed in.
My GIT_AUTHKEY was exactly the way it was suppose to be. It seems something with the parsing of the input was getting tripped up and was throwing the error 'URL using bad/illegal format or missing URL'.
I replaced the GIT_AUTHKEY with value that did not contain '/' and issue went away.
What I found out is the https git url is constructed by inserting GIT_AUTHUSER and GIT_AUTHKEY. If these secrets contain characters that need to be urlencoded than an invalid url is constructed.
For https urls the values taken from the secret should be urlencoded before constructing the url.
As a workaround it is possible to create the secret with the values that urlencoded manually before.
For my use case the values come from AWS CodeCommit and contain the characters '/' and '+'. I had to replace these characters by '%2F' and '%2B'.
I think this could be related to #3330, #3339
Credentials are logged when they should be occluded by a feature to prevent logging credentials, because of special characters that are HTTP escaped/CGI encoded in their password.
Flux v1 is formally superseded since the GitOps Toolkit APIs have been declared stable:
https://fluxcd.io/docs/migration/timetable/
The repo will remain in maintenance for some time, but no new features can be accepted. Bugs can be addressed if they are critical and there is a PR to resolve it, but soon only CVEs can be addressed in Flux v1, and new users are all recommended to use Flux v2 for some time now.
Thanks for using Flux!
Describe the bug
When trying to stand up flux in a fresh kubernetes cluster using HTTPS git url's it seems is impossible to get working. Following the configuration documentation here https://docs.fluxcd.io/en/1.18.0/guides/use-git-https.html
To Reproduce
Standup flux on a new kubernetes cluster with the following option while also removing
--ssh-keygen-dir
flagDoes not lead to a HTTPS clone of the git repo you get
I also validated that by going through and fixing up the problems with SSH keygen (fixing the volume mounts and adding back the --ssh-keygen-dir flag i can get flux to proceed to the following logs lines but clearly you can see its not doing the right thing (and it never actually clones and applys during the sync-loop.
clearly the above is trying to use SSH
but then later the logs spit out this
Expected behavior
Following the documentation to perform an HTTPS based clone forces flux to use HTTPS instead of SSH.
We were looking into using HTTPS clones for all our flux daemons because our Github org is going to be put behind SAML and we have a service account with a personal access token that will be SSO enabled so we were thinking of using that instead of having to work with SSH keys.
Additional context