search

containers / fetchit

FetchIt is used to manage the life cycle and configuration of Podman containers
https://fetchit.readthedocs.io/
GNU Affero General Public License v3.0
124 stars 23 forks source link

kube play target does not work with private repositories #281

Closed alexanderniebuhr closed 1 year ago

alexanderniebuhr commented 1 year ago

pretty sure I followed the docs. The pat works for configReload but not for kube plays. Some logs:

2022-12-20T22:15:22.726Z        INFO    engine/fetchit.go:365   Processing git target:  Method: config Name: config
2022-12-20T22:15:23.001Z        INFO    engine/config.go:208    Current config backup placed at /opt/mount/config-backup.yaml
2022-12-20T22:15:23.001Z        INFO    engine/config.go:215    Config updates found from url: https://raw.githubusercontent.com/alexanderniebuhr/lab/main/fetchit/config.yaml, will load new targets
2022-12-20T22:15:23.001Z        INFO    engine/config.go:63     Updated config processed, restarting with new targets
2022-12-20T22:15:23.006Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:15:23.308Z        INFO    engine/fetchit.go:365   Processing git target: https://github.com/alexanderniebuhr/lab Method: kube Name: home-assistant
2022-12-20T22:15:23.308Z        INFO    engine/fetchit.go:365   Processing git target:  Method: config Name: config
2022-12-20T22:20:00.007Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:20:00.478Z        ERROR   engine/kube.go:55       Error moving to current: Failed to get current commit: Error opening repository lab to fetch current commit: repository does not exist
2022-12-20T22:20:00.478Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:20:00.625Z        ERROR   engine/kube.go:55       Error moving to current: Failed to get current commit: Error opening repository lab to fetch current commit: repository does not exist
2022-12-20T22:26:02.836Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:26:03.167Z        INFO    engine/fetchit.go:365   Processing git target:  Method: config Name: config
2022-12-20T22:26:03.167Z        INFO    engine/fetchit.go:365   Processing git target: https://github.com/alexanderniebuhr/lab Method: kube Name: home-assistant
2022-12-20T22:26:03.167Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:26:03.302Z        ERROR   engine/kube.go:55       Error moving to current: Failed to get current commit: Error opening repository lab to fetch current commit: repository does not exist
2022-12-20T22:28:42.383Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:28:42.640Z        INFO    engine/fetchit.go:365   Processing git target: https://github.com/alexanderniebuhr/lab Method: kube Name: home-assistant
2022-12-20T22:28:42.640Z        INFO    engine/fetchit.go:365   Processing git target:  Method: config Name: config
2022-12-20T22:28:42.641Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:28:42.771Z        ERROR   engine/kube.go:55       Error moving to current: Failed to get current commit: Error opening repository lab to fetch current commit: repository does not exist
2022-12-20T22:28:42.829Z        INFO    engine/config.go:208    Current config backup placed at /opt/mount/config-backup.yaml
2022-12-20T22:28:42.830Z        INFO    engine/config.go:215    Config updates found from url: https://raw.githubusercontent.com/alexanderniebuhr/lab/main/fetchit/config.yaml, will load new targets
2022-12-20T22:28:42.830Z        INFO    engine/config.go:63     Updated config processed, restarting with new targets
2022-12-20T22:28:42.836Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-20T22:28:42.953Z        INFO    engine/fetchit.go:365   Processing git target: https://github.com/alexanderniebuhr/lab Method: kube Name: home-assistant
2022-12-20T22:28:42.953Z        INFO    engine/fetchit.go:365   Processing git target:  Method: config Name: config
cooktheryan commented 1 year ago

@alexanderniebuhr let me test this tomorrow. This may be a documentation issue

cooktheryan commented 1 year ago

Can you try to use the example file examples/pat-testing-kube.yaml

Here is the output from my system

rcook  @ ~/go/src/github.com/redhat-et/fetchit
pat-kube-check└─ $ podman run -d --rm --name fetchit     -v fetchit-volume:/opt     -v $HOME/.fetchit:/opt/mount     -v /run/user/$(id -u)/podman//podman.sock:/run/podman/podman.sock     --security-opt label=disable     quay.io/fetchit/fetchit:latest
Trying to pull quay.io/fetchit/fetchit:latest...
Getting image source signatures
Copying blob 455cd3476656 done  
Copying blob eee1869219e1 done  
Copying blob 3e7f57f156fa done  
Copying blob c9ba6b2d6c27 done  
Copying config c1dbddc792 done  
Writing manifest to image destination
Storing signatures
22d1e3b57b4cb2e6edfa1822dacaf173c0e54e13b769a1040c1d1eef710cf461
rcook  @ ~/go/src/github.com/redhat-et/fetchit
pat-kube-check└─ $ podman logs -f fetchit
initializing fetchit
starting fetchit
2022-12-22T14:39:26.888Z    INFO    engine/fetchit.go:401   git clone https://github.com/containers/fetchit main --recursive
2022-12-22T14:39:31.475Z    INFO    engine/fetchit.go:365   Processing git target: https://github.com/containers/fetchit Method: kube Name: kube-ex
2022-12-22T14:39:31.612Z    INFO    engine/kube.go:90   Creating podman container from fetchit/examples/kube/1-pvc.yaml using kube method
2022-12-22T14:39:31.654Z    INFO    engine/kube.go:160  Created pods from spec in fetchit/examples/kube/1-pvc.yaml
2022-12-22T14:39:31.654Z    INFO    engine/kube.go:90   Creating podman container from fetchit/examples/kube/2-example.yaml using kube method
2022-12-22T14:39:38.441Z    INFO    engine/kube.go:160  Created pods from spec in fetchit/examples/kube/2-example.yaml
2022-12-22T14:39:38.441Z    INFO    engine/kube.go:90   Creating podman container from fetchit/examples/kube/3-example.yaml using kube method
rcook  @ ~/go/src/github.com/redhat-et/fetchit
pat-kube-check└─ $ cat /home/rcook/.fetchit/config.yaml 
# for this test, start with this config, then wait to be sure the
# targetConfigs from .fetchit/config.yaml are populated
# and for follow-up test, push a change to the config and confirm
# new targetConfigs are fetched & run
targetConfigs:
- url: https://github.com/containers/fetchit
  pat: MYSECRETPAT
  kube:
  - name: kube-ex
    targetPath: examples/kube
    schedule: "*/1 * * * *"
  branch: main
alexanderniebuhr commented 1 year ago

@cooktheryan, I tried this and it does not work for private repositories in my case. (I tested it with the repo set to public for just a minute and it gave another error). I think it does not work for private repositories, because the fetch for commit data is called without pat. Therefore the url to the repo will return 404, because the repo is private.

private repo

[root@applications ~]# cat .fetchit/config.yaml 
targetConfigs:
- url: https://github.com/alexanderniebuhr/lab
  pat: ghp_XXXXXXXXXXXXXXXXXXXXXXXXXXX
  kube:
  - name: kube-ex
    targetPath: apps/home-assistant.yaml
    schedule: "*/1 * * * *"
  branch: main
[root@applications ~]# podman run -d --rm --name fetchit -v fetchit-volume:/opt -v $HOME/.fetchit:/opt/mount -v /run/podman/podman.sock:/run/podman/podman.sock --privileged quay.io/fetchit/fetchit:latest
b10bf00918c1fef67f96c9148859cdaf24f879d13593762221ec84dbc8e40a0a
[root@applications ~]# podman logs -f fetchit
initializing fetchit
starting fetchit
2022-12-22T16:25:45.183Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-22T16:25:45.391Z        INFO    engine/fetchit.go:365   Processing git target: https://github.com/alexanderniebuhr/lab Method: kube Name: kube-ex
2022-12-22T16:25:45.391Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-22T16:25:45.515Z        ERROR   engine/kube.go:55       Error moving to current: Failed to get current commit: Error opening repository lab to fetch current commit: repository does not exist
2022-12-22T16:26:00.004Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-22T16:26:00.145Z        ERROR   engine/kube.go:55       Error moving to current: Failed to get current commit: Error opening repository lab to fetch current commit: repository does not exist

public repo

[root@applications ~]# podman logs -f fetchit
initializing fetchit
starting fetchit
2022-12-22T16:28:55.960Z        INFO    engine/fetchit.go:401   git clone https://github.com/alexanderniebuhr/lab main --recursive
2022-12-22T16:28:56.424Z        INFO    engine/fetchit.go:365   Processing git target: https://github.com/alexanderniebuhr/lab Method: kube Name: kube-ex
2022-12-22T16:28:56.564Z        ERROR   engine/kube.go:62       Error moving current to latest: Failed to apply changes: Error getting tree from hash ae71047c3889ea00b58f183e8b9feef6ce31aa16: Error getting sub tree at apps/home-assistant.yaml from commit at ae71047c3889ea00b58f183e8b9feef6ce31aa16 from repository lab: directory not found
2022-12-22T16:29:00.173Z        ERROR   engine/kube.go:62       Error moving current to latest: Failed to apply changes: Error getting tree from hash ae71047c3889ea00b58f183e8b9feef6ce31aa16: Error getting sub tree at apps/home-assistant.yaml from commit at ae71047c3889ea00b58f183e8b9feef6ce31aa16 from repository lab: directory not found
alexanderniebuhr commented 1 year ago

Interestingly the PAT Token is not used. I am confused?

image
cooktheryan commented 1 year ago

There was an error in the docs and the test case after a merge happened that changed the structure. I pushed a fix #283 sorry about that. Ill try to look into the networking piece before break

alexanderniebuhr commented 1 year ago

@cooktheryan is it supposed to work with the changed syntax and private repositories. I still can't fetch any commit data if the repository is private?

cooktheryan commented 1 year ago

Was just looking at it and ran a test locally. So I had classic token without repo access and I got the following.

podman logs -f fetchit
initializing fetchit
starting fetchit
2022-12-26T20:27:39.254Z    INFO    engine/fetchit.go:401   git clone https://github.com/cooktheryan/aap-rhacm main --recursive
2022-12-26T20:27:39.468Z    INFO    engine/fetchit.go:365   Processing git target: https://github.com/cooktheryan/aap-rhacm Method: kube Name: kube-ex
2022-12-26T20:27:39.468Z    INFO    engine/fetchit.go:401   git clone https://github.com/cooktheryan/aap-rhacm main --recursive
2022-12-26T20:27:39.538Z    ERROR   engine/kube.go:55   Error moving to current: Failed to get current commit: Error opening repository aap-rhacm to fetch current commit: repository does not exist

Then I enabled repo access for the PAT and then things cleared up

2022-12-26T20:28:00.010Z    INFO    engine/fetchit.go:401   git clone https://github.com/cooktheryan/aap-rhacm main --recursive
2022-12-26T20:28:00.417Z    INFO    engine/kube.go:90   Creating podman container from aap-rhacm/examples/kube/1-pvc.yaml using kube method
2022-12-26T20:28:00.527Z    INFO    engine/kube.go:160  Created pods from spec in aap-rhacm/examples/kube/1-pvc.yaml
2022-12-26T20:28:00.527Z    INFO    engine/kube.go:90   Creating podman container from aap-rhacm/examples/kube/2-example.yaml using kube method
2022-12-26T20:28:01.597Z    INFO    engine/kube.go:160  Created pods from spec in aap-rhacm/examples/kube/2-example.yaml
2022-12-26T20:28:01.597Z    INFO    engine/kube.go:90   Creating podman container from aap-rhacm/examples/kube/3-example.yaml using kube method
2022-12-26T20:28:03.500Z    INFO    engine/kube.go:160  Created pods from spec in aap-rhacm/examples/kube/3-example.yaml
2022-12-26T20:28:03.501Z    INFO    engine/common.go:97 Moved kube-ex from 000000000 to 5bb7fd06bc1279a68ec0137a283a544fab064113 for git target https://github.com/cooktheryan/aap-rhacm

2022-12-26T20:29:00.129Z    INFO    engine/common.go:99 No changes applied to git target aap-rhacm this run, kube currently at 5bb7fd06b
2022-12-26T20:30:00.131Z    INFO    engine/common.go:99 No changes applied to git target aap-rhacm this run, kube currently at 5bb7fd06b

image

cooktheryan commented 1 year ago

If we can't get this worked out by the 3 Jan I would be willing to jump on a call. I think this could benefit the HA community

alexanderniebuhr commented 1 year ago

Okay just regenerated a new token, now it works.. Even if the old token had all the permissions, as you could see in the screenshot. Maybe it was generated too long ago. Now I just need #280, to be able to use this for everything.