search

netlify / git-gateway

A Gateway to Git APIs
MIT License
404 stars 89 forks source link

cannot get files from netlifycms admin panel using Gitlab repo setup #44

Closed artemtech closed 3 years ago

artemtech commented 4 years ago

- Do you want to request a feature or report a bug? i think it is bug

- What is the current behavior? self hosted git-gateway cannot get files on gitlab repository, it return 404 status when fetching contents, especially inside a folder.

working condition example: https://gitlab.com/api/v4/projects/myprojectid/repository/files/content%2Fproduct%2F2015-01-04-first-post.md/raw?ref=dev

but git-gateway when i see the log of git-gateway instance, the %2F turned into / like this: https://gitlab.com/api/v4/projects/myprojectid/repository/files/content/product/2015-01-04-first-post.md/raw?ref=dev

- If the current behavior is a bug, please provide the steps to reproduce.

  1. setup git-gateway with gitlab credentials, using personal_access_token and project_id, example: 
    GITGATEWAY_GITLAB_ACCESS_TOKEN=your_access_token
GITGATEWAY_GITLAB_ACCESS_TOKEN_TYPE=personal_access
GITGATEWAY_GITLAB_REPO=your_project_id
  2. in browser, try to open collections in netlifycms admin panel, and keep monitoring your git-gateway log
  3. check Proxying to Gitlab in git-gateway log, it turned %2F into / in target file, and status=404

- What is the expected behavior? git-gateway can fetch the content, with url inside folder keep the %2F instead of converting it into / reference: https://docs.gitlab.com/ce/api/repository_files.html#get-file-from-repository

- Please mention your Go version, and operating system version. dockerized git-gateway: go version go1.13.5 linux/amd64 (Debian 10 Buster)

mraerino commented 4 years ago

I was not able to easily gather from the code why this is happening. We are using a reverse-proxy in front of the service that may be involved with this. If you can provide more logs, I can maybe find something out.

Reproducing your setup is rather complicated.

heopz-vpf commented 4 years ago

i have same problem.. https://community.netlify.com/t/wrong-link-generation-to-gitlab-when-using-folders-subfolders-at-collection/6578/2

And still not working now

heopz-vpf commented 4 years ago

Is there any updates? How to fix it?

mraerino commented 4 years ago

If you provide more/better logs in any way, we can try to triage. I'm saying i cannot reproduce.

artemtech commented 4 years ago

So sorry for my late reply @mraerino . My problem actually same as @heopz-vpf described in forum. here im attached logs that generated by git-gateway at my server:

Attaching to netlifycms_git-gateway_1
git-gateway_1  |time="2020-01-16T06:19:39Z" level=info msg="git-gateway API started on: 0.0.0.0:9999" 
git-gateway_1  |time="2020-01-17T15:10:20Z" level=info msg="request started" component=api method=GET path=/settings referer="http://localhost:1313/" remote_addr="36.78.159.239:53164" request_id=faec1916-dccf-4cb9-a6e3-800f010c654d 
git-gateway_1  |time="2020-01-17T15:10:20Z" level=info msg="Getting auth token" 
git-gateway_1  |time="2020-01-17T15:10:20Z" level=info msg="Parsing JWT claims: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NzkyNzc0MTYsInN1YiI6IjE3ZTIzNGVlLTdhNDQtNDAyZi05ZThlLTE1NTcwNTg3MTU1ZSIsImVtYWlsIjoid2VhcmV6ZWN0ZXJAZ21haWwuY29tIiwiYXBwX21ldGFkYXRhIjp7fSwidXNlcl9tZXRhZGF0YSI6e319.kV4eI-0pZDJenEcZete9uog1-4dbWgRy5F-LL82xDUw" 
git-gateway_1  |time="2020-01-17T15:10:20Z" level=info msg="request completed" component=api duration=6727965 method=GET path=/settings referer="http://localhost:1313/" remote_addr="36.78.159.239:53164" request_id=faec1916-dccf-4cb9-a6e3-800f010c654d status=200 
git-gateway_1  |time="2020-01-17T15:10:25Z" level=info msg="request started" component=api method=GET path=/settings referer="http://localhost:1313/" remote_addr="36.78.159.239:53188" request_id=0c6d31ce-c67e-439c-9b36-3b4942697d94 
git-gateway_1  |time="2020-01-17T15:10:25Z" level=info msg="Getting auth token" 
git-gateway_1  |time="2020-01-17T15:10:25Z" level=info msg="Parsing JWT claims: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NzkyNzc0MTYsInN1YiI6IjE3ZTIzNGVlLTdhNDQtNDAyZi05ZThlLTE1NTcwNTg3MTU1ZSIsImVtYWlsIjoid2VhcmV6ZWN0ZXJAZ21haWwuY29tIiwiYXBwX21ldGFkYXRhIjp7fSwidXNlcl9tZXRhZGF0YSI6e319.kV4eI-0pZDJenEcZete9uog1-4dbWgRy5F-LL82xDUw" 
git-gateway_1  |time="2020-01-17T15:10:25Z" level=info msg="request completed" component=api duration=545133 method=GET path=/settings referer="http://localhost:1313/" remote_addr="36.78.159.239:53188" request_id=0c6d31ce-c67e-439c-9b36-3b4942697d94 status=200 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="request started" component=api method=HEAD path=/gitlab/repository/tree referer="http://localhost:1313/" remote_addr="36.78.159.239:53208" request_id=39050242-27fa-40c1-b731-707f29883d22 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="Getting auth token" 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="Parsing JWT claims: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NzkyNzc0MTYsInN1YiI6IjE3ZTIzNGVlLTdhNDQtNDAyZi05ZThlLTE1NTcwNTg3MTU1ZSIsImVtYWlsIjoid2VhcmV6ZWN0ZXJAZ21haWwuY29tIiwiYXBwX21ldGFkYXRhIjp7fSwidXNlcl9tZXRhZGF0YSI6e319.kV4eI-0pZDJenEcZete9uog1-4dbWgRy5F-LL82xDUw" 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="Proxying to GitLab: https://gitlab.com/api/v4/projects/15871690/repository/tree?path=content%2Fartikel&ref=dev&recursive=false&ts=1579273821111" component=api method=HEAD path=/gitlab/repository/tree referer="http://localhost:1313/" remote_addr="36.78.159.239:53208" request_id=39050242-27fa-40c1-b731-707f29883d22 token_type=personal_access 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="request started" component=api method=GET path=/gitlab/repository/tree referer="http://localhost:1313/" remote_addr="36.78.159.239:53214" request_id=8d3d77f5-96b6-4135-bdd3-52e601f6f50e 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="Getting auth token" 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="Parsing JWT claims: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NzkyNzc0MTYsInN1YiI6IjE3ZTIzNGVlLTdhNDQtNDAyZi05ZThlLTE1NTcwNTg3MTU1ZSIsImVtYWlsIjoid2VhcmV6ZWN0ZXJAZ21haWwuY29tIiwiYXBwX21ldGFkYXRhIjp7fSwidXNlcl9tZXRhZGF0YSI6e319.kV4eI-0pZDJenEcZete9uog1-4dbWgRy5F-LL82xDUw" 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="Proxying to GitLab: https://gitlab.com/api/v4/projects/15871690/repository/tree?path=static%2Fuploads&ref=dev&per_page=100&recursive=false&ts=1579273821152" component=api method=GET path=/gitlab/repository/tree referer="http://localhost:1313/" remote_addr="36.78.159.239:53214" request_id=8d3d77f5-96b6-4135-bdd3-52e601f6f50e token_type=personal_access 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="request started" component=api method=GET path=/gitlab/repository/files/.lfsconfig/raw referer="http://localhost:1313/" remote_addr="36.78.159.239:53220" request_id=93ec682b-c0a3-4d26-b387-2ec7c3134c43 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="Getting auth token" 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="Parsing JWT claims: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NzkyNzc0MTYsInN1YiI6IjE3ZTIzNGVlLTdhNDQtNDAyZi05ZThlLTE1NTcwNTg3MTU1ZSIsImVtYWlsIjoid2VhcmV6ZWN0ZXJAZ21haWwuY29tIiwiYXBwX21ldGFkYXRhIjp7fSwidXNlcl9tZXRhZGF0YSI6e319.kV4eI-0pZDJenEcZete9uog1-4dbWgRy5F-LL82xDUw" 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="Proxying to GitLab: https://gitlab.com/api/v4/projects/15871690/repository/files/.lfsconfig/raw?ref=dev&ts=1579273821152" component=api method=GET path=/gitlab/repository/files/.lfsconfig/raw referer="http://localhost:1313/" remote_addr="36.78.159.239:53220" request_id=93ec682b-c0a3-4d26-b387-2ec7c3134c43 token_type=personal_access 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="request started" component=api method=GET path=/gitlab/repository/files/.gitattributes/raw referer="http://localhost:1313/" remote_addr="36.78.159.239:53226" request_id=c944ee3e-d73e-4cc7-93d4-1bee8404451f 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="Getting auth token" 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="Parsing JWT claims: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NzkyNzc0MTYsInN1YiI6IjE3ZTIzNGVlLTdhNDQtNDAyZi05ZThlLTE1NTcwNTg3MTU1ZSIsImVtYWlsIjoid2VhcmV6ZWN0ZXJAZ21haWwuY29tIiwiYXBwX21ldGFkYXRhIjp7fSwidXNlcl9tZXRhZGF0YSI6e319.kV4eI-0pZDJenEcZete9uog1-4dbWgRy5F-LL82xDUw" 
git-gateway_1  |time="2020-01-17T15:10:26Z" level=info msg="Proxying to GitLab: https://gitlab.com/api/v4/projects/15871690/repository/files/.gitattributes/raw?ref=dev&ts=1579273821153" component=api method=GET path=/gitlab/repository/files/.gitattributes/raw referer="http://localhost:1313/" remote_addr="36.78.159.239:53226" request_id=c944ee3e-d73e-4cc7-93d4-1bee8404451f token_type=personal_access 
git-gateway_1  |time="2020-01-17T15:10:27Z" level=info msg="request completed" component=api duration=1143631060 gitlab_lb=fe-09-lb-gprd gitlab_ratelimit_remaining= gitlab_request_id= method=HEAD path=/gitlab/repository/tree referer="http://localhost:1313/" remote_addr="36.78.159.239:53208" request_id=39050242-27fa-40c1-b731-707f29883d22 status=200 
git-gateway_1  |time="2020-01-17T15:10:27Z" level=info msg="request completed" component=api duration=1115255290 gitlab_lb=fe-22-lb-gprd gitlab_ratelimit_remaining= gitlab_request_id= method=GET path=/gitlab/repository/files/.lfsconfig/raw referer="http://localhost:1313/" remote_addr="36.78.159.239:53220" request_id=93ec682b-c0a3-4d26-b387-2ec7c3134c43 status=404 
git-gateway_1  |time="2020-01-17T15:10:27Z" level=info msg="request completed" component=api duration=1050291557 gitlab_lb=fe-22-lb-gprd gitlab_ratelimit_remaining= gitlab_request_id= method=GET path=/gitlab/repository/files/.gitattributes/raw referer="http://localhost:1313/" remote_addr="36.78.159.239:53226" request_id=c944ee3e-d73e-4cc7-93d4-1bee8404451f status=404 
git-gateway_1  |time="2020-01-17T15:10:27Z" level=info msg="request completed" component=api duration=1140851702 gitlab_lb=fe-08-lb-gprd gitlab_ratelimit_remaining= gitlab_request_id= method=GET path=/gitlab/repository/tree referer="http://localhost:1313/" remote_addr="36.78.159.239:53214" request_id=8d3d77f5-96b6-4135-bdd3-52e601f6f50e status=200 
git-gateway_1  |time="2020-01-17T15:10:28Z" level=info msg="request started" component=api method=GET path=/gitlab/repository/tree referer="http://localhost:1313/" remote_addr="36.78.159.239:53236" request_id=42debb71-a0c2-467c-a8b4-6a76238d016a 
git-gateway_1  |time="2020-01-17T15:10:28Z" level=info msg="Getting auth token" 
git-gateway_1  |time="2020-01-17T15:10:28Z" level=info msg="Parsing JWT claims: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NzkyNzc0MTYsInN1YiI6IjE3ZTIzNGVlLTdhNDQtNDAyZi05ZThlLTE1NTcwNTg3MTU1ZSIsImVtYWlsIjoid2VhcmV6ZWN0ZXJAZ21haWwuY29tIiwiYXBwX21ldGFkYXRhIjp7fSwidXNlcl9tZXRhZGF0YSI6e319.kV4eI-0pZDJenEcZete9uog1-4dbWgRy5F-LL82xDUw" 
git-gateway_1  |time="2020-01-17T15:10:28Z" level=info msg="Proxying to GitLab: https://gitlab.com/api/v4/projects/15871690/repository/tree?id=15871690&page=1&path=content%2Fartikel&per_page=20&recursive=false&ref=dev&ts=1579273823289" component=api method=GET path=/gitlab/repository/tree referer="http://localhost:1313/" remote_addr="36.78.159.239:53236" request_id=42debb71-a0c2-467c-a8b4-6a76238d016a token_type=personal_access 
git-gateway_1  |time="2020-01-17T15:10:29Z" level=info msg="request completed" component=api duration=1076686781 gitlab_lb=fe-22-lb-gprd gitlab_ratelimit_remaining= gitlab_request_id= method=GET path=/gitlab/repository/tree referer="http://localhost:1313/" remote_addr="36.78.159.239:53236" request_id=42debb71-a0c2-467c-a8b4-6a76238d016a status=200 
git-gateway_1  |time="2020-01-17T15:10:31Z" level=info msg="request started" component=api method=GET path=/gitlab/repository/files/content/artikel/artikel-dari-cms.md/raw referer="http://localhost:1313/" remote_addr="36.78.159.239:53252" request_id=cd3cc98f-ba00-4120-b61e-3243b732e2a4 
git-gateway_1  |time="2020-01-17T15:10:31Z" level=info msg="Getting auth token" 
git-gateway_1  |time="2020-01-17T15:10:31Z" level=info msg="Parsing JWT claims: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NzkyNzc0MTYsInN1YiI6IjE3ZTIzNGVlLTdhNDQtNDAyZi05ZThlLTE1NTcwNTg3MTU1ZSIsImVtYWlsIjoid2VhcmV6ZWN0ZXJAZ21haWwuY29tIiwiYXBwX21ldGFkYXRhIjp7fSwidXNlcl9tZXRhZGF0YSI6e319.kV4eI-0pZDJenEcZete9uog1-4dbWgRy5F-LL82xDUw" 
git-gateway_1  |time="2020-01-17T15:10:31Z" level=info msg="Proxying to GitLab: https://gitlab.com/api/v4/projects/15871690/repository/files/content/artikel/artikel-dari-cms.md/raw?ref=dev&ts=1579273825632" component=api method=GET path=/gitlab/repository/files/content/artikel/artikel-dari-cms.md/raw referer="http://localhost:1313/" remote_addr="36.78.159.239:53252" request_id=cd3cc98f-ba00-4120-b61e-3243b732e2a4 token_type=personal_access 
git-gateway_1  |time="2020-01-17T15:10:31Z" level=info msg="request started" component=api method=GET path=/gitlab/repository/files/content/artikel/2015-01-04-first-post.md/raw referer="http://localhost:1313/" remote_addr="36.78.159.239:53256" request_id=8dd960f1-41dd-4bdb-a381-0e5b016c3569 
git-gateway_1  |time="2020-01-17T15:10:31Z" level=info msg="Getting auth token" 
git-gateway_1  |time="2020-01-17T15:10:31Z" level=info msg="Parsing JWT claims: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NzkyNzc0MTYsInN1YiI6IjE3ZTIzNGVlLTdhNDQtNDAyZi05ZThlLTE1NTcwNTg3MTU1ZSIsImVtYWlsIjoid2VhcmV6ZWN0ZXJAZ21haWwuY29tIiwiYXBwX21ldGFkYXRhIjp7fSwidXNlcl9tZXRhZGF0YSI6e319.kV4eI-0pZDJenEcZete9uog1-4dbWgRy5F-LL82xDUw" 
git-gateway_1  |time="2020-01-17T15:10:31Z" level=info msg="Proxying to GitLab: https://gitlab.com/api/v4/projects/15871690/repository/files/content/artikel/2015-01-04-first-post.md/raw?ref=dev&ts=1579273825635" component=api method=GET path=/gitlab/repository/files/content/artikel/2015-01-04-first-post.md/raw referer="http://localhost:1313/" remote_addr="36.78.159.239:53256" request_id=8dd960f1-41dd-4bdb-a381-0e5b016c3569 token_type=personal_access 
git-gateway_1  |time="2020-01-17T15:10:31Z" level=info msg="request completed" component=api duration=251042320 gitlab_lb=fe-22-lb-gprd gitlab_ratelimit_remaining= gitlab_request_id= method=GET path=/gitlab/repository/files/content/artikel/artikel-dari-cms.md/raw referer="http://localhost:1313/" remote_addr="36.78.159.239:53252" request_id=cd3cc98f-ba00-4120-b61e-3243b732e2a4 status=404 
git-gateway_1  |time="2020-01-17T15:10:31Z" level=info msg="request completed" component=api duration=245944148 gitlab_lb=fe-09-lb-gprd gitlab_ratelimit_remaining= gitlab_request_id= method=GET path=/gitlab/repository/files/content/artikel/2015-01-04-first-post.md/raw referer="http://localhost:1313/" remote_addr="36.78.159.239:53256" request_id=8dd960f1-41dd-4bdb-a381-0e5b016c3569 status=404 
git-gateway_1  |time="2020-01-17T15:10:31Z" level=info msg="request started" component=api method=GET path=/gitlab/repository/files/content/artikel/2015-02-20-test-markdown.md/raw referer="http://localhost:1313/" remote_addr="36.78.159.239:53260" request_id=abc3acbe-95e6-459b-8037-b725428a4200 
git-gateway_1  |time="2020-01-17T15:10:31Z" level=info msg="Getting auth token" 
git-gateway_1  |time="2020-01-17T15:10:31Z" level=info msg="Parsing JWT claims: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NzkyNzc0MTYsInN1YiI6IjE3ZTIzNGVlLTdhNDQtNDAyZi05ZThlLTE1NTcwNTg3MTU1ZSIsImVtYWlsIjoid2VhcmV6ZWN0ZXJAZ21haWwuY29tIiwiYXBwX21ldGFkYXRhIjp7fSwidXNlcl9tZXRhZGF0YSI6e319.kV4eI-0pZDJenEcZete9uog1-4dbWgRy5F-LL82xDUw" 
git-gateway_1  |time="2020-01-17T15:10:31Z" level=info msg="Proxying to GitLab: https://gitlab.com/api/v4/projects/15871690/repository/files/content/artikel/2015-02-20-test-markdown.md/raw?ref=dev&ts=1579273825634" component=api method=GET path=/gitlab/repository/files/content/artikel/2015-02-20-test-markdown.md/raw referer="http://localhost:1313/" remote_addr="36.78.159.239:53260" request_id=abc3acbe-95e6-459b-8037-b725428a4200 token_type=personal_access 
git-gateway_1  |time="2020-01-17T15:10:31Z" level=info msg="request completed" component=api duration=234625942 gitlab_lb=fe-09-lb-gprd gitlab_ratelimit_remaining= gitlab_request_id= method=GET path=/gitlab/repository/files/content/artikel/2015-02-20-test-markdown.md/raw referer="http://localhost:1313/" remote_addr="36.78.159.239:53260" request_id=abc3acbe-95e6-459b-8037-b725428a4200 status=404

and here a screenshot from firefox when successful login into dashboard, (see URL Params at Headers) firefox log

as you can see, the url params that "%2F" text in firefox turned into "/" in git-gateway log.

artemtech commented 3 years ago

it can be solved in webserver config level, avoid to decode request uri.