search

go-gitea / gitea

Git with a cup of tea! Painless self-hosted all-in-one software development service, including Git hosting, code review, team collaboration, package registry and CI/CD
https://gitea.com
MIT License
43.87k stars 5.38k forks source link

LFS locking API issue with fcgi+unix protocol #12215

Open vvrein opened 4 years ago

vvrein commented 4 years ago

Description

Not working config:

Gitea configuration

[server]
PROTOCOL                    = fcgi+unix
DOMAIN                         = https://gitea.tort.icu
HTTP_ADDR                   = /home/gitea/gitea.sock
SSH_DOMAIN                = gitea.tort.icu
ROOT_URL                      = https://gitea.tort.icu/
DISABLE_SSH                 = false
START_SSH_SERVER      = false
SSH_PORT                      = 22
LFS_START_SERVER       = true
LFS_CONTENT_PATH     = /home/gitea/app/data/lfs
LFS_JWT_SECRET            =  ***
LFS_HTTP_AUTH_EXPIRY   = 30m
LFS_MAX_FILE_SIZE            = 0
LFS_LOCK_PAGING_NUM  = 100
OFFLINE_MODE                  = true
DISABLE_ROUTER_LOG     = true
STATIC_URL_PREFIX           = /yV5kAhH

Nginx configuration

        location / {
                include fastcgi_params;
                fastcgi_pass unix:/home/gitea/gitea.sock;
        }

Produces output:

dgit lfs push origin master
Remote "origin" does not support the LFS locking API. Consider disabling it with:
  $ git config lfs.https://gitea.tort.icu/vrein/dotfiles.git/info/lfs.locksverify false

Nginx log:

138.201.246.200 - - [11/Jul/2020:18:50:18 +0300] "GET /api/internal/serv/command/6/vrein/dotfiles?mode=1&verb=git-upload-pack HTTP/1.1" 200 128 "-" "GiteaServer"
138.201.246.200 - - [11/Jul/2020:18:50:18 +0300] "POST /api/internal/ssh/6/update/1 HTTP/1.1" 200 38 "-" "GiteaServer"
138.201.246.200 - - [11/Jul/2020:18:50:19 +0300] "GET /api/internal/serv/command/6/vrein/dotfiles?mode=2&verb=git-lfs-authenticate&verb=upload HTTP/1.1" 200 128 "-" "GiteaServer"
45.129.99.75 - - [11/Jul/2020:18:50:50 +0300] "POST /vrein/dotfiles.git/info/lfs/locks/verify HTTP/2.0" 200 0 "-" "git-lfs/2.11.0 (GitHub; linux amd64; go 1.14.4)"

Working config:

Gitea configuration

[server]
PROTOCOL                   = http
DOMAIN                       = https://gitea.tort.icu
HTTP_ADDR                 = 127.0.0.1
HTTP_PORT                 = 3000
SSH_DOMAIN             = gitea.tort.icu
ROOT_URL                   = https://gitea.tort.icu/
DISABLE_SSH              = false
START_SSH_SERVER    = false
SSH_PORT                    = 22
LFS_START_SERVER     = true
LFS_CONTENT_PATH   = /home/gitea/app/data/lfs
LFS_JWT_SECRET         = ***
LFS_HTTP_AUTH_EXPIRY = 30m
LFS_MAX_FILE_SIZE           = 0
LFS_LOCK_PAGING_NUM = 100
OFFLINE_MODE                 = true
DISABLE_ROUTER_LOG    = true
STATIC_URL_PREFIX          = /yV5kAhH
Nginx configuration
        location / {
                #include fastcgi_params;
                #fastcgi_pass unix:/home/gitea/gitea.sock;
                proxy_pass http://127.0.0.1:3000;
        }

Produces output:

dgit lfs push origin master
Locking support detected on remote "origin". Consider enabling it with:
  $ git config lfs.https://gitea.tort.icu/vrein/dotfiles.git/info/lfs.locksverify true

Nginx log:

45.129.99.75 - - [11/Jul/2020:18:53:18 +0300] "POST /vrein/dotfiles.git/info/lfs/locks/verify HTTP/2.0" 200 38 "-" "git-lfs/2.11.0 (GitHub; linux amd64; go 1.14.4)

dgit is an alias: dgit='git --git-dir "${HOME}"/.dotfiles/.git --work-tree=${HOME}'

Screenshots

Locking works after config change: image

zeripath commented 4 years ago

This is odd. I'm not immediately sure how FCGI could be responsible for this... I suspect the issue is that Gitea is not being passed the authorization headers.

zeripath commented 4 years ago

Another option is that the "WWW-Authenticate" header although being set by Gitea is not being passed back to the client

vvrein commented 4 years ago

1) I've checked "WWW-Authenticate" headers.. aaaand without any luck

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"'
                      '"$sent_http_www_authenticate" "$http_www_authenticate"';
138.201.246.200 - - [12/Jul/2020:02:25:51 +0300] "GET /api/internal/serv/command/6/vrein/dotfiles?mode=1&verb=git-upload-pack HTTP/1.1" 200 128 "-" "GiteaServer" "-""-" "-"
138.201.246.200 - - [12/Jul/2020:02:25:51 +0300] "POST /api/internal/ssh/6/update/1 HTTP/1.1" 200 38 "-" "GiteaServer" "-""-" "-"
138.201.246.200 - - [12/Jul/2020:02:25:52 +0300] "GET /api/internal/serv/command/6/vrein/dotfiles?mode=2&verb=git-lfs-authenticate&verb=upload HTTP/1.1" 200 128 "-" "GiteaServer" "-""-" "-"
45.129.99.75 - - [12/Jul/2020:02:26:23 +0300] "POST /vrein/dotfiles.git/info/lfs/locks/verify HTTP/2.0" 200 0 "-" "git-lfs/2.11.0 (GitHub; linux amd64; go 1.14.4)" "-""-" "-"

2) I think (but certainly not sure), that Gitea should on its own go to its socket, and do that whatever stuff there. Some time ago, near 6-10 moths, I was not able to push/pull to the gitea repo, when HTTP_ADDR was set to unix socket, and I was able get around this with setting LOCAL_ROOT_URL to separate nginx "localhost only" host :) And after some time this was fixed. Buuut.... As of writing this, I'm less and less sure that Gitea was going to its unix socket, and not "nginx front" :)

vvrein commented 4 years ago

Seems yet Gitea can somehow authenticate itself, since making same request from gitea host by curl or from browser returns 403 error

zeripath commented 4 years ago

@vvrein look at the IP addresses you'll see that git-lfs sends its lock requests using http to connect directly to the server even if you are using SSH. It therefore needs to be able to authenticate over http - usually using a token that is passed back to the git-lfs process following GET /api/internal/serv/command/6/vrein/dotfiles?mode=2&verb=git-lfs-authenticate&verb=upload.

Git LFS will then POST to /vrein/dotfiles.git/info/lfs/locks/verify to check the locks with that token in a Authorization header. Now if that doesn't get passed to Gitea then that won't work.

In particular Gitea requires an Accept header that matches application/vnd.git-lfs+json and the Authorization header.

It would be helpful to see the actual Gitea logs rather than the Nginx logs or Gitea's router logs. In particular it would be useful to see if there are Warn level logs with Authentication failure for provided token with Error from modules/lfs/server.go.

vvrein commented 4 years ago

@zeripath Sorry for my inactivity with this issue, I was busy by a bunch of offline tasks.

In returning to this:

vvrein commented 4 years ago

Logging config:

[database]
......
LOG_SQL  = false

...

[log]
MODE      = file
LEVEL     = Trace
ROOT_PATH = /home/gitea/app/log

Logs from gitea.log immediately after doing dgit lfs push origin master

2020/07/19 09:17:40 ...s/auth/sso/oauth2.go:33:CheckOAuthAccessToken() [T] ParseOAuth2Token: signature is invalid
2020/07/19 09:17:40 ...s/context/context.go:331:func1() [D] Session ID: 68fc3b5db3d6d106
2020/07/19 09:17:40 ...s/context/context.go:332:func1() [D] CSRF Token: R4SPJU16Q0drNeCqCjedC0ze3-o6MTU5NTEzOTQ2MDg1MjI5MTU3MQ
2020/07/19 09:17:40 ...ters/private/serv.go:363:ServCommand() [D] Serv Results:
        IsWiki: false
        IsDeployKey: false
        KeyID: 3        KeyName: vrein@tokk_ed25519
        UserName: vrein
        UserID: 1
        OwnerName: vrein
        RepoName: dotfiles
        RepoID: 1
2020/07/19 09:17:40 ...s/auth/sso/oauth2.go:33:CheckOAuthAccessToken() [T] ParseOAuth2Token: signature is invalid
2020/07/19 09:17:40 ...s/context/context.go:331:func1() [D] Session ID: 950123538d707547
2020/07/19 09:17:40 ...s/context/context.go:332:func1() [D] CSRF Token: 5-sAlaf_O33lTvTTG-5Hc26cFL86MTU5NTEzOTQ2MDkyNjk1MTExOA
2020/07/19 09:17:41 ...s/auth/sso/oauth2.go:33:CheckOAuthAccessToken() [T] ParseOAuth2Token: signature is invalid
2020/07/19 09:17:41 ...s/context/context.go:331:func1() [D] Session ID: 2105f6d6f89d0844
2020/07/19 09:17:41 ...s/context/context.go:332:func1() [D] CSRF Token: gBkw2VfXEtAzMpj4GRNDdZkNB6Y6MTU5NTEzOTQ2MTk5MjA5NTY2NA
2020/07/19 09:17:41 models/repo.go:410:getUnits() [T] repo.Units: [0xc0054a3da0 0xc0054a3e00 0xc0054a3f20 0xc00552c030 0xc00552c090]
2020/07/19 09:17:41 ...s/repo_permission.go:154:func1() [T] Permission Loaded for 824722536800:vrein in 824722536864:vrein/dotfiles:
        Permissions: AccessMode: 824722537696:owner, 5 Units, 0 UnitsMode(s): [
        Units[0]: ID: 824722537072 RepoID: 824722537104 Type: 824722537728:UnitTypeCode Config: {}
        Units[1]: ID: 824722537248 RepoID: 824722537280 Type: 824722537760:UnitTypeIssues Config: {"EnableTimetracker":true,"AllowOnlyContributorsToTrackTime":true,"EnableDependencies":true}
        Units[2]: ID: 824722537424 RepoID: 824722537456 Type: 824722537792:UnitTypePullRequests Config: {"IgnoreWhitespaceConflicts":false,"AllowMerge":true,"AllowRebase":true,"AllowRebaseMerge":true,"AllowSquash":true}
        Units[3]: ID: 824722537520 RepoID: 824722537552 Type: 824722537824:UnitTypeReleases Config: {}
        Units[4]: ID: 824722537616 RepoID: 824722537648 Type: 824722537856:UnitTypeWiki Config: {} ]
2020/07/19 09:17:41 ...ters/private/serv.go:363:ServCommand() [D] Serv Results:
        IsWiki: false
        IsDeployKey: false
        KeyID: 3        KeyName: vrein@tokk_ed25519
        UserName: vrein
        UserID: 1
        OwnerName: vrein
        RepoName: dotfiles
        RepoID: 1
2020/07/19 09:17:42 ...s/context/context.go:331:func1() [D] Session ID: 540f50f2157779ab
2020/07/19 09:17:42 ...s/context/context.go:332:func1() [D] CSRF Token: GQhlmoBjm5ocJkb0mjS7RvywBYQ6MTU5NTEzOTQ2MjM4MTAzNjYwMg
2020/07/19 09:17:42 models/repo.go:410:getUnits() [T] repo.Units: [0xc00552dcb0 0xc00552dd10 0xc00552dd70 0xc00552ddd0 0xc00552de30]
2020/07/19 09:17:42 ...s/repo_permission.go:154:func1() [T] Permission Loaded for 824723282240:vrein in 824723282304:vrein/dotfiles:
        Permissions: AccessMode: 824723282896:owner, 5 Units, 0 UnitsMode(s): [
        Units[0]: ID: 824723282432 RepoID: 824723282464 Type: 824723282928:UnitTypeCode Config: {}
        Units[1]: ID: 824723282528 RepoID: 824723282560 Type: 824723282960:UnitTypeIssues Config: {"EnableTimetracker":true,"AllowOnlyContributorsToTrackTime":true,"EnableDependencies":true}
        Units[2]: ID: 824723282624 RepoID: 824723282656 Type: 824723282992:UnitTypePullRequests Config: {"IgnoreWhitespaceConflicts":false,"AllowMerge":true,"AllowRebase":true,"AllowRebaseMerge":true,"AllowSquash":true}
        Units[3]: ID: 824723282720 RepoID: 824723282752 Type: 824723283024:UnitTypeReleases Config: {}
        Units[4]: ID: 824723282816 RepoID: 824723282848 Type: 824723283056:UnitTypeWiki Config: {} ]

Could auth/sso/oauth2.go:33:CheckOAuthAccessToken() [T] ParseOAuth2Token: signature is invalid be the issue?

zeripath commented 4 years ago

Hi! Sorry I've been a little busy with work.

The ParseOAuth2Token trace message is normal and is not the problem here.

Let's quickly go back to what happens when you push over SSH. We expect that gitea will get the following requests:

FROM_SSH "GET /api/internal/serv/command/{keyID}/{owner}/{reponame}?mode=1&verb=git-upload-pack"
FROM_SSH "POST /api/internal/ssh/6/update/1"
FROM_SSH "GET /api/internal/serv/command/{keyID}/{owner}/{reponame}?mode=2&verb=git-lfs-authenticate&verb=upload"
OVER_HTTP "POST /{owner}/{reponame}.git/info/lfs/locks/verify"

The first 3 - do not go through your proxy - they are direct communications between Gitea web and Gitea serv.

The third returns a secret token that is passed back to lfs communicate over the POST as the header Authorization: token xxxxxx

The last fourth is the interesting one. The logs that I presume relate to this request are:

2020/07/19 09:17:42 ...s/context/context.go:331:func1() [D] Session ID: 540f50f2157779ab
2020/07/19 09:17:42 ...s/context/context.go:332:func1() [D] CSRF Token: GQhlmoBjm5ocJkb0mjS7RvywBYQ6MTU5NTEzOTQ2MjM4MTAzNjYwMg
2020/07/19 09:17:42 models/repo.go:410:getUnits() [T] repo.Units: [0xc00552dcb0 0xc00552dd10 0xc00552dd70 0xc00552ddd0 0xc00552de30]
2020/07/19 09:17:42 ...s/repo_permission.go:154:func1() [T] Permission Loaded for 824723282240:vrein in 824723282304:vrein/dotfiles:
        Permissions: AccessMode: 824723282896:owner, 5 Units, 0 UnitsMode(s): [
        Units[0]: ID: 824723282432 RepoID: 824723282464 Type: 824723282928:UnitTypeCode Config: {}
        Units[1]: ID: 824723282528 RepoID: 824723282560 Type: 824723282960:UnitTypeIssues Config: {"EnableTimetracker":true,"AllowOnlyContributorsToTrackTime":true,"EnableDependencies":true}
        Units[2]: ID: 824723282624 RepoID: 824723282656 Type: 824723282992:UnitTypePullRequests Config: {"IgnoreWhitespaceConflicts":false,"AllowMerge":true,"AllowRebase":true,"AllowRebaseMerge":true,"AllowSquash":true}
        Units[3]: ID: 824723282720 RepoID: 824723282752 Type: 824723283024:UnitTypeReleases Config: {}
        Units[4]: ID: 824723282816 RepoID: 824723282848 Type: 824723283056:UnitTypeWiki Config: {} ]

Which actually implies that the Authorization succeeds - and you are detected as the owner.

Which leads to the question as to what is going on.

I wonder if the Content-Type response is not being passed back out of the FCGI?

Gitea should be getting the Request Content-Type otherwise we would have [I] Attempt access LOCKs without accepting the correct media type: application/vnd.git-lfs+json

vvrein commented 4 years ago

@zeripath Thanks! I will try to dig into FCGI <--> nginx headers on this week.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.