truenas / charts

TrueNAS SCALE Apps Catalogs & Charts
BSD 3-Clause "New" or "Revised" License
305 stars 294 forks source link

Gitea LFS push error 405 #2381

Closed SamWindell closed 6 months ago

SamWindell commented 6 months ago

TrueNAS Scale version Dragonfish-24.04-RC.1.

I have installed Gitea as an App. I am using datasets to store the apps data and I made those datasets have full permissions for the apps user. Everything is working great. Gitea web interface works at http://192.168.68.103:30008/. I can push/pull. However, if I git push with LFS data included in the commit. It always produces an error:

LFS: Client error &{%!!(string=http) %!!(string=) %!!(*url.Userinfo=<nil>) %!!(string=192.168.68.103) %!!(string=/ui/) %!!(string=) %!!(bool=false) %!!(bool=false) %!!(string=) %!!(string=) %!!(string=)}s(MISSING) from HTTP 405
error: failed to push some refs to 'http://192.168.68.103:30008/sam/test.git'

Also it takes ~10 seconds before giving the error, even if the repo only contains a few bytes of data. Pushing without LFS takes ~1 ms.

I haven't changed Gitea's app.ini config at all.

I'm new to all this so please direct me if needed.

My datasets showing full permissions for apps

Screenshot 2024-04-14 at 17 55 09

Possible repro

Gitea Docker Log Snippet

Here is a snippet from tailing the logs of the gitea docker. It seems to show a POST operation to sam/test.git/info/lfs/objects/batch for resulting in OK. I've never found these logs to contain any error at all.

2024-04-14T17:42:02.410329865+01:00 2024/04/14 16:42:02 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 172.16.0.1:34060, 200 OK in 0.1ms @ healthcheck/check.go:67(healthcheck.Check)
2024-04-14T17:42:02.474080067+01:00 2024/04/14 16:42:02 ...eb/routing/logger.go:102:func1() [I] router: completed POST /sam/test.git/info/lfs/objects/batch for 192.168.68.100:59512, 200 OK in 80.4ms @ lfs/server.go:156(lfs.BatchHandler)
2024-04-14T17:42:04.561937108+01:00 2024/04/14 16:42:04 ...eb/routing/logger.go:102:func1() [I] router: completed POST /sam/test.git/info/lfs/objects/batch for 192.168.68.100:59512, 200 OK in 43.3ms @ lfs/server.go:156(lfs.BatchHandler)
2024-04-14T17:42:08.651336681+01:00 2024/04/14 16:42:08 ...eb/routing/logger.go:102:func1() [I] router: completed POST /sam/test.git/info/lfs/objects/batch for 192.168.68.100:59512, 200 OK in 44.3ms @ lfs/server.go:156(lfs.BatchHandler)
2024-04-14T17:42:12.409740496+01:00 2024/04/14 16:42:12 ...c/net/http/server.go:2136:ServeHTTP() [I] PING DATABASE postgres
2024-04-14T17:42:12.409870589+01:00 2024/04/14 16:42:12 ...c/net/http/server.go:2136:ServeHTTP() [I] PING DATABASE postgres
2024-04-14T17:42:12.409930328+01:00 2024/04/14 16:42:12 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 172.16.0.1:56508, 200 OK in 0.2ms @ healthcheck/check.go:67(healthcheck.Check)
2024-04-14T17:42:12.409952490+01:00 2024/04/14 16:42:12 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 172.16.0.1:56510, 200 OK in 0.2ms @ healthcheck/check.go:67(healthcheck.Check)
2024-04-14T17:42:16.739371230+01:00 2024/04/14 16:42:16 ...eb/routing/logger.go:102:func1() [I] router: completed POST /sam/test.git/info/lfs/objects/batch for 192.168.68.100:59512, 200 OK in 45.3ms @ lfs/server.go:156(lfs.BatchHandler)
2024-04-14T17:42:22.409073780+01:00 2024/04/14 16:42:22 ...c/net/http/server.go:2136:ServeHTTP() [I] PING DATABASE postgres
2024-04-14T17:42:22.409151548+01:00 2024/04/14 16:42:22 ...c/net/http/server.go:2136:ServeHTTP() [I] PING DATABASE postgres
2024-04-14T17:42:22.409214129+01:00 2024/04/14 16:42:22 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 172.16.0.1:41568, 200 OK in 0.2ms @ healthcheck/check.go:67(healthcheck.Check)
2024-04-14T17:42:22.409231619+01:00 2024/04/14 16:42:22 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 172.16.0.1:41580, 200 OK in 0.2ms @ healthcheck/check.go:67(healthcheck.Check)
2024-04-14T17:42:26.828020046+01:00 2024/04/14 16:42:26 ...eb/routing/logger.go:102:func1() [I] router: completed POST /sam/test.git/info/lfs/objects/batch for 192.168.68.100:59512, 200 OK in 43.5ms @ lfs/server.go:156(lfs.BatchHandler)
2024-04-14T17:42:32.411030418+01:00 2024/04/14 16:42:32 ...c/net/http/server.go:2136:ServeHTTP() [I] PING DATABASE postgres
2024-04-14T17:42:32.411434222+01:00 2024/04/14 16:42:32 ...c/net/http/server.go:2136:ServeHTTP() [I] PING DATABASE postgres
2024-04-14T17:42:32.411806891+01:00 2024/04/14 16:42:32 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 172.16.0.1:42558, 200 OK in 0.9ms @ healthcheck/check.go:67(healthcheck.Check)
2024-04-14T17:42:32.411967758+01:00 2024/04/14 16:42:32 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 172.16.0.1:42572, 200 OK in 0.7ms @ healthcheck/check.go:67(healthcheck.Check)
2024-04-14T17:42:36.908743815+01:00 2024/04/14 16:42:36 ...eb/routing/logger.go:102:func1() [I] router: completed POST /sam/test.git/info/lfs/objects/batch for 192.168.68.100:59512, 200 OK in 45.5ms @ lfs/server.go:156(lfs.BatchHandler)
2024-04-14T17:42:42.409195793+01:00 2024/04/14 16:42:42 ...c/net/http/server.go:2136:ServeHTTP() [I] PING DATABASE postgres
2024-04-14T17:42:42.409289150+01:00 2024/04/14 16:42:42 ...c/net/http/server.go:2136:ServeHTTP() [I] PING DATABASE postgres
2024-04-14T17:42:42.409392943+01:00 2024/04/14 16:42:42 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 172.16.0.1:55412, 200 OK in 0.2ms @ healthcheck/check.go:67(healthcheck.Check)
2024-04-14T17:42:42.409414488+01:00 2024/04/14 16:42:42 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 172.16.0.1:55410, 200 OK in 0.2ms @ healthcheck/check.go:67(healthcheck.Check)
2024-04-14T17:42:52.409893418+01:00 2024/04/14 16:42:52 ...c/net/http/server.go:2136:ServeHTTP() [I] PING DATABASE postgres
2024-04-14T17:42:52.410010705+01:00 2024/04/14 16:42:52 ...c/net/http/server.go:2136:ServeHTTP() [I] PING DATABASE postgres
2024-04-14T17:42:52.410027679+01:00 2024/04/14 16:42:52 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 172.16.0.1:55626, 200 OK in 0.2ms @ healthcheck/check.go:67(healthcheck.Check)
2024-04-14T17:42:52.410181289+01:00 2024/04/14 16:42:52 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 172.16.0.1:55624, 200 OK in 0.2ms @ healthcheck/check.go:67(healthcheck.Check)
2024-04-14T17:43:02.409298029+01:00 2024/04/14 16:43:02 ...c/net/http/server.go:2136:ServeHTTP() [I] PING DATABASE postgres
2024-04-14T17:43:02.409409466+01:00 2024/04/14 16:43:02 ...c/net/http/server.go:2136:ServeHTTP() [I] PING DATABASE postgres
2024-04-14T17:43:02.409540511+01:00 2024/04/14 16:43:02 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 172.16.0.1:59724, 200 OK in 0.2ms @ healthcheck/check.go:67(healthcheck.Check)
2024-04-14T17:43:02.409672488+01:00 2024/04/14 16:43:02 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/healthz for 172.16.0.1:59722, 200 OK in 0.3ms @ healthcheck/check.go:67(healthcheck.Check)
stavros-k commented 6 months ago

Can you try running the push again with those flags added? Maybe we can see something there https://github.com/git-lfs/git-lfs/discussions/5190#discussioncomment-4159975

SamWindell commented 6 months ago

Thank you for taking a look. Here's the output git push with those flags. It looks like it might be the same problem as in that link. I can see in the output git-lfs making multiple retries. out.txt

stavros-k commented 6 months ago

Hmm I see it redirects to /ui which would explain the timeout then.

Have you enabled LFS with environment variables? I think it should be

key: GITEA__server__LFS_START_SERVER
value: true
SamWindell commented 6 months ago

I'm not sure what you mean about enabling LFS with environment variables.

I recognise the "LFS_START_SERVER" bit though, and I can see it's enabled in gitea's app.ini config, which I haven't manually changed at all. Here is what that file looks like.

root@truenas[~]# cd /mnt/Main\ Storage\ 2/gitea/config 
root@truenas[/mnt/Main Storage 2/gitea/config]# cat app.ini 
APP_NAME = FrozenVault
RUN_USER = git
RUN_MODE = prod
WORK_PATH = /var/lib/gitea

[repository]
ROOT = /var/lib/gitea/git/repositories

[repository.local]
LOCAL_COPY_PATH = /tmp/gitea/local-repo

[repository.upload]
TEMP_PATH = /tmp/gitea/uploads

[server]
APP_DATA_PATH = /var/lib/gitea
SSH_DOMAIN = 192.168.68.103
HTTP_PORT = 30008
ROOT_URL = http://192.168.68.103
DISABLE_SSH = false
; In rootless gitea container only internal ssh server is supported
START_SSH_SERVER = true
SSH_PORT = 30009
SSH_LISTEN_PORT = 30009
BUILTIN_SSH_SERVER_USER = git
LFS_START_SERVER = true
PROTOCOL = http
DOMAIN = 192.168.68.103
LFS_JWT_SECRET = redacted
OFFLINE_MODE = false

[database]
PATH = /var/lib/gitea/data/gitea.db
DB_TYPE = postgres
HOST = gitea-postgres
NAME = gitea
USER = gitea
PASSWD = redacted
SCHEMA = 
SSL_MODE = disable
LOG_SQL = false

[session]
PROVIDER_CONFIG = /var/lib/gitea/data/sessions
PROVIDER = file

[picture]
AVATAR_UPLOAD_PATH = /var/lib/gitea/data/avatars
REPOSITORY_AVATAR_UPLOAD_PATH = /var/lib/gitea/data/repo-avatars

[attachment]
PATH = /var/lib/gitea/data/attachments

[log]
ROOT_PATH = /var/lib/gitea/data/log
MODE = console
LEVEL = info

[security]
INSTALL_LOCK = true
SECRET_KEY = 
REVERSE_PROXY_LIMIT = 1
REVERSE_PROXY_TRUSTED_PROXIES = *
INTERNAL_TOKEN = redacted
PASSWORD_HASH_ALGO = pbkdf2

[service]
DISABLE_REGISTRATION = false
REQUIRE_SIGNIN_VIEW = false
REGISTER_EMAIL_CONFIRM = false
ENABLE_NOTIFY_MAIL = false
ALLOW_ONLY_EXTERNAL_REGISTRATION = false
ENABLE_CAPTCHA = false
DEFAULT_KEEP_EMAIL_PRIVATE = false
DEFAULT_ALLOW_CREATE_ORGANIZATION = true
DEFAULT_ENABLE_TIMETRACKING = true
NO_REPLY_ADDRESS = noreply.localhost

[lfs]
PATH = /var/lib/gitea/git/lfs

[mailer]
ENABLED = false

[openid]
ENABLE_OPENID_SIGNIN = true
ENABLE_OPENID_SIGNUP = true

[cron.update_checker]
ENABLED = false

[repository.pull-request]
DEFAULT_MERGE_STYLE = merge

[repository.signing]
DEFAULT_TRUST_MODEL = committer

[oauth2]
JWT_SECRET = redacted
root@truenas[/mnt/Main Storage 2/gitea/config]# 
stavros-k commented 6 months ago

Okay it looks like it's enabled.

Can you check your git config on your machine? I'm intersted in the lfs.url? You might have to manually define it, in order to include the port (30008 in your case)

SamWindell commented 6 months ago

Here is my local git config. Git and git-lfs are installed with Nix on my machines. I can try a non-Nix machine.

cat .config/git/config
[core]
        pager = "/nix/store/r4isn17qdn4nan8z7g67b2lasg5xfg96-delta-0.16.5/bin/delta"

[credential "https://gist.github.com"]
        helper = "!/nix/store/fi435ay6f8fsh14x5h63hpi7vck3flxc-gh-2.39.1/bin/gh auth git-credential"

[credential "https://github.com"]
        helper = "!/nix/store/fi435ay6f8fsh14x5h63hpi7vck3flxc-gh-2.39.1/bin/gh auth git-credential"

[filter "lfs"]
        clean = "git-lfs clean -- %f"
        process = "git-lfs filter-process"
        required = true
        smudge = "git-lfs smudge -- %f"

[init]
        defaultBranch = "main"

[interactive]
        diffFilter = "/nix/store/r4isn17qdn4nan8z7g67b2lasg5xfg96-delta-0.16.5/bin/delta --color-only"
stavros-k commented 6 months ago

Can you try setting lfs.url?

git config --add lfs.url "http://192.168.68.103:30008"
SamWindell commented 6 months ago

No luck I'm afraid. I'm on my mac machine now, and I tried it on Windows (not Nix) too.

After the config --add lfs.url, git-push works quicker now but still the same error.

❯ git config --add lfs.url "http://192.168.68.103:30008"

test main
❯ git config --list
credential.helper=osxkeychain
core.pager=/nix/store/55w70dh0m9yr5xzvdk6m23rdhkrl2s4b-delta-0.16.5/bin/delta
credential.https://gist.github.com.helper=!/nix/store/p4prr07sfhrg25k1qmsl99i3r34nc931-gh-2.39.1/bin/gh auth git-credential
credential.https://github.com.helper=!/nix/store/p4prr07sfhrg25k1qmsl99i3r34nc931-gh-2.39.1/bin/gh auth git-credential
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
filter.lfs.smudge=git-lfs smudge -- %f
interactive.difffilter=/nix/store/55w70dh0m9yr5xzvdk6m23rdhkrl2s4b-delta-0.16.5/bin/delta --color-only
user.email=info@frozenplain.com
user.name=Sam Windell
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.precomposeunicode=true
lfs.repositoryformatversion=0
lfs.url=http://192.168.68.103:30008
remote.origin.url=http://192.168.68.103:30008/sam/test.git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
lfs.http://192.168.68.103:30008/sam/test.git/info/lfs.access=basic

test main
git push -u origin main
Remote "origin" does not support the Git LFS locking API. Consider disabling it with:
  $ git config lfs.http://192.168.68.103:30008.locksverify false
Uploading LFS objects:   0% (0/1), 0 B | 0 B/s, done.
batch response: Client error &{%!!(string=http) %!!(string=) %!!(*url.Userinfo=<nil>) %!!(string=192.168.68.103:30008) %!!(string=/objects/batch) %!!(string=) %!!(bool=false) %!!(bool=false) %!!(string=) %!!(string=) %!!(string=)}s(MISSING) from HTTP 405
error: failed to push some refs to 'http://192.168.68.103:30008/sam/test.git'

❯ GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git push -u origin main
17:48:51.174614 git.c:463               trace: built-in: git push -u origin main
17:48:51.177393 run-command.c:659       trace: run_command: GIT_DIR=.git git remote-http origin http://192.168.68.103:30008/sam/test.git
17:48:51.191184 git.c:749               trace: exec: git-remote-http origin http://192.168.68.103:30008/sam/test.git
17:48:51.191887 run-command.c:659       trace: run_command: git-remote-http origin http://192.168.68.103:30008/sam/test.git
17:48:51.211986 http.c:820              == Info: Couldn't find host 192.168.68.103 in the .netrc file; using defaults
17:48:51.212672 http.c:820              == Info:   Trying 192.168.68.103:30008...
17:48:51.224588 http.c:820              == Info: Connected to 192.168.68.103 (192.168.68.103) port 30008
17:48:51.224618 http.c:767              => Send header, 0000000223 bytes (0x000000df)
17:48:51.224625 http.c:779              => Send header: GET /sam/test.git/info/refs?service=git-receive-pack HTTP/1.1
17:48:51.224629 http.c:779              => Send header: Host: 192.168.68.103:30008
17:48:51.224630 http.c:779              => Send header: User-Agent: git/2.42.0
17:48:51.224632 http.c:779              => Send header: Accept: */*
17:48:51.224635 http.c:779              => Send header: Accept-Encoding: deflate, gzip, br, zstd
17:48:51.224637 http.c:779              => Send header: Accept-Language: en-GB, *;q=0.9
17:48:51.224639 http.c:779              => Send header: Pragma: no-cache
17:48:51.224640 http.c:779              => Send header:
17:48:51.234001 http.c:767              <= Recv header, 0000000027 bytes (0x0000001b)
17:48:51.234008 http.c:779              <= Recv header: HTTP/1.1 401 Unauthorized
17:48:51.234011 http.c:767              <= Recv header, 0000000066 bytes (0x00000042)
17:48:51.234013 http.c:779              <= Recv header: Cache-Control: max-age=0, private, must-revalidate, no-transform
17:48:51.234016 http.c:767              <= Recv header, 0000000041 bytes (0x00000029)
17:48:51.234018 http.c:779              <= Recv header: Content-Type: text/plain; charset=utf-8
17:48:51.234020 http.c:767              <= Recv header, 0000000075 bytes (0x0000004b)
17:48:51.234023 http.c:779              <= Recv header: Set-Cookie: i_like_gitea=a4f91157ef077ecc; Path=/; HttpOnly; SameSite=Lax
17:48:51.234026 http.c:767              <= Recv header, 0000000121 bytes (0x00000079)
17:48:51.234030 http.c:779              <= Recv header: Set-Cookie: _csrf=7hd8eyEw5ylpZeLW2nPs9BGfGLo6MTcxMzQ1ODkzMTI1NTUyNjQzMA; Path=/; Max-Age=86400; HttpOnly; SameSite=Lax
17:48:51.234035 http.c:767              <= Recv header, 0000000039 bytes (0x00000027)
17:48:51.234037 http.c:779              <= Recv header: Www-Authenticate: Basic realm="Gitea"
17:48:51.234039 http.c:767              <= Recv header, 0000000033 bytes (0x00000021)
17:48:51.234041 http.c:779              <= Recv header: X-Content-Type-Options: nosniff
17:48:51.234043 http.c:767              <= Recv header, 0000000029 bytes (0x0000001d)
17:48:51.234066 http.c:779              <= Recv header: X-Frame-Options: SAMEORIGIN
17:48:51.234073 http.c:767              <= Recv header, 0000000037 bytes (0x00000025)
17:48:51.234077 http.c:779              <= Recv header: Date: Thu, 18 Apr 2024 16:48:51 GMT
17:48:51.234080 http.c:767              <= Recv header, 0000000020 bytes (0x00000014)
17:48:51.234082 http.c:779              <= Recv header: Content-Length: 13
17:48:51.234084 http.c:767              <= Recv header, 0000000002 bytes (0x00000002)
17:48:51.234086 http.c:779              <= Recv header:
17:48:51.234091 http.c:820              == Info: Connection #0 to host 192.168.68.103 left intact
17:48:51.234362 run-command.c:659       trace: run_command: 'git credential-osxkeychain get'
17:48:51.245202 git.c:749               trace: exec: git-credential-osxkeychain get
17:48:51.245519 run-command.c:659       trace: run_command: git-credential-osxkeychain get
17:48:51.262376 http.c:820              == Info: Found bundle for host: 0x600000bd5050 [serially]
17:48:51.262391 http.c:820              == Info: Can not multiplex, even if we wanted to
17:48:51.262400 http.c:820              == Info: Re-using existing connection with host 192.168.68.103
17:48:51.262406 http.c:820              == Info: Server auth using Basic with user 'sam'
17:48:51.262484 http.c:767              => Send header, 0000000266 bytes (0x0000010a)
17:48:51.262493 http.c:779              => Send header: GET /sam/test.git/info/refs?service=git-receive-pack HTTP/1.1
17:48:51.262494 http.c:779              => Send header: Host: 192.168.68.103:30008
17:48:51.262496 http.c:779              => Send header: Authorization: Basic <redacted>
17:48:51.262498 http.c:779              => Send header: User-Agent: git/2.42.0
17:48:51.262499 http.c:779              => Send header: Accept: */*
17:48:51.262500 http.c:779              => Send header: Accept-Encoding: deflate, gzip, br, zstd
17:48:51.262501 http.c:779              => Send header: Accept-Language: en-GB, *;q=0.9
17:48:51.262503 http.c:779              => Send header: Pragma: no-cache
17:48:51.262504 http.c:779              => Send header:
17:48:51.356123 http.c:767              <= Recv header, 0000000017 bytes (0x00000011)
17:48:51.356130 http.c:779              <= Recv header: HTTP/1.1 200 OK
17:48:51.356135 http.c:767              <= Recv header, 0000000053 bytes (0x00000035)
17:48:51.356138 http.c:779              <= Recv header: Cache-Control: no-cache, max-age=0, must-revalidate
17:48:51.356142 http.c:767              <= Recv header, 0000000060 bytes (0x0000003c)
17:48:51.356144 http.c:779              <= Recv header: Content-Type: application/x-git-receive-pack-advertisement
17:48:51.356148 http.c:767              <= Recv header, 0000000040 bytes (0x00000028)
17:48:51.356150 http.c:779              <= Recv header: Expires: Fri, 01 Jan 1980 00:00:00 GMT
17:48:51.356153 http.c:767              <= Recv header, 0000000018 bytes (0x00000012)
17:48:51.356155 http.c:779              <= Recv header: Pragma: no-cache
17:48:51.356158 http.c:767              <= Recv header, 0000000075 bytes (0x0000004b)
17:48:51.356163 http.c:779              <= Recv header: Set-Cookie: i_like_gitea=09f576ccfb1c861a; Path=/; HttpOnly; SameSite=Lax
17:48:51.356166 http.c:767              <= Recv header, 0000000121 bytes (0x00000079)
17:48:51.356171 http.c:779              <= Recv header: Set-Cookie: _csrf=JcrP5A0dxj4Bf6i03Pg98UkX-886MTcxMzQ1ODkzMTI5MzEyNTI4OQ; Path=/; Max-Age=86400; HttpOnly; SameSite=Lax
17:48:51.356285 http.c:767              <= Recv header, 0000000029 bytes (0x0000001d)
17:48:51.356288 http.c:779              <= Recv header: X-Frame-Options: SAMEORIGIN
17:48:51.356291 http.c:767              <= Recv header, 0000000037 bytes (0x00000025)
17:48:51.356293 http.c:779              <= Recv header: Date: Thu, 18 Apr 2024 16:48:51 GMT
17:48:51.356297 http.c:767              <= Recv header, 0000000021 bytes (0x00000015)
17:48:51.356299 http.c:779              <= Recv header: Content-Length: 229
17:48:51.356302 http.c:767              <= Recv header, 0000000002 bytes (0x00000002)
17:48:51.356304 http.c:779              <= Recv header:
17:48:51.356323 http.c:820              == Info: Connection #0 to host 192.168.68.103 left intact
17:48:51.356339 run-command.c:659       trace: run_command: 'git credential-osxkeychain store'
17:48:51.365573 git.c:749               trace: exec: git-credential-osxkeychain store
17:48:51.366007 run-command.c:659       trace: run_command: git-credential-osxkeychain store
17:48:51.389013 run-command.c:659       trace: run_command: .git/hooks/pre-push origin http://192.168.68.103:30008/sam/test.git
17:48:51.396826 git.c:749               trace: exec: git-lfs pre-push origin http://192.168.68.103:30008/sam/test.git
17:48:51.397128 run-command.c:659       trace: run_command: git-lfs pre-push origin http://192.168.68.103:30008/sam/test.git
17:48:51.424274 trace git-lfs: exec: git 'version'
17:48:51.429668 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote' '-v'
17:48:51.434044 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
17:48:51.437339 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
17:48:51.440848 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
17:48:51.444144 trace git-lfs: exec: git 'config' '--includes' '-l'
17:48:51.447582 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
17:48:51.450830 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
17:48:51.455252 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
17:48:51.459720 trace git-lfs: pre-push: refs/heads/main 99d2b5e81136f20006b6639f7de5e11ac89a735d refs/heads/main 0000000000000000000000000000000000000000
17:48:51.459949 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
17:48:51.463681 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'origin'
17:48:51.517192 trace git-lfs: exec: /usr/bin/security 'list-keychains'
17:48:51.527848 trace git-lfs: exec: /usr/bin/security 'find-certificate' '-a' '-p' '-c' '192.168.68.103:30008' '/Library/Keychains/System.keychain'
17:48:51.538589 trace git-lfs: exec: /usr/bin/security 'find-certificate' '-a' '-p' '-c' '192.168.68.103' '/Library/Keychains/System.keychain'
17:48:51.548606 trace git-lfs: HTTP: POST http://192.168.68.103:30008/locks/verify
> POST /locks/verify HTTP/1.1
> Host: 192.168.68.103:30008
> Accept: application/vnd.git-lfs+json
> Content-Length: 34
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.4.0 (3.4.0; darwin arm64; go 1.21.4)
>
{"ref":{"name":"refs/heads/main"}}17:48:51.559846 trace git-lfs: HTTP: 405

< HTTP/1.1 405 Method Not Allowed
< Content-Length: 0
< Allow: GET
< Allow: GET
< Allow: GET
< Cache-Control: max-age=0, private, must-revalidate, no-transform
< Date: Thu, 18 Apr 2024 16:48:51 GMT
< Set-Cookie: i_like_gitea=7b46257e23d89778; Path=/; HttpOnly; SameSite=Lax
< Set-Cookie: _csrf=RMHiTBf3MlKhcpWe2tYdTR00Bak6MTcxMzQ1ODkzMTU4NjEyNDQyNQ; Path=/; Max-Age=86400; HttpOnly; SameSite=Lax
< X-Frame-Options: SAMEORIGIN
<
Remote "origin" does not support the Git LFS locking API. Consider disabling it with:
  $ git config lfs.http://192.168.68.103:30008.locksverify false
17:48:51.559996 trace git-lfs: tq: running as batched queue, batch size of 100
17:48:51.560073 trace git-lfs: run_command: git rev-list --objects --ignore-missing --not --remotes=origin --stdin --
17:48:51.560077 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-list' '--objects' '--ignore-missing' '--not' '--remotes=origin' '--stdin' '--'
17:48:51.560873 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'cat-file' '--batch-check'
17:48:51.561488 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-common-dir'
17:48:51.565825 trace git-lfs: tq: sending batch of size 1
17:48:51.566168 trace git-lfs: api: batch 1 files
17:48:51.566203 trace git-lfs: HTTP: POST http://192.168.68.103:30008/objects/batch
> POST /objects/batch HTTP/1.1
> Host: 192.168.68.103:30008
> Accept: application/vnd.git-lfs+json
> Content-Length: 222
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.4.0 (3.4.0; darwin arm64; go 1.21.4)
>
{"operation":"upload","objects":[{"oid":"6667b2d1aab6a00caa5aee5af8ad9f1465e567abf1c209d15727d57b3e8f6e5f","size":5}],"transfers":["basic","ssh","lfs-standalone-file"],"ref":{"name":"refs/heads/main"},"hash_algo":"sha256"}17:48:51.570698 trace git-lfs: HTTP: 405

< HTTP/1.1 405 Method Not Allowed
< Content-Length: 0
< Allow: GET
< Cache-Control: max-age=0, private, must-revalidate, no-transform
< Date: Thu, 18 Apr 2024 16:48:51 GMT
< Set-Cookie: i_like_gitea=342147b29f3cdfed; Path=/; HttpOnly; SameSite=Lax
< Set-Cookie: _csrf=QKbFVWIDeVBax2flvsMH6Njvk_k6MTcxMzQ1ODkzMTU5NjczMTQwNg; Path=/; Max-Age=86400; HttpOnly; SameSite=Lax
< X-Frame-Options: SAMEORIGIN
<
17:48:51.570736 trace git-lfs: api error: Client error &{%!!(string=http) %!!(string=) %!!(*url.Userinfo=<nil>) %!!(string=192.168.68.103:30008) %!!(string=/objects/batch) %!!(string=) %!!(bool=false) %!!(bool=false) %!!(string=) %!!(string=) %!!(string=)}s(MISSING) from HTTP 405
batch response: Client error &{%!!(string=http) %!!(string=) %!!(*url.Userinfo=<nil>) %!!(string=192.168.68.103:30008) %!!(string=/objects/batch) %!!(string=) %!!(bool=false) %!!(bool=false) %!!(string=) %!!(string=) %!!(string=)}s(MISSING) from HTTP 405
Uploading LFS objects:   0% (0/1), 0 B | 0 B/s, done.
error: failed to push some refs to 'http://192.168.68.103:30008/sam/test.git'
stavros-k commented 6 months ago

Hmm I'll have to do some checks and get back.

Meanwhile, does other operations work? eg listing or pulling files from lfs?

SamWindell commented 6 months ago

Thanks, let me know if I can provide anything more.

I tried pushing using SSH rather than HTTP and still same issue: HTTP 405.

With regard to other operations, it's a little tricky because the server is empty of LFS file because I can't do the initial push.

Here's a couple more outputs:

❯ git lfs env
git-lfs/3.4.0 (3.4.0; linux amd64; go 1.21.4)
git version 2.42.0

Endpoint=http://192.168.68.103:30008/sam/test.git/info/lfs (auth=basic)
LocalWorkingDir=/home/sam/Projects/test
LocalGitDir=/home/sam/Projects/test/.git
LocalGitStorageDir=/home/sam/Projects/test/.git
LocalMediaDir=/home/sam/Projects/test/.git/lfs/objects
LocalReferenceDirs=
TempDir=/home/sam/Projects/test/.git/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/home/sam/Projects/test/.git/lfs
AccessDownload=basic
AccessUpload=basic
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=/nix/store/zrs710jpfn7ngy5z4c6rrwwjq33b2a0y-git-2.42.0/libexec/git-core
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

test main
❯ gitv lfs fetch --all origin
20:11:56.484239 git.c:749               trace: exec: git-lfs fetch --all origin
20:11:56.484262 run-command.c:659       trace: run_command: git-lfs fetch --all origin
20:11:56.486510 trace git-lfs: exec: git 'version'
20:11:56.487575 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
20:11:56.488411 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
20:11:56.489202 trace git-lfs: exec: git 'config' '--includes' '--local' 'lfs.repositoryformatversion'
20:11:56.489968 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
20:11:56.490807 trace git-lfs: exec: git 'config' '--includes' '-l'
20:11:56.491755 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
20:11:56.492527 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
20:11:56.493367 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
20:11:56.494321 trace git-lfs: run_command: git rev-list --objects --all --stdin --
20:11:56.494325 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-list' '--objects' '--all' '--stdin' '--'
20:11:56.494459 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'cat-file' '--batch-check'
20:11:56.494585 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-common-dir'
fetch: 1 object found, done.
fetch: Fetching all references...
20:11:56.495662 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
20:11:56.496632 trace git-lfs: tq: running as batched queue, batch size of 100
20:11:56.496712 trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore
20:11:56.496719 trace git-lfs: filepathfilter: creating pattern "**/.git" of type gitignore
20:11:56.496737 trace git-lfs: filepathfilter: accepting "tmp"
SamWindell commented 6 months ago

Just noticed this error in Gitea web panel -> Site Administration.

I wonder if my ROOT_URL needs the port: http://192.168.68.103:30008/ I will try this shortly.

image

SamWindell commented 6 months ago

Yep that was it, it's working now. Thanks for looking into this anyways. It seems like it was my misunderstanding of the App setup options. I assumed that because the setup asked for the port and the root-url separately that it would append the port onto the root url if it needed it. But you must be explicit. This is the Gitea Apps 'edit' page on truenas with the correct root-url: image