go-gitea / gitea

Git with a cup of tea! Painless self-hosted all-in-one software development service, including Git hosting, code review, team collaboration, package registry and CI/CD
https://gitea.com
MIT License
44.21k stars 5.42k forks source link

Permission denied (publickey) #21861

Closed tonydm closed 1 year ago

tonydm commented 1 year ago

Description

Without any changes to the host, docker image version, docker-compose.yml file or data directory (config files), I can no longer access my repos via ssh. I can, however, still access the repos via the webui. I've read through the numerous issues posted w/o finding a cause or resolution. I've seen a number of posts that appear to be stale/closed w/o a resolution.

I have "Verified" keys in the user profile via the Webui. I also have a YubiKey configured. I removed it to verify that wasn't the cause. I've verified that the container /data/git/.ssh/authorized_keys file contains the relevant public keys. I've verified that the /etc/ssh/sshd_config file points to the /data/git/.ssh directory.

I even installed tcpdump (apk add tcpdump) to confirm that my connection from the client was indeed reaching the correct server and that there was no firewall filter blocking traffic to the docker container.

After verifying that everything seemed configured properly, etc., I moved to the next step and issued a docker pull to pull down the latest (today 11/18/2022). Still no joy. I don't understand the cause and hesitate at a complete reinstall as I find no reason for the failure in the first place. I don't want to repeat this scenario.

Thank you very much

within the container

bash-5.1# cat /data/git/.ssh/authorized_keys
# gitea public key
command="/usr/local/bin/gitea --config=/data/gitea/conf/app.ini serv key-2",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,no-user-rc,restrict ssh-ed25519 <key> <comment>

within the container

bash-5.1# cat /etc/ssh/sshd_config 
Port 22
Protocol 2

AddressFamily any
ListenAddress 0.0.0.0
ListenAddress ::

LogLevel INFO

HostKey /data/ssh/ssh_host_ed25519_key

HostKey /data/ssh/ssh_host_rsa_key

HostKey /data/ssh/ssh_host_ecdsa_key

HostKey /data/ssh/ssh_host_dsa_key

AuthorizedKeysFile .ssh/authorized_keys
AuthorizedPrincipalsFile .ssh/authorized_principals
TrustedUserCAKeys /data/git/.ssh/gitea-trusted-user-ca-keys.pem
CASignatureAlgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa

UseDNS no
AllowAgentForwarding no
AllowTcpForwarding no
PrintMotd no

PermitUserEnvironment yes
PermitRootLogin no
ChallengeResponseAuthentication no
PasswordAuthentication no
PermitEmptyPasswords no

AllowUsers git

Banner none
Subsystem sftp /usr/lib/ssh/sftp-server

AcceptEnv GIT_PROTOCOL

Gitea Version

v1.17.3

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

https://gist.github.com/tonydm/ea7f990515f7891f4c61507e8f0821c6

Screenshots

No response

Git Version

2.34.1

Operating System

Ubuntu 22.04.1 LTS

How are you running Gitea?

Cloned gitea repo and made relative docker-compose.yml file changes.

~/DockerServices/gitea ................................................................................... user@svr1-dc1 02:41:59 PM
> cat docker-compose.yml 
version: "3.8"

networks:
  gitea:
    external: false

volumes:
  gitea_data:
    external: true

  database:
    driver_opts:
      type: nfs
      o: addr=10.0.2.246
      device: ":/mnt/BigPool/git-repository/gitea-database"

services:
  server:
    image: gitea/gitea:latest
    container_name: gitea_server
    environment:
      - USER_UID=1001
      - USER_GID=1001
      - GITEA__database__DB_TYPE=mysql
      - GITEA__database__HOST=db:3306
      - GITEA__database__NAME=gitea
      - GITEA__database__USER=<db_user>
      - GITEA__database__PASSWD=<db_password>
      - DISABLE_REGISTRATION=true
      - REQUIRE_SIGNIN_VIEW=true
    restart: unless-stopped
    networks:
      - gitea
    volumes:
      - gitea_data:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    ports:
      - "3000:3000"
      - "222:22"
    depends_on:
      - db

  db:
    image: mysql:8
    container_name: gitea_db
    restart: unless-stopped
    environment:
      - MYSQL_ROOT_PASSWORD=<db_root_password>
      - MYSQL_USER=><db_user>
      - MYSQL_PASSWORD=<db_password>
      - MYSQL_DATABASE=gitea
      - PUID=1001
      - PGID=1001
    networks:
      - gitea
    volumes:
      - database:/var/lib/mysql

Database

MySQL

tonydm commented 1 year ago

With further investigation, I think I found why the authentication is failing. According to the sshd stderr output, there is a permissions or ownership issue.

~/DockerServices/gitea main ................................................................. 1m 30s user@svr1-dc1 05:02:55 PM
> docker inspect gitea_server| grep log
        "LogPath": "/var/lib/docker/containers/c78871ef4cf88d94c7392e7f3fe4ae0a5793174dfe80e5e3b277d8c691caaeac/c78871ef4cf88d94c7392e7f3fe4ae0a5793174dfe80e5e3b277d8c691caaeac-json.log",
~/DockerServices/gitea main ........................................................................ user@svr1-dc1 05:03:32 PM
> sudo tail -f /var/lib/docker/containers/c78871ef4cf88d94c7392e7f3fe4ae0a5793174dfe80e5e3b277d8c691caaeac/c78871ef4cf88d94c7392e7f3fe4ae0a5793174dfe80e5e3b277d8c691caaeac-json.log
[sudo] password for user: 
{"log":"chown: /data/ssh/ssh_host_dsa_key: Invalid argument\n","stream":"stderr","time":"2022-11-18T16:56:14.814645558Z"}
{"log":"chown: /data/ssh/ssh_host_dsa_key.pub: Invalid argument\n","stream":"stderr","time":"2022-11-18T16:56:14.815418222Z"}
{"log":"chown: /data/ssh/ssh_host_ecdsa_key: Invalid argument\n","stream":"stderr","time":"2022-11-18T16:56:14.816133861Z"}
{"log":"chown: /data/ssh/ssh_host_ecdsa_key.pub: Invalid argument\n","stream":"stderr","time":"2022-11-18T16:56:14.816938926Z"}
{"log":"chown: /data/ssh/ssh_host_ed25519_key: Invalid argument\n","stream":"stderr","time":"2022-11-18T16:56:14.817904488Z"}
{"log":"chown: /data/ssh/ssh_host_ed25519_key.pub: Invalid argument\n","stream":"stderr","time":"2022-11-18T16:56:14.818830153Z"}
{"log":"chown: /data/ssh/ssh_host_rsa_key: Invalid argument\n","stream":"stderr","time":"2022-11-18T16:56:14.81961883Z"}
{"log":"chown: /data/ssh/ssh_host_rsa_key.pub: Invalid argument\n","stream":"stderr","time":"2022-11-18T16:56:14.820309201Z"}
{"log":"Server listening on :: port 22.\r\n","stream":"stdout","time":"2022-11-18T16:56:14.892206031Z"}
{"log":"Server listening on 0.0.0.0 port 22.\r\n","stream":"stdout","time":"2022-11-18T16:56:14.892254756Z"}
{"log":"Authentication refused: bad ownership or modes for file /data/git/.ssh/authorized_keys\r\n","stream":"stdout","time":"2022-11-18T17:03:48.85431296Z"}
{"log":"Authentication refused: bad ownership or modes for file /data/git/.ssh/authorized_keys\r\n","stream":"stdout","time":"2022-11-18T17:03:48.860547911Z"}
{"log":"Connection closed by authenticating user git 10.0.2.5 port 47316 [preauth]\r\n","stream":"stdout","time":"2022-11-18T17:03:48.868574737Z"}

The permissions are correct for the root dir and files respectively (644 for .ssh/, 600 for authorized_keys) But ownership nobody is in question. I have no idea of how/where/when the ownership was set to nobody. But where I'm perplexed is that I haven't touched the NFS Share permissions since it was set up to host the data for the Gitea data

bash-5.1# 
bash-5.1# ll /data/git/
total 46
drwxr-xr-x    5 nobody   nobody           7 Nov 18 13:51 ./
drwxrwxrwx    6 nobody   nobody           6 Jun 22 15:06 ../
-rw-------    1 nobody   nobody         170 Nov 18 13:51 .bash_history
-rw-r--r--    1 nobody   nobody         190 Jun 22 15:07 .gitconfig
drwx------    2 nobody   nobody           8 Nov 18 16:44 .ssh/
drwxr-xr-x    2 nobody   nobody           2 Jun 22 15:07 lfs/
drwxr-xr-x    3 nobody   nobody           3 Jun 22 15:14 repositories/
bash-5.1# 
bash-5.1# 
bash-5.1# ll /data/git/.ssh/
total 67
drwx------    2 nobody   nobody           8 Nov 18 16:44 ./
drwxr-xr-x    5 nobody   nobody           7 Nov 18 13:51 ../
-rw-------    1 nobody   nobody         549 Nov 18 16:47 authorized_keys
-rw-------    1 nobody   nobody         274 Jun 22 15:08 authorized_keys_1659480267.gitea_bak
-rw-------    1 nobody   nobody         549 Nov 16 20:17 authorized_keys_1668630208.gitea_bak
-rw-------    1 nobody   nobody         549 Nov 16 20:23 authorized_keys_1668782632.gitea_bak
-rw-------    1 nobody   nobody         274 Nov 18 14:43 authorized_keys_1668789855.gitea_bak
-rw-------    1 nobody   nobody          25 Jun 22 15:06 environment
bash-5.1# 
bash-5.1# 

git user - /etc/passwd & /etc/group

bash-5.1# 
bash-5.1# cat /etc/passwd | grep git
git:x:1001:1001:Linux User,,,:/data/git:/bin/bash
bash-5.1# 
bash-5.1# 
bash-5.1# cat /etc/group | grep git
git:x:1001:git
bash-5.1# 
bash-5.1# 

attempting to change ownership fails. may be related to the data residing on TrueNAS NFS share?

bash-5.1# chown -R git:git /data/git/.ssh
chown: /data/git/.ssh/environment: Invalid argument
chown: /data/git/.ssh/authorized_keys_1659480267.gitea_bak: Invalid argument
chown: /data/git/.ssh/authorized_keys_1668630208.gitea_bak: Invalid argument
chown: /data/git/.ssh/authorized_keys: Invalid argument
chown: /data/git/.ssh/authorized_keys_1668789855.gitea_bak: Invalid argument
chown: /data/git/.ssh/authorized_keys_1668782632.gitea_bak: Invalid argument
chown: /data/git/.ssh: Invalid argument
chown: /data/git/.ssh: Invalid argument
bash-5.1#
bash-5.1# chown -R 1001:1001 /data/git/.ssh/
chown: /data/git/.ssh/environment: Invalid argument
chown: /data/git/.ssh/authorized_keys_1659480267.gitea_bak: Invalid argument
chown: /data/git/.ssh/authorized_keys_1668630208.gitea_bak: Invalid argument
chown: /data/git/.ssh/authorized_keys: Invalid argument
chown: /data/git/.ssh/authorized_keys_1668789855.gitea_bak: Invalid argument
chown: /data/git/.ssh/authorized_keys_1668782632.gitea_bak: Invalid argument
chown: /data/git/.ssh/: Invalid argument
chown: /data/git/.ssh/: Invalid argument
tonydm commented 1 year ago

Progress, I believe I've resolved the SSH issue. But now WebUI and access to repos via ssh is broken. With that said, I had created a volume for the NFS mount in Portainer. Portainer had set it to NFSv4. While this had worked from the initial deployment until yesterday, some Googling seemed to indicate that NFSv4 has some issues with root_squash [chown: changing ownership of `.': Invalid argument](https://serverfault.com/questions/443957/chown-changing-ownership-of-invalid-argument).

I suppose that means I have to re-deploy from scratch. POOP!

Some detail

When I tried to create another Volume mount, via Portainer, and add nfsv3 as another option, the container creation would fail because the volume mount would fail. So I moved the volume mount config into the docker-compose.yml file

no v3 option is specified, which then defaults to v3 unless v4 is specified

  gitea_data:
    driver_opts:
      type: nfs
      o: addr=10.0.2.246
      device: ":/mnt/BigPool/git-repository"

and file ownership now shows the correct ownership

bash-5.1# cd data/
bash-5.1# ll
total 49
drwxrwxrwx    6 root     root             6 Jun 22 15:06 ./
drwxr-xr-x    1 root     root          4096 Nov 18 17:45 ../
drwxr-xr-x    5 git      git              7 Nov 18 17:35 git/
drwxr-xr-x   15 git      git             15 Aug 22 17:18 gitea/
drwxr-xr-x    2 999      ping             2 Jun 22 14:56 gitea-database/
drwx------    2 root     root            10 Jun 22 15:06 ssh/
bash-5.1# cd git
bash-5.1# ll
total 46
drwxr-xr-x    5 git      git              7 Nov 18 17:35 ./
drwxrwxrwx    6 root     root             6 Jun 22 15:06 ../
-rw-------    1 root     git            170 Nov 18 13:51 .bash_history
-rw-r--r--    1 root     git            190 Jun 22 15:07 .gitconfig
drwx------    2 git      git              8 Nov 18 16:44 .ssh/
drwxr-xr-x    2 root     git              2 Jun 22 15:07 lfs/
drwxr-xr-x    3 root     git              3 Jun 22 15:14 repositories/
bash-5.1# cd .ssh/
bash-5.1# ll
total 67
drwx------    2 git      git              8 Nov 18 16:44 ./
drwxr-xr-x    5 git      git              7 Nov 18 17:35 ../
-rw-------    1 git      git            549 Nov 18 16:47 authorized_keys
-rw-------    1 root     git            274 Jun 22 15:08 authorized_keys_1659480267.gitea_bak
-rw-------    1 root     git            549 Nov 16 20:17 authorized_keys_1668630208.gitea_bak
-rw-------    1 root     git            549 Nov 16 20:23 authorized_keys_1668782632.gitea_bak
-rw-------    1 root     git            274 Nov 18 14:43 authorized_keys_1668789855.gitea_bak
-rw-------    1 git      git             25 Jun 22 15:06 environment

The WebUI now displays Bad Gateway and a git push fails... still, but for another reason. But no longer shows Permission denied (publickey)

➜  .dotfiles git:(master) git push
ssh: connect to host 10.0.2.106 port 222: Connection refused
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

SSH now appears successful. I am prompted for my passphrase which means that the authorized_keys file is being read.

➜  .dotfiles git:(master) ssh -vvvT -i ~/.ssh/user -l git -p 222 10.0.2.106 
OpenSSH_8.9p1 Ubuntu-3, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/user/.ssh/config line 2: Applying options for *
debug3: kex names ok: [diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group-exchange-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,curve25519-sha256,curve25519-sha256@libssh.org,sntrup761x25519-sha512@openssh.com]
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 10.0.2.106 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'
debug1: auto-mux: Trying existing master
debug1: Control socket "/tmp/ssh-git@10.0.2.106:222" does not exist
debug3: ssh_connect_direct: entering
debug1: Connecting to 10.0.2.106 [10.0.2.106] port 222.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: Connection established.
debug1: identity file /home/user/.ssh/user type 3
debug1: identity file /home/user/.ssh/user-cert type -1
debug1: identity file /home/user/.ssh/user type 3
debug1: identity file /home/user/.ssh/user-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.0
debug1: compat_banner: match: OpenSSH_9.0 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 10.0.2.106:222 as 'git'
debug3: put_host_port: [10.0.2.106]:222
debug3: record_hostkey: found key type ED25519 in file /home/user/.ssh/known_hosts:404
debug3: load_hostkeys_file: loaded 1 keys from [10.0.2.106]:222
debug1: load_hostkeys: fopen /home/user/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:<hash>
debug3: put_host_port: [10.0.2.106]:222
debug3: put_host_port: [10.0.2.106]:222
debug3: record_hostkey: found key type ED25519 in file /home/user/.ssh/known_hosts:404
debug3: load_hostkeys_file: loaded 1 keys from [10.0.2.106]:222
debug1: load_hostkeys: fopen /home/user/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '[10.0.2.106]:222' is known and matches the ED25519 host key.
debug1: Found key in /home/user/.ssh/known_hosts:404
debug3: send packet: type 21
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: ssh_fetch_identitylist: agent contains no identities
debug1: Will attempt key: /home/user/.ssh/user ED25519 SHA256:<hash> explicit
debug1: Will attempt key: /home/user/.ssh/user ED25519 SHA256:<hash> explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug1: kex_input_ext_info: publickey-hostbound@openssh.com=<0>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/user/.ssh/user ED25519 SHA256:<hash> explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /home/user/.ssh/user ED25519 SHA256:<hash> explicit
debug3: sign_and_send_pubkey: using publickey-hostbound-v00@openssh.com with ED25519 SHA256:<hash>
debug3: sign_and_send_pubkey: signing using ssh-ed25519 SHA256:<hash>
Enter passphrase for key '/home/user/.ssh/user': 
debug3: send packet: type 50
debug3: receive packet: type 52
Authenticated to 10.0.2.106 ([10.0.2.106]:222) using "publickey".
debug1: setting up multiplex master socket
debug3: muxserver_listen: temporary control path /tmp/ssh-git@10.0.2.106:222.T84Y9BtlmxZac3m7
debug2: fd 4 setting O_NONBLOCK
debug3: fd 4 is O_NONBLOCK
debug3: fd 4 is O_NONBLOCK
debug1: channel 0: new [/tmp/ssh-git@10.0.2.106:222]
debug3: muxserver_listen: mux listener channel 0 fd 4
debug2: fd 3 setting TCP_NODELAY
debug3: set_sock_tos: set socket 3 IP_TOS 0x08
debug1: control_persist_detach: backgrounding master process
debug2: control_persist_detach: background process is 1614869
debug2: fd 4 setting O_NONBLOCK
debug1: forking to background
debug1: Entering interactive session.
debug1: pledge: id
debug1: multiplexing control connection
debug2: fd 5 setting O_NONBLOCK
debug3: fd 5 is O_NONBLOCK
debug1: channel 1: new [mux-control]
debug3: channel_post_mux_listener: new mux channel 1 fd 5
debug3: mux_master_read_cb: channel 1: hello sent
debug3: mux_master_read_cb: channel 1 packet type 0x00000001 len 4
debug2: mux_master_process_hello: channel 1 client version 4
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_master_read_cb: channel 1 packet type 0x10000004 len 4
debug2: mux_master_process_alive_check: channel 1: alive check
debug3: mux_client_request_alive: done pid = 1614871
debug3: mux_client_request_session: session request sent
debug3: mux_master_read_cb: channel 1 packet type 0x10000002 len 70
debug2: mux_master_process_new_session: channel 1: request tty 0, X 0, agent 0, subsys 0, term "xterm-256color", cmd "", env 1
debug3: mux_master_process_new_session: got fds stdin 6, stdout 7, stderr 8
debug1: channel 2: new [client-session]
debug2: mux_master_process_new_session: channel_new: 2 linked to control channel 1
debug2: channel 2: send open
debug3: send packet: type 90
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: client_input_hostkeys: received ED25519 key SHA256:<hash>
debug3: client_input_hostkeys: received RSA key SHA256:<hash>
debug3: client_input_hostkeys: received ECDSA key SHA256:<hash>
debug3: client_input_hostkeys: received DSA key SHA256:<hash>
debug3: client_input_hostkeys: ssh-dss key not permitted by HostkeyAlgorithms
debug3: put_host_port: [10.0.2.106]:222
debug1: client_input_hostkeys: searching /home/user/.ssh/known_hosts for [10.0.2.106]:222 / (none)
debug3: hostkeys_foreach: reading file "/home/user/.ssh/known_hosts"
debug3: hostkeys_find: found ecdsa-sha2-nistp256 key under different name/addr at /home/user/.ssh/known_hosts:267
debug3: hostkeys_find: found ssh-ed25519 key at /home/user/.ssh/known_hosts:404
debug1: client_input_hostkeys: searching /home/user/.ssh/known_hosts2 for [10.0.2.106]:222 / (none)
debug1: client_input_hostkeys: hostkeys file /home/user/.ssh/known_hosts2 does not exist
debug3: client_input_hostkeys: 3 server keys: 2 new, 18446744073709551615 retained, 2 incomplete match. 0 to remove
debug1: client_input_hostkeys: host key found matching a different name/address, skipping UserKnownHostsFile update
debug3: receive packet: type 4
debug1: Remote: /data/git/.ssh/authorized_keys:2: key options: command
debug3: receive packet: type 4
debug1: Remote: /data/git/.ssh/authorized_keys:2: key options: command
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 2: callback start
debug2: client_session2_setup: id 2
debug1: Sending environment.
debug1: channel 2: setting env LANG = "en_US.UTF-8"
debug2: channel 2: request env confirm 0
debug3: send packet: type 98
debug2: channel 2: request shell confirm 1
debug3: send packet: type 98
debug3: mux_session_confirm: sending success reply
debug2: channel_input_open_confirmation: channel 2: callback done
debug2: channel 2: open confirm rwindow 0 rmax 32768
debug1: mux_client_request_session: master session id: 2
debug2: channel 2: rcvd adjust 2097152
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 2
debug2: shell request accepted on channel 2

debug2: channel 2: rcvd ext data 22
Gitea: Internal error
debug2: channel 2: written 22 to efd 8
debug3: receive packet: type 96
debug2: channel 2: rcvd eof
debug2: channel 2: output open -> drain
debug2: channel 2: obuf empty
debug2: chan_shutdown_write: channel 2: (i0 o1 sock -1 wfd 7 efd 8 [write])
debug2: channel 2: output drain -> closed
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 2 rtype exit-status reply 0
debug3: mux_exit_message: channel 2: exit message, exitval 1
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0
debug2: channel 2: rcvd eow
debug2: chan_shutdown_read: channel 2: (i0 o3 sock -1 wfd 6 efd 8 [write])
debug2: channel 2: input open -> closed
debug3: receive packet: type 97
debug2: channel 2: rcvd close
debug3: channel 2: will not send data after close
debug2: channel 2: send close
debug3: send packet: type 97
debug2: channel 2: is dead
debug2: channel 2: gc: notify user
debug3: mux_master_session_cleanup_cb: entering for channel 2
debug2: channel 1: rcvd close
debug2: channel 1: output open -> drain
debug2: chan_shutdown_read: channel 1: (i0 o1 sock 5 wfd 5 efd -1 [closed])
debug2: channel 1: input open -> closed
debug2: channel 2: gc: user detached
debug2: channel 2: is dead
debug2: channel 2: garbage collecting
debug1: channel 2: free: client-session, nchannels 3
debug3: channel 2: status: The following connections are open:
  #1 mux-control (t16 nr0 i3/0 o1/16 e[closed]/0 fd 5/5/-1 sock 5 cc -1 io 0x03/0x00)
  #2 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/8 sock -1 cc -1 io 0x00/0x00)

debug2: channel 1: obuf empty
debug2: chan_shutdown_write: channel 1: (i3 o1 sock 5 wfd 5 efd -1 [closed])
debug2: channel 1: output drain -> closed
debug2: channel 1: is dead (local)
debug2: channel 1: gc: notify user
debug3: mux_master_control_cleanup_cb: entering for channel 1
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: channel 1: gc: user detached
debug2: channel 1: is dead (local)
debug2: channel 1: garbage collecting
debug2: Received exit status from master 1
debug1: channel 1: free: mux-control, nchannels 2
debug3: channel 1: status: The following connections are open:
  #1 mux-control (t16 nr0 i3/0 o3/0 e[closed]/0 fd 5/5/-1 sock 5 cc -1 io 0x00/0x03)
tonydm commented 1 year ago

I have redeployed gitea making a change to how I access my NFS share. Instead of creating the Volume mounts in the docker-compose.yml file, I've mounted the NFS share directly at the host level. Then changed the volumes to bind mounts in the compose file. While it may not have been necessary, it simplified the mounting of the two separate paths pointing to the same NFS share. All is working as expected.

Please forgive the verbosity throughout the post, but perhaps anyone else encountering similar issues can find inspiration in resolving their issue.

NFS mount

~/DockerServices/gitea ................................................................................... user@svr1-dc1 11:12:43 AM
> sudo mount -t nfs -o vers=3 10.0.2.246:/mnt/BigPool/git-repository gitrepo-nfsshare

persist mount on reboot

~/DockerServices/gitea ................................................................................... user@srv1-dc1 11:16:25 AM
> cat /etc/fstab 
# /etc/fstab: static file system information.
#
/dev/disk/by-id/dm-uuid-LVM-WtRex5kmN3wvl0XvshuqATbgSyZ8oac1Hs55XX700lyNmeUeGiQuMEHZAXeZnZLq / ext4 defaults 0 1
/dev/disk/by-uuid/27eb65be-6a46-483b-8509-42e5f185d6d3 /boot ext4 defaults 0 1
/swap.img   none    swap    sw  0   0

# Gitea git repository on TrueNAS NFS Server
10.0.2.246:/mnt/BigPool/git-repository /home/user/DockerServices/gitea/gitrepo-nfsshare   nfs     defaults        0       0

docker-compose.yml

~/DockerServices/gitea ................................................................................... user@svr1-dc1 11:17:31 AM
> cat docker-compose.yml  
version: "3.8"

services:
  server:
    image: gitea/gitea:latest
    container_name: gitea_server
    restart: unless-stopped
    environment:
      - USER_UID=1001
      - USER_GID=1001
      - GITEA__database__DB_TYPE=mysql
      - GITEA__database__HOST=db:3306
      - GITEA__database__NAME=gitea
      - GITEA__database__USER=<db_user>
      - GITEA__database__PASSWD=<db_password>
      - DISABLE_REGISTRATION=true
      - REQUIRE_SIGNIN_VIEW=true
    networks:
      - gitea
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      - ./gitrepo-nfsshare/data:/data
    ports:
      - "3000:3000"
      - "222:22"
    depends_on:
      - db

  db:
    image: mysql:8
    container_name: gitea_database
    restart: unless-stopped
    environment:
      - PUID=1001
      - PGID=1001
      - MYSQL_DATABASE=gitea
      - MYSQL_ROOT_PASSWORD=<db_root_password>
      - MYSQL_USER=<db_user>
      - MYSQL_PASSWORD=<db_password>
    networks:
      - gitea
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      - ./gitrepo-nfsshare/database:/var/lib/mysql

host

~/DockerServices/gitea ................................................................................... user@svr1-dc1 11:24:46 PM
> ll gitrepo-nfsshare/data/git/
total 2.5K
drwxr-xr-x 5 user user 5 2022-11-18 20:33 ./
drwxrwxrwx 5 root root 5 2022-11-18 20:31 ../
drwxrwxrwx 2 root user 2 2022-11-18 20:33 lfs/
drwxrwxrwx 3 root user 3 2022-11-18 20:50 repositories/
drwx------ 2 user user 4 2022-11-18 20:33 .ssh/
~/DockerServices/gitea ................................................................................... user@svr1-dc1 11:24:50 PM
> ll gitrepo-nfsshare/data/git/.ssh 
total 9.0K
drwx------ 2 user user   4 2022-11-18 20:33 ./
drwxr-xr-x 5 user user   5 2022-11-18 20:33 ../
-rw------- 1 root user 563 2022-11-18 22:23 authorized_keys
-rw------- 1 user user  25 2022-11-18 20:31 environment

container

~/DockerServices/gitea ................................................................................... user@srv1-dc1 11:35:25 AM
> docker exec -it gitea_server bash
bash-5.1# 
bash-5.1# ll
total 3
drwxr-xr-x    5 git      git              5 Nov 18 20:33 ./
drwxrwxrwx    5 root     root             5 Nov 18 20:31 ../
drwx------    2 git      git              4 Nov 18 20:33 .ssh/
drwxrwxrwx    2 root     git              2 Nov 18 20:33 lfs/
drwxrwxrwx    3 root     git              3 Nov 18 20:50 repositories/
bash-5.1#
bash-5.1# ll .ssh
total 9
drwx------    2 git      git              4 Nov 18 20:33 ./
drwxr-xr-x    5 git      git              5 Nov 18 20:33 ../
-rw-------    1 root     git            563 Nov 18 22:23 authorized_keys
-rw-------    1 git      git             25 Nov 18 20:31 environment
bash-5.1# 

I want to thank the devs for including the -e option in the openssh run script which enabled the sshd daemon error output needed to see what was really going on without the need to rebuild the image. Kudos!

~/DockerServices/gitea ................................................................................... user@srv1-dc1 11:40:49 AM
> cat ./gitrepo/docker/root/etc/s6/openssh/run 
#!/bin/bash
[[ -f ./setup ]] && source ./setup

pushd /root >/dev/null
exec su-exec root /usr/sbin/sshd -D -e 2>&1
popd
tonydm commented 1 year ago

Issue resolved

Nomsplease commented 1 year ago

Just noting here I ran into similar permissions issues with Docker.

My git user inside the docker was unable to access its .ssh directory, even though the directory and ls showed they had the correct permissions. These were even more permissive than necessary.

ef6f45677b44:~$ ls -lash
total 0      
     0 drwxr-xr-x    1 git      users         38 Nov 18 13:31 .
     0 drwxrwxr-x    1 git      users         40 Nov 21 16:39 ..
     0 drw-r--r--    1 git      users         52 Nov 18 13:31 .ssh
     0 drwxrwxr-x    1 git      users          0 Nov 18 13:31 lfs
     0 drwxrwxr-x    1 git      users         20 Nov 18 14:12 repositories
ef6f45677b44:~$ id
uid=99(git) gid=100(users) groups=100(users),100(users)
ef6f45677b44:~$ cd .ssh/
-bash: cd: .ssh/: Permission denied

I used the root user in the docker to open up the permissions entirely on the .ssh directory. After that I used the git user to set the permissions correctly on the directory which. After which ssh worked just as expected.

/data/git # ls -lash
total 4K     
     0 drwxr-xr-x    1 git      users         64 Nov 21 18:01 .
     0 drwxrwxr-x    1 git      users         40 Nov 21 16:39 ..
     4 -rw-------    1 git      users         72 Nov 21 18:01 .bash_history
     0 drw-r--r--    1 git      users         52 Nov 18 13:31 .ssh
     0 drwxrwxr-x    1 git      users          0 Nov 18 13:31 lfs
     0 drwxrwxr-x    1 git      users         20 Nov 18 14:12 repositories
/data/git # chmod 755 .ssh
/data/git # su - git
ef6f45677b44:~$ chmod -R 755 .ssh
ef6f45677b44:~$ ls -lash
total 4K     
     0 drwxr-xr-x    1 git      users         64 Nov 21 18:01 .
     0 drwxrwxr-x    1 git      users         40 Nov 21 16:39 ..
     4 -rw-------    1 git      users         72 Nov 21 18:01 .bash_history
     0 drwxr-xr-x    1 git      users         52 Nov 18 13:31 .ssh
     0 drwxrwxr-x    1 git      users          0 Nov 18 13:31 lfs
     0 drwxrwxr-x    1 git      users         20 Nov 18 14:12 repositories
ef6f45677b44:~$ cd .ssh/
ef6f45677b44:~/.ssh$ ls -lash
total 8K     
     0 drwxr-xr-x    1 git      users         52 Nov 18 13:31 .
     0 drwxr-xr-x    1 git      users         64 Nov 21 18:01 ..
     4 -rwxr-xr-x    1 git      users        284 Nov 21 16:21 authorized_keys
     4 -rwxr-xr-x    1 git      users         25 Nov 18 12:20 environment
ef6f45677b44:~/.ssh$ chmod 600 authorized_keys environment 
ef6f45677b44:~/.ssh$ ls -lash
total 8K     
     0 drwxr-xr-x    1 git      users         52 Nov 18 13:31 .
     0 drwxr-xr-x    1 git      users         64 Nov 21 18:01 ..
     4 -rw-------    1 git      users        284 Nov 21 16:21 authorized_keys
     4 -rw-------    1 git      users         25 Nov 18 12:20 environment

I'm not sure how these permissions got so out of sync or why the git user could not access the directory, but this fixed it for me. Hopefully this stops someone else from pulling their hair out for outs due to the container being a bit silly with its permissions.

99rgosse commented 1 year ago

Just noting here I ran into similar permissions issues with Docker.

My git user inside the docker was unable to access its .ssh directory, even though the directory and ls showed they had the correct permissions. These were even more permissive than necessary.

I was having the same issue after upgrading docker gitea from 1.16.8 to 1.17.3 I solved this by setting the HOME_PATH in app.ini:

[git]
HOME_PATH                    = /data/git/
gylove1994 commented 1 year ago

The error log

2022/12/09 02:33:02 ...user/setting/keys.go:181:KeysPost() [E] AddPublicKey: addKey: open /data/git/.ssh/authorized_keys: permission denied

Due to this caused the error:

root@*:/home/git/.ssh# ls -all
total 24
drwxrwxrwx 2 root root 4096 Dec  8 04:21 .
drwxr-xr-x 3 root root 4096 Dec  8 04:18 ..
!!!!!!!!!!!!!!!!! -rwxrwxrwx 1 root root  740 Dec  8 04:13 authorized_keys
!!!!!!!!!!!!!!!!! -rwxrwxrwx 1 root root   25 Dec  8 04:21 environment
-rwxrwxrwx 1 git  git  3381 Dec  8 04:13 id_rsa
-rwxrwxrwx 1 git  git   740 Dec  8 04:13 id_rsa.pub

how to fix

try to use: chown -R 1000:1000 ./.ssh (1000:1000 is your git user uid:gid) in your git home path.

It could be able to fix the problem.

About chown command

The chown command is used to change the owner and group of a file or directory. The -R option specifies that the owner and group should be recursively changed for all files and directories in subdirectories.

For example, if you use the above command, the .ssh directory and all its subdirectories and files will have their owner and group changed to the user with user ID 1000 and the group with group ID 1000.

The syntax for this command is:

chown [options] user:group file

Typically, you need superuser privileges to change the owner or group of a file or directory. So if you want to use the chown command, you can use the sudo command to gain superuser privileges, for example:

sudo chown -R 1000:1000 ./.ssh

Please note that you should be careful when changing the owner and group of a file or directory, as this can potentially damage the integrity of the file system or cause other unintended consequences.