search

picosh / pico

hacker labs - open source and managed web services leveraging SSH
https://pico.sh
MIT License
774 stars 28 forks source link

ssh fails with "pq: sorry, too many clients already" #11

Closed aaronpkelly closed 1 year ago

aaronpkelly commented 1 year ago

i am unable to ssh fully into my account.

I am able to start the ssh session:

ssh -v prose.sh
OpenSSH_9.1p1, OpenSSL 3.0.7 1 Nov 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Executing command: '/nix/store/qqa28hmysc23yy081d178jfd9a1yk8aw-bash-5.2-p15/bin/bash -c '/nix/store/rk1108fqfrqwcdaymsv67s6vmdiagvdc-gnupg-2.3.7/bin/gpg-connect-agent --quiet updatestartuptty /bye >/dev/null 2>&1''
debug1: /etc/ssh/ssh_config line 9: Applying options for *
debug1: Connecting to prose.sh [2603:c020:400a:d000:b08d:5485:c9e7:cdba] port 22.
debug1: Connection established.
debug1: identity file /home/aaron/.ssh/id_rsa type 0
debug1: identity file /home/aaron/.ssh/id_rsa-cert type -1
debug1: identity file /home/aaron/.ssh/id_ecdsa type -1
debug1: identity file /home/aaron/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/aaron/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/aaron/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/aaron/.ssh/id_ed25519 type 3
debug1: identity file /home/aaron/.ssh/id_ed25519-cert type -1
debug1: identity file /home/aaron/.ssh/id_ed25519_sk type -1
debug1: identity file /home/aaron/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/aaron/.ssh/id_xmss type -1
debug1: identity file /home/aaron/.ssh/id_xmss-cert type -1
debug1: identity file /home/aaron/.ssh/id_dsa type -1
debug1: identity file /home/aaron/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1
debug1: compat_banner: match: OpenSSH_7.6p1 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to prose.sh:22 as 'aaron'
debug1: load_hostkeys: fopen /home/aaron/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:lrDIkMPkXM/qbG5+N4TAINPIc64ART/FBd7gz1OE08g
debug1: load_hostkeys: fopen /home/aaron/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'prose.sh' is known and matches the ED25519 host key.
debug1: Found key in /home/aaron/.ssh/known_hosts:1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: agent returned 2 keys
debug1: Will attempt key: /home/aaron/.ssh/id_ed25519 ED25519 SHA256:Fek0ORzwKRkIJKeR/zCf+U49oaaqho6etw5/q+lQ8ts agent
debug1: Will attempt key: /home/aaron/.ssh/id_rsa RSA SHA256:aWrj0lEHvGz6iuo55wzWZ5hLoOU6mXTq/HZlnu5tje4 agent
debug1: Will attempt key: /home/aaron/.ssh/id_ecdsa
debug1: Will attempt key: /home/aaron/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/aaron/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/aaron/.ssh/id_xmss
debug1: Will attempt key: /home/aaron/.ssh/id_dsa
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/aaron/.ssh/id_ed25519 ED25519 SHA256:Fek0ORzwKRkIJKeR/zCf+U49oaaqho6etw5/q+lQ8ts agent
debug1: Server accepts key: /home/aaron/.ssh/id_ed25519 ED25519 SHA256:Fek0ORzwKRkIJKeR/zCf+U49oaaqho6etw5/q+lQ8ts agent
Authenticated to prose.sh ([2603:c020:400a:d000:b08d:5485:c9e7:cdba]:22) using "publickey".
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
Connection to prose.sh closed.
Transferred: sent 2648, received 31692 bytes, in 40.3 seconds
Bytes per second: sent 65.7, received 785.8
debug1: Exit status 0

And I get this login screen:

prose.sh

  a blog platform for hackers.

  To get started, enter a username.
  Then create a folder locally (e.g. ~/blog).
  Then write your post in markdown files (e.g. hello-world.md).
  Finally, send your files to us:

  scp ~/blog/*.md prose.sh:/

  Enter a username

  >

But when I enter my username, it gives me this error:

  Enter a username

  > aaron

     OK       Cancel

  Oh, what? There was a curious error we were not expecting.
  pq: sorry, too many clients already
aaronpkelly commented 1 year ago

Just to note - I can scp files with no problems, using the same key:

scp -v test.md prose.sh:/
Executing: program /nix/store/9jvvng8cpjxzqvvbpcibl5lr16k4qcb9-openssh-9.1p1/bin/ssh host prose.sh, user (unspecified), command sftp
OpenSSH_9.1p1, OpenSSL 3.0.7 1 Nov 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Executing command: '/nix/store/qqa28hmysc23yy081d178jfd9a1yk8aw-bash-5.2-p15/bin/bash -c '/nix/store/rk1108fqfrqwcdaymsv67s6vmdiagvdc-gnupg-2.3.7/bin/gpg-connect-agent --quiet updatestartuptty /bye >/dev/null 2>&1''
debug1: /etc/ssh/ssh_config line 9: Applying options for *
debug1: Connecting to prose.sh [2603:c020:400a:d000:b08d:5485:c9e7:cdba] port 22.
debug1: Connection established.
debug1: identity file /home/aaron/.ssh/id_rsa type 0
debug1: identity file /home/aaron/.ssh/id_rsa-cert type -1
debug1: identity file /home/aaron/.ssh/id_ecdsa type -1
debug1: identity file /home/aaron/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/aaron/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/aaron/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/aaron/.ssh/id_ed25519 type 3
debug1: identity file /home/aaron/.ssh/id_ed25519-cert type -1
debug1: identity file /home/aaron/.ssh/id_ed25519_sk type -1
debug1: identity file /home/aaron/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/aaron/.ssh/id_xmss type -1
debug1: identity file /home/aaron/.ssh/id_xmss-cert type -1
debug1: identity file /home/aaron/.ssh/id_dsa type -1
debug1: identity file /home/aaron/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1
debug1: compat_banner: match: OpenSSH_7.6p1 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug1: Authenticating to prose.sh:22 as 'aaron'
debug1: load_hostkeys: fopen /home/aaron/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:lrDIkMPkXM/qbG5+N4TAINPIc64ART/FBd7gz1OE08g
debug1: load_hostkeys: fopen /home/aaron/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'prose.sh' is known and matches the ED25519 host key.
debug1: Found key in /home/aaron/.ssh/known_hosts:1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: agent returned 2 keys
debug1: Will attempt key: /home/aaron/.ssh/id_ed25519 ED25519 SHA256:Fek0ORzwKRkIJKeR/zCf+U49oaaqho6etw5/q+lQ8ts agent
debug1: Will attempt key: /home/aaron/.ssh/id_rsa RSA SHA256:aWrj0lEHvGz6iuo55wzWZ5hLoOU6mXTq/HZlnu5tje4 agent
debug1: Will attempt key: /home/aaron/.ssh/id_ecdsa
debug1: Will attempt key: /home/aaron/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/aaron/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/aaron/.ssh/id_xmss
debug1: Will attempt key: /home/aaron/.ssh/id_dsa
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/aaron/.ssh/id_ed25519 ED25519 SHA256:Fek0ORzwKRkIJKeR/zCf+U49oaaqho6etw5/q+lQ8ts agent
debug1: Server accepts key: /home/aaron/.ssh/id_ed25519 ED25519 SHA256:Fek0ORzwKRkIJKeR/zCf+U49oaaqho6etw5/q+lQ8ts agent
Authenticated to prose.sh ([2603:c020:400a:d000:b08d:5485:c9e7:cdba]:22) using "publickey".
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: Sending subsystem: sftp
test.md                                                                                                                                                                           100%   12     0.1KB/s   00:00
scp: debug1: truncating at 12
https://aaron.prose.sh/test
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 2440, received 1872 bytes, in 0.8 seconds
Bytes per second: sent 3196.6, received 2452.4
debug1: Exit status 0
tusharhero commented 1 year ago

I am having the same problem. But I am trying to create a new user.

   prose.sh                                                    

  a blog platform for hackers.                                 

  To get started, enter a username.                            
  Then create a folder locally (e.g. ~/blog).                  
  Then write your post in markdown files (e.g. hello-world.md).
  Finally, send your files to us:                              

  scp ~/blog/*.md prose.sh:/                                   

  Enter a username                                             

  > tusharhero                                                 

     OK       Cancel                                           

  Oh, what? There was a curious error we were not expecting.   
  pq: sorry, too many clients
tusharhero commented 1 year ago

Is this related?

tusharhero commented 1 year ago

I tried it again and it just works now.

neurosnap commented 1 year ago

Greetings! I just wanted to send a quick message that we are aware of degraded services and looking into it. I'll follow up here when we have a resolution and will write a postmortem. Thanks for your patience!

neurosnap commented 1 year ago

Alright the issue should be resolved!

We also installed alerting in our #pico.sh irc channel to be notified more quickly when services are degraded.

Thanks!

aaronpkelly commented 1 year ago

Resolved for me now, thanks!