search

YoshitakaMo / localcolabfold

ColabFold on your local PC
MIT License
608 stars 134 forks source link

Checking if my IP was rate limited? #251

Closed donal1 closed 2 months ago

donal1 commented 2 months ago

Hi Local Colabfold team,

I am always tried to limit my requests to more then one a minute. In this case the proteins are quite large thus it gives large time lags between results. But I am noticing that my almost all my requests are failing. Am I missing something?

Thanks for the code its greats on all my other projects.

ohjeyy93 commented 2 months ago

Hello,

I started getting same problem today... since few hours ago...

donal1 commented 2 months ago

yeah...

milot-mirdita commented 2 months ago

Do you have error messages?

It will show you if you are currently rate limited in the terminal output.

milot-mirdita commented 2 months ago

Is it still an issue right now? It might have been an intermittent issue only

donal1 commented 2 months ago

Error while fetching result from MSA server. Retrying... (1/5)
2024-08-28 17:42:06,395 Error: HTTPSConnectionPool(host='api.colabfold.com', port=443): Read timed out.

Yeah I am still getting timeouts

andrew0901 commented 2 months ago

Hello, I got the exact same error as well.

milot-mirdita commented 2 months ago

Can you please execute the commands below on the node where colabfold_batch fails to help diagnose what might be going wrong:

dig api.colabfold.com A
dig api.colabfold.com AAAA
dig api.colabfold.com A @1.1.1.1
dig api.colabfold.com AAAA @1.1.1.1
ping -c 1 api.colabfold.com
traceroute api.colabfold.com
curl "https://api.colabfold.com/queue"
openssl s_client -connect api.colabfold.com:443

If you don't have dig installed, please try executing instead the following:

nslookup -query=A api.colabfold.com
nslookup -query=AAAA api.colabfold.com
nslookup -query=A api.colabfold.com 1.1.1.1
nslookup -query=AAAA api.colabfold.com 1.1.1.1

Previously it was always some DNS issue, but I've not seen Read timed out. yet.

maxh190 commented 2 months ago

I got the exact same errors as well.

When I ran the following commands:

nslookup -query=A api.colabfold.com
nslookup -query=AAAA api.colabfold.com
nslookup -query=A api.colabfold.com 1.1.1.1
nslookup -query=AAAA api.colabfold.com 1.1.1.1

I got the following:

Server:     10.183.10.235
Address:    10.183.10.235#53

Non-authoritative answer:
Name:   api.colabfold.com
Address: 147.46.145.74

Server:     10.183.10.235
Address:    10.183.10.235#53

Non-authoritative answer:
*** Can't find api.colabfold.com: No answer

Server:     1.1.1.1
Address:    1.1.1.1#53

Non-authoritative answer:
Name:   api.colabfold.com
Address: 147.46.145.74

Server:     1.1.1.1
Address:    1.1.1.1#53

Non-authoritative answer:
*** Can't find api.colabfold.com: No answer

Can you please let me know what's wrong with it?

swansonk14 commented 2 months ago

Hi @milot-mirdita,

I'm getting the same issue as of a couple of minutes ago: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.colabfold.com', port=443): Read timed out

Below I've included the result of running each of your suggested commands. Please let me know if you have any ideas about what's going wrong or suggestions for how to fix it.

Thanks, Kyle

dig api.colabfold.com A

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.16.tuxcare.els1 <<>> api.colabfold.com A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53001
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 13, ADDITIONAL: 27

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;api.colabfold.com.     IN  A

;; ANSWER SECTION:
api.colabfold.com.  1146    IN  A   147.46.145.74

;; AUTHORITY SECTION:
.           109897  IN  NS  l.root-servers.net.
.           109897  IN  NS  d.root-servers.net.
.           109897  IN  NS  i.root-servers.net.
.           109897  IN  NS  m.root-servers.net.
.           109897  IN  NS  g.root-servers.net.
.           109897  IN  NS  e.root-servers.net.
.           109897  IN  NS  a.root-servers.net.
.           109897  IN  NS  j.root-servers.net.
.           109897  IN  NS  k.root-servers.net.
.           109897  IN  NS  b.root-servers.net.
.           109897  IN  NS  h.root-servers.net.
.           109897  IN  NS  f.root-servers.net.
.           109897  IN  NS  c.root-servers.net.

;; ADDITIONAL SECTION:
l.root-servers.net. 109897  IN  A   199.7.83.42
j.root-servers.net. 109897  IN  A   192.58.128.30
f.root-servers.net. 109897  IN  A   192.5.5.241
h.root-servers.net. 109897  IN  A   198.97.190.53
d.root-servers.net. 109897  IN  A   199.7.91.13
b.root-servers.net. 109897  IN  A   170.247.170.2
k.root-servers.net. 109897  IN  A   193.0.14.129
i.root-servers.net. 109897  IN  A   192.36.148.17
m.root-servers.net. 109897  IN  A   202.12.27.33
e.root-servers.net. 109897  IN  A   192.203.230.10
g.root-servers.net. 109897  IN  A   192.112.36.4
c.root-servers.net. 109897  IN  A   192.33.4.12
a.root-servers.net. 109897  IN  A   198.41.0.4
l.root-servers.net. 109897  IN  AAAA    2001:500:9f::42
j.root-servers.net. 109897  IN  AAAA    2001:503:c27::2:30
f.root-servers.net. 109897  IN  AAAA    2001:500:2f::f
h.root-servers.net. 109897  IN  AAAA    2001:500:1::53
d.root-servers.net. 109897  IN  AAAA    2001:500:2d::d
b.root-servers.net. 109897  IN  AAAA    2801:1b8:10::b
k.root-servers.net. 109897  IN  AAAA    2001:7fd::1
i.root-servers.net. 109897  IN  AAAA    2001:7fe::53
m.root-servers.net. 109897  IN  AAAA    2001:dc3::35
e.root-servers.net. 109897  IN  AAAA    2001:500:a8::e
g.root-servers.net. 109897  IN  AAAA    2001:500:12::d0d
c.root-servers.net. 109897  IN  AAAA    2001:500:2::c
a.root-servers.net. 109897  IN  AAAA    2001:503:ba3e::2:30

;; Query time: 0 msec
;; SERVER: 10.18.0.12#53(10.18.0.12)
;; WHEN: Thu Aug 29 10:27:28 PDT 2024
;; MSG SIZE  rcvd: 845

dig api.colabfold.com AAAA

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.16.tuxcare.els1 <<>> api.colabfold.com AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32479
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;api.colabfold.com.     IN  AAAA

;; AUTHORITY SECTION:
colabfold.com.      2917    IN  SOA pdns1.registrar-servers.com. hostmaster.registrar-servers.com. 1724468003 43200 3600 604800 3601

;; Query time: 0 msec
;; SERVER: 10.18.0.12#53(10.18.0.12)
;; WHEN: Thu Aug 29 10:27:58 PDT 2024
;; MSG SIZE  rcvd: 117

dig api.colabfold.com A @1.1.1.1

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.16.tuxcare.els1 <<>> api.colabfold.com A @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3135
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;api.colabfold.com.     IN  A

;; ANSWER SECTION:
api.colabfold.com.  1800    IN  A   147.46.145.74

;; Query time: 110 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Aug 29 10:28:27 PDT 2024
;; MSG SIZE  rcvd: 62

dig api.colabfold.com AAAA @1.1.1.1

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.16.tuxcare.els1 <<>> api.colabfold.com AAAA @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16320
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;api.colabfold.com.     IN  AAAA

;; AUTHORITY SECTION:
colabfold.com.      3601    IN  SOA pdns1.registrar-servers.com. hostmaster.registrar-servers.com. 1724468003 43200 3600 604800 3601

;; Query time: 4 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Aug 29 10:28:43 PDT 2024
;; MSG SIZE  rcvd: 117

ping -c 1 api.colabfold.com

PING api.colabfold.com (147.46.145.74) 56(84) bytes of data.
64 bytes from leb-mule.snu.ac.kr (147.46.145.74): icmp_seq=1 ttl=52 time=137 ms

--- api.colabfold.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 137.363/137.363/137.363/0.000 ms

traceroute api.colabfold.com

I don't have traceroute installed on my system.

curl "https://api.colabfold.com/queue"

{"queued":0}

openssl s_client -connect api.colabfold.com:443

Connecting to 147.46.145.74
CONNECTED(00000003)
depth=2 C=US, O=Internet Security Research Group, CN=ISRG Root X1
verify return:1
depth=1 C=US, O=Let's Encrypt, CN=R11
verify return:1
depth=0 CN=api.colabfold.com
verify return:1
---
Certificate chain
 0 s:CN=api.colabfold.com
   i:C=US, O=Let's Encrypt, CN=R11
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 16 15:02:38 2024 GMT; NotAfter: Nov 14 15:02:37 2024 GMT
 1 s:C=US, O=Let's Encrypt, CN=R11
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIF8zCCBNugAwIBAgISBPRSvqjbrRB1gLPpPl9sYdWKMA0GCSqGSIb3DQEBCwUA
MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD
EwNSMTEwHhcNMjQwODE2MTUwMjM4WhcNMjQxMTE0MTUwMjM3WjAcMRowGAYDVQQD
ExFhcGkuY29sYWJmb2xkLmNvbTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoC
ggIBAIthj8yf2l2esJCHscLSktrfON9p6AojXlaTyUU9BAy8+/v1vbNr4BH6w5/h
DK8USgkMGHJYE+qThP6dgFqv6cjnKX3lON9IHWaY8xf0QXmJ7DM4aEuBHdB5YAkE
bRlCMfdaI5DGUOxkciMZh7PniTbjmyaWJ/Jsg+0zCbwAoXISSIEZWw5zgnhYd6BG
IxIiL+Q73GmrOMN6FjeBrER1P2ajY+vXcRIKFQ0THLZRAn2pmGg0/oGwHnSDIUhy
vVRhoMzk+yv2Ial5juruA40+Rqq+taG8MXL/qpekRn56J0gqG1t/bP5gReL3ufPT
Hqn9xnoedxjj4yCp+MPY280klR3HmWTGZiV5uJNbfegWZE4eNcCSfIURn86vPLfq
ygF+/pv1jVAR8t5xO2nH/lxBBrQxIA7Ed/QFlJA1Rcf9aSMeSxSWz0duKPM9j3cu
AxXrGGLxT/VUQu5ZaXLiKO49/NOOzLRrvh3is99k2O5PVRvfXj7Al7gHqK1VKZyS
k2uFSjEGaAif90+JrmBu4vajdUgukX/RhRBsabmbomUg/Tng3OOxbkNqwaj8elfb
WjchZqIyO3ckQTWERMBbOB0UgwgQe62CdypeK/lc4JJ5cFQ7VSUfN8vKsktLc7/a
3BId4yv8PlGkOswGVZsD0KX7owparZRmOs4XnSJOfx/Xg4Z9AgMBAAGjggIWMIIC
EjAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMC
MAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFFQ9Ujf9XRDZJcmBCCHp/BC7/dxSMB8G
A1UdIwQYMBaAFMXPRqTq9MPAemyVxC2wXpIvJuO5MFcGCCsGAQUFBwEBBEswSTAi
BggrBgEFBQcwAYYWaHR0cDovL3IxMS5vLmxlbmNyLm9yZzAjBggrBgEFBQcwAoYX
aHR0cDovL3IxMS5pLmxlbmNyLm9yZy8wHAYDVR0RBBUwE4IRYXBpLmNvbGFiZm9s
ZC5jb20wEwYDVR0gBAwwCjAIBgZngQwBAgEwggEFBgorBgEEAdZ5AgQCBIH2BIHz
APEAdgA/F0tP1yJHWJQdZRyEvg0S7ZA3fx+FauvBvyiF7PhkbgAAAZFb69jMAAAE
AwBHMEUCIGx6uSjmfDg+eJgbTbQCK3dU7ocPGgKlNgCrmqXCYl2eAiEAjPQBPTlt
HFVRluqRF/0mRKvN37jrjnaCh0QYLpRCCIgAdwDuzdBk1dsazsVct520zROiModG
fLzs3sNRSFlGcR+1mwAAAZFb69jZAAAEAwBIMEYCIQD7UJswkvaeF3ECeNrLxbAW
aTue5nAafcTe40+rTT4jAQIhAOZLLRPxgHVSwOfFB40tRl3UAQWRaGYp2M+TRBwh
5yPZMA0GCSqGSIb3DQEBCwUAA4IBAQBYZJ/MzITTqFhja0Kn4jA+/wwMPPGnLYCC
Uyeu3Y+0vV0R6CUCUMCG9TlFhu2zzOep73KiG9D9OR1P5yNCSaBw7msle0SyM6tP
TAWW9AH0erWubPqCzjL39lfbzRA2nbmF681sliM/FIk6VMo1Y7+vIQc6ts3g7M0d
WO7kPnJqutoYIW1q2EvvlvP2BdrRqtFLumc0YQ8cP4cF45+tRqQSAcQh/g1jM0MR
uEhpWA2AFcAzl57rx2LZ/KSWU0/d99QXpgDSYcAYKQFTp4vXE2wNmmB57vJlCgXZ
pxmPC7CDIss9Vxc3dALP98lYs0cgMOSImqo9zJLV2z0nuliMDp4y
-----END CERTIFICATE-----
subject=CN=api.colabfold.com
issuer=C=US, O=Let's Encrypt, CN=R11
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3638 bytes and written 405 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: B9BBFD3DB7A8120E3A62646C826D7C03D29E2D997426FD1979809E8D10FDA67D
    Session-ID-ctx:
    Resumption PSK: 6DF7F758B147D38A9451ED1A5F9051A8045402F7D784B55A7D1A6C78930BE846D5BEC30EF578EF4C4676A49861DBACB9
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 8f bf fd 44 27 b5 83 85-81 90 68 f3 c0 df 0b e5   ...D'.....h.....
    0010 - 66 e3 e7 63 3a 1f 8d 67-ea 78 d0 4b 09 cd 61 97   f..c:..g.x.K..a.
    0020 - a3 0d a2 50 72 41 9a b1-ea fb 46 04 1f 97 dd d2   ...PrA....F.....
    0030 - ec 48 3f 3b 50 24 c4 29-35 e3 4c eb 6e 30 a2 74   .H?;P$.)5.L.n0.t
    0040 - 6d 66 0e c2 43 43 ee 39-1e 29 78 5a b0 01 ce eb   mf..CC.9.)xZ....
    0050 - 85 b2 c5 80 86 8c 0d e6-73 2e 04 52 01 b5 2b f5   ........s..R..+.
    0060 - 3f b5 12 27 72 bb 77 a7-ba 2f 0c e1 b4 d4 ad 3c   ?..'r.w../.....<
    0070 - c2 eb 09 68 4e bd 77 84-a9 6c b1 0c 90 f5 b9 5e   ...hN.w..l.....^
    0080 - 50 63 fc a3 7e 4c 09 57-c8 e0 93 b9 62 04 13 11   Pc..~L.W....b...
    0090 - ba f3 b3 58 cf d9 2b 49-7e 1f 76 88 60 7b a8 07   ...X..+I~.v.`{..
    00a0 - 33 71 94 2d 78 f6 37 6c-5c ee 29 a6 58 78 dc 08   3q.-x.7l\.).Xx..
    00b0 - a6 4d 10 88 66 ea f5 dc-ab 5a d5 f1 51 c8 76 fa   .M..f....Z..Q.v.
    00c0 - 4f d3 fd e4 c8 8c a4 93-5b 59 f0 da d2 1c df 21   O.......[Y.....!
    00d0 - aa ba d3 39 4d 88 0f a8-37 bb e4 ce ae bb 72 f1   ...9M...7.....r.
    00e0 - 74 b7 08 8e 73 b6 f3 38-14 f2 a3 6f 78 b1 63 78   t...s..8...ox.cx

    Start Time: 1724952604
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: E89B826A9E8F66B7EC3156EC45D15C95F65B37B190B2C0907F570DFC3FC95A9A
    Session-ID-ctx:
    Resumption PSK: 3A3341D75E5373F7991F3EDADC52B4D3354F72AF7F7F142A553F0479E774C04F89876557A9AAB9EC989AC34745239EF9
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 8f bf fd 44 27 b5 83 85-81 90 68 f3 c0 df 0b e5   ...D'.....h.....
    0010 - 0d 6f f4 5b 8d 40 04 e1-5f 77 42 f7 70 09 9b bd   .o.[.@.._wB.p...
    0020 - 3b 3a 8d 75 60 33 19 68-b1 e5 0b 11 d4 dc 22 d2   ;:.u`3.h......".
    0030 - fd a0 28 d1 02 31 53 e6-00 30 11 eb 4e 44 ef ef   ..(..1S..0..ND..
    0040 - 47 a6 82 26 f7 99 f9 eb-e4 ef a0 83 97 51 14 70   G..&.........Q.p
    0050 - 88 c9 95 d2 e0 83 bf cd-68 3e 70 8d b2 55 07 65   ........h>p..U.e
    0060 - 22 54 c7 06 a5 14 69 42-52 e3 dc d3 af f5 96 f4   "T....iBR.......
    0070 - b7 9a 14 a4 b4 30 df 84-31 61 db 63 a2 26 d0 60   .....0..1a.c.&.`
    0080 - be 58 33 61 f5 24 c1 ea-aa a0 c9 ed fb dc 7b 58   .X3a.$........{X
    0090 - 4b b7 32 fa 7f bd 52 b7-4b fc 6a c0 46 d3 69 c1   K.2...R.K.j.F.i.
    00a0 - b3 1d 8a fb 8d f5 0b 14-3e 02 06 eb 57 58 a5 98   ........>...WX..
    00b0 - 13 eb 3a 18 26 e2 e4 04-e1 e4 c1 1d 2b 9d f5 33   ..:.&.......+..3
    00c0 - d7 69 b8 54 2c f1 2a c0-3b 00 38 06 9a b0 dd e0   .i.T,.*.;.8.....
    00d0 - ec 82 bc 13 87 6a 7e 0c-f9 3b 09 41 1a 3c bc fb   .....j~..;.A.<..
    00e0 - c0 8f 6b 1e f7 01 5d bd-27 31 a0 17 92 d6 5c d7   ..k...].'1....\.

    Start Time: 1724952604
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed
milot-mirdita commented 2 months ago

As far as I can currently tell it seems to be some kind of institute wide issue on our side. Some connection (with any pattern I can see) get extremely slow and unreliable.

We will try to coordinate with IT tomorrow. Hopefully they can help us resolve this.

milot-mirdita commented 2 months ago

Please check if the issue is still happening at everyone who had issues. Our IT claims to have resolved the issue, it might take a while to propagate though.

donal1 commented 2 months ago

still getting the error. Will try again early next week to see if changes.

Thanks Milot

milot-mirdita commented 2 months ago

I have implemented a workaround until our IT can fully solve this. Please check that ping -c 1 api.colabfold.com shows 205.185.124.98 as IP and not 147.46.145.74. If its still the later, then please try again in a few minutes when the DNS has expired from cache.

Afterwards, colabfold's MSA server should work in a reasonable speed again.

(From the other thread https://github.com/YoshitakaMo/localcolabfold/issues/252)

donal1 commented 2 months ago

Hi Milot,

it works.

Best

swansonk14 commented 2 months ago

Hi @milot-mirdita,

It works for me too. Thank you for fixing it!

Best, Kyle

TigerWindWood commented 2 months ago

Hi @milot-mirdita ,

It seems that a timeout issue still occurs when predicting amino acid sequences longer than 1300. Sequences shorter than 1300 amino acids run without any problems.

I0904 16:34:19.428889 47330920132352 objects.py:189] You chose to calculate MSA with mmseq2.
Please also cite: Mirdita M, Schütze K, Moriwaki Y, Heo L, Ovchinnikov S and Steinegger M. ColabFold: Making protein folding accessible to all. Nature Methods (2022) doi: 10.1038/s41592-022-01488-1
SUBMIT:   0%|                                                                                                          | 0/150 [elapsed: 00:00 remaining: ?]W0904 16:34:30.301096 47330920132352 colabfold.py:94] Timeout while submitting to MSA server. Retrying...
W0904 16:34:40.733729 47330920132352 colabfold.py:94] Timeout while submitting to MSA server. Retrying...
W0904 16:34:51.166950 47330920132352 colabfold.py:94] Timeout while submitting to MSA server. Retrying...

Best