search

gap-packages / curlInterface

Easy web access for GAP
https://gap-packages.github.io/curlInterface/
Other
0 stars 5 forks source link

Test hangs in GAP 4.12.0 (curlInterface 2.2.3) #42

Closed frankluebeck closed 3 days ago

frankluebeck commented 2 years ago

The test for the package curlInterface hangs indefinitely. It hangs in line 139 of the file pkg/curlinterface/tst/basic.tst, the command is

CurlRequest("www.google.com", "HEAD" , "");
fingolfin commented 2 years ago

I can't reproduce this. On both a MacBook and an Ubuntu server, I get this:

gap> LoadPackage("curl");
─────────────────────────────────────────────────────────────────────
Loading  curlInterface 2.2.3 (Simple Web Access)
by Christopher Jefferson (http://caj.host.cs.st-andrews.ac.uk/) and
   Michael Young (http://mct25.host.cs.st-andrews.ac.uk/).
Homepage: https://gap-packages.github.io/curlInterface/
Report issues at https://github.com/gap-packages/curlInterface/issues
─────────────────────────────────────────────────────────────────────
true
gap> CurlRequest("www.google.com", "HEAD" , "");
rec( result := "", success := true )

Can you provide any further details? Which OS is this on? Which version of libcurl is installed (BTW this should be made easier to determine, thus I submitted issue #45)?

fingolfin commented 2 years ago

@thomasbreuer: can you reproduce?

ThomasBreuer commented 2 years ago

@fingolfin On my machine, these tests run without problem. (I had run them quite a few times during the last days ...)

frankluebeck commented 2 years ago

The problem seems indeed more complicated.

On our office computers (Debian 10 or 11), calling GAP 4.12.0 (rsync-distribution) or GAP master, with or without -r and/or -A, the mentioned command hangs indefinitely.

On my notebook (Ubuntu) with (good Gigabit) private internet connection with exactly the same GAP 4.12.0 installation (rsync) the command works as you show above.

But other problems occur: Calling repeatedly within the curlinterface directory GAP and TestDirectory("tst"); needs between 30 seconds and 5 minutes, hanging for a while in different lines - not the one mentioned above. And it shows varying numbers of errors, I have seen all numbers from 0 to 4 errors. Example:

12> gap
 ┌─── ───┐   GAP 4.12.0 of 2022-08-18
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: x86_64-pc-linux-gnu-default64-kv8
 Configuration:  gmp 6.1.2, GASMAN, readline
 Loading the library and packages ...
 Packages:   Alnuth 3.2.1, AtlasRep 2.1.5, AutPGrp 1.11, Browse 1.8.14, 
             CTblLib 1.3.4, EDIM 1.3.5, FactInt 1.6.3, Forms 1.2.8, 
             GAPDoc 1.6.6, genss 1.6.7, IO 4.7.2, orb 4.8.5, Polycyclic 2.16, 
             PrimGrp 3.4.2, recog 1.3.2, SmallGrp 1.5, SpinSym 1.5.2, 
             TomLib 1.2.9, TransGrp 3.6.3
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
gap> TestDirectory("tst");
Architecture: x86_64-pc-linux-gnu-default64-kv8

testing: /usr/local/ca/gap-rsync/pkg/curlinterface/tst/basic.tst
# line 160 of 179 (89%)*   Trying 34.227.213.82...
* TCP_NODELAY set
* Connected to www.httpbin.org (34.227.213.82) port 80 (#0)
> GET /get HTTP/1.1
Host: www.httpbin.org
User-Agent: curlInterface/GAP package
Accept: */*

< HTTP/1.1 200 OK
< Date: Thu, 08 Sep 2022 13:30:55 GMT
< Content-Type: application/json
< Content-Length: 277
< Connection: keep-alive
< Server: gunicorn/19.9.0
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< 
* Connection #0 to host www.httpbin.org left intact
     119 ms (0 ms GC) and 317KB allocated for basic.tst
testing: /usr/local/ca/gap-rsync/pkg/curlinterface/tst/errors.tst
       2 ms (0 ms GC) and 68.8KB allocated for errors.tst
-----------------------------------
total       121 ms (0 ms GC) and 386KB allocated
              0 failures in 2 files

#I  No errors detected while testing

13> gap
 ┌───────┐   GAP 4.12.0 of 2022-08-18
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: x86_64-pc-linux-gnu-default64-kv8
 Configuration:  gmp 6.1.2, GASMAN, readline
 Loading the library and packages ...
 Packages:   Alnuth 3.2.1, AtlasRep 2.1.5, AutPGrp 1.11, Browse 1.8.14, 
             CTblLib 1.3.4, EDIM 1.3.5, FactInt 1.6.3, Forms 1.2.8, 
             GAPDoc 1.6.6, genss 1.6.7, IO 4.7.2, orb 4.8.5, Polycyclic 2.16, 
             PrimGrp 3.4.2, recog 1.3.2, SmallGrp 1.5, SpinSym 1.5.2, 
             TomLib 1.2.9, TransGrp 3.6.3
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
gap> TestDirectory("tst");
Architecture: x86_64-pc-linux-gnu-default64-kv8

testing: tst/basic.tst
# line 160 of 179 (89%)*   Trying 3.94.154.124...
* TCP_NODELAY set
* Connected to www.httpbin.org (3.94.154.124) port 80 (#0)
> GET /get HTTP/1.1
Host: www.httpbin.org
User-Agent: curlInterface/GAP package
Accept: */*

< HTTP/1.1 200 OK
< Date: Thu, 08 Sep 2022 13:32:21 GMT
< Content-Type: application/json
< Content-Length: 277
< Connection: keep-alive
< Server: gunicorn/19.9.0
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< 
* Connection #0 to host www.httpbin.org left intact
########> Diff in tst/basic.tst:76  
# Input is:
PositionSublist(r.result, "my\\u0000first\\u0000field") <> fail;
# Expected output:
true
# But found:
false
########
     178 ms (0 ms GC) and 846KB allocated for basic.tst
testing: tst/errors.tst
       0 ms (0 ms GC) and 68.6KB allocated for errors.tst
-----------------------------------
total       178 ms (0 ms GC) and 914KB allocated
              1 failures in 1 of 2 files

#I  Errors detected while testing

false
gap> 
14> gap
 ┌───────┐   GAP 4.12.0 of 2022-08-18
 │  GAP  │   https://www.gap-system.org
 └───────┘   Architecture: x86_64-pc-linux-gnu-default64-kv8
 Configuration:  gmp 6.1.2, GASMAN, readline
 Loading the library and packages ...
 Packages:   Alnuth 3.2.1, AtlasRep 2.1.5, AutPGrp 1.11, Browse 1.8.14, 
             CTblLib 1.3.4, EDIM 1.3.5, FactInt 1.6.3, Forms 1.2.8, 
             GAPDoc 1.6.6, genss 1.6.7, IO 4.7.2, orb 4.8.5, Polycyclic 2.16, 
             PrimGrp 3.4.2, recog 1.3.2, SmallGrp 1.5, SpinSym 1.5.2, 
             TomLib 1.2.9, TransGrp 3.6.3
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
gap> TestDirectory("tst");
Architecture: x86_64-pc-linux-gnu-default64-kv8

testing: tst/basic.tst
# line 160 of 179 (89%)*   Trying 54.147.68.244...
* TCP_NODELAY set
* Connected to www.httpbin.org (54.147.68.244) port 80 (#0)
> GET /get HTTP/1.1
Host: www.httpbin.org
User-Agent: curlInterface/GAP package
Accept: */*

< HTTP/1.1 200 OK
< Date: Thu, 08 Sep 2022 13:38:49 GMT
< Content-Type: application/json
< Content-Length: 277
< Connection: keep-alive
< Server: gunicorn/19.9.0
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< 
* Connection #0 to host www.httpbin.org left intact
########> Diff in tst/basic.tst:63  
# Input is:
PositionSublist(r.result, "\"field1\": \"true\"") <> fail;
# Expected output:
true
# But found:
false
########
########> Diff in tst/basic.tst:65
# Input is:
PositionSublist(r.result, "\"field2\": \"17\"") <> fail;  
# Expected output:
true
# But found:
false
########
########> Diff in tst/basic.tst:120
# Input is:
PositionSublist(r.result, "\"animal\": \"tiger\"") <> fail;
# Expected output:
true
# But found:
false
########
########> Diff in tst/basic.tst:122
# Input is:
PositionSublist(r.result, "\"material\": \"cotton\"") <> fail;
# Expected output:
true
# But found:
false
########
     165 ms (0 ms GC) and 813KB allocated for basic.tst
testing: tst/errors.tst
       1 ms (0 ms GC) and 68.6KB allocated for errors.tst
-----------------------------------
total       166 ms (0 ms GC) and 882KB allocated
              4 failures in 1 of 2 files

#I  Errors detected while testing

false
fingolfin commented 2 years ago

I've now tried running TestDirectory("tst"); as you described, about 10 times each, with self-compiled GAP and curlinterface on the following machines; for the Linux machines I also tried with your rsync distro

I was not able to reproduce the issue on any of these.

It would be helpful to know more about what exactly goes wrong. Based on the failures you reported above I created this test sequence. Perhaps you can run it and tell us the resulting error.

post_string := List("animal=tiger&material=cotton", letter -> letter);;
while true do
    Print("sending query\n");
    r := PostToURL("httpbin.org/post", post_string, rec(verifyCert := true));;
    if PositionSublist(r.result, "\"animal\": \"tiger\"") = fail then
        Error("bad result: ", r.result);
    fi;
    Sleep(1);
od;

For reference, if all went well then r.result should look something like this:

{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "animal": "tiger",
    "material": "cotton"
  },
  "headers": {
    "Accept": "*/*",
    "Content-Length": "28",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "curlInterface/GAP package",
    "X-Amzn-Trace-Id": "Root=1-632258f2-3883b8f161bb5a4816a9c1ce"
  },
  "json": null,
  "origin": "1.2.3.4",
  "url": "http://httpbin.org/post"
}
frankluebeck commented 2 years ago

The test loop in the previous loop runs without problems on all computers I tried.

Just now I can also no longer reproduce the problems on my notebook with private internet connection (shown in my previous post above). The call TestDirectory("tst"); in the curlinterface directory now needs reproducibly 2-3 seconds and shows no errors (just the somewhat disturbing output from tst/basic.tst, line 160).

But the original problem with CurlRequest("www.google.com", "HEAD" , ""); on our office computers remains (tried various machines with Debian 10 and Debian 11, also another user, and with -A and/or -r). The problem is the same with standalone curl, that is curl --head www.google.com in a shell. Some variants work (same in GAP with curlInterface, and standalone curl):

CurlRequest("www.google.com", "HEAD" , "");  # hangs
CurlRequest("http://www.google.com", "HEAD" , "");  # hangs
CurlRequest("www.google.com", "GET" , "");  # works
CurlRequest("https://www.google.com", "HEAD" , "");  # works
CurlRequest("www.heise.de", "HEAD" , "")  # works, sends 301 and redirects to https
fingolfin commented 2 years ago

Just a random guess, but: Could it be that the firewall for that machine blocks e.g. UDP port 443 for outward connections? These days google.com uses QUIC resp. HTTP/3 over that port instead of standard HTTP(S) over TCP port 443.

fingolfin commented 3 days ago

Never was able to reproduce this in any way, and since curl itself seems to be affected in the same way, it seems like a libcurl issue.