search

johnmehr / gitup

A minimalist, dependency-free FreeBSD program to clone/pull Git repositories.
BSD 2-Clause "Simplified" License
51 stars 9 forks source link

HTTP proxy support? #48

Closed michael-o closed 3 years ago

michael-o commented 3 years ago

This might affect many corporate users which do not have an open network, but must use a HTTP proxy to tunnel all traffic. Such a proxy can be used in two ways:

  1. For regular HTTP (which does not apply here)
  2. Tuneling arbitrary traffic with CONNECT which would work here.

We used to have a closed network, but now have both a ZScaler based transparent proxy as well as an expliclit HTTP proxy which can route arbitrary traffic.

This is a request to evaluate HTTP proxy support in gitup. It could be done in two ways:

  1. Put it in gitup.conf
  2. Support HTTPS_PROXY and NO_PROXY environment variables
johnmehr commented 3 years ago

I believe I've got basic unauthenticated proxy support added now (it works when I route it through a local apache server configured as a forward proxy). How does it look? I'll start working on support for authenticated connections this weekend. Thanks!

michael-o commented 3 years ago

Testing... will get back to you soon

michael-o commented 3 years ago

The process crashes for me with a SIGSEGV. OpenSSL says goodbye.

tcpdump looks OK:

# tcpdump "tcp port 9400"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bge0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:51:50.959539 IP deblndw011x.ad001.siemens.net.50041 > de2.coia.siemens.net.9400: Flags [S], seq 809768188, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 3896082360 ecr 0], length 0
16:51:50.968558 IP de2.coia.siemens.net.9400 > deblndw011x.ad001.siemens.net.50041: Flags [S.], seq 1725422818, ack 809768189, win 65535, options [mss 1460,nop,wscale 5,sackOK,TS val 3232851007 ecr 3896082360], length 0
16:51:50.968572 IP deblndw011x.ad001.siemens.net.50041 > de2.coia.siemens.net.9400: Flags [.], ack 1, win 1027, options [nop,nop,TS val 3896082369 ecr 3232851007], length 0
16:51:51.171759 IP deblndw011x.ad001.siemens.net.50041 > de2.coia.siemens.net.9400: Flags [F.], seq 1, ack 1, win 1027, options [nop,nop,TS val 3896082572 ecr 3232851007], length 0
16:51:51.181995 IP de2.coia.siemens.net.9400 > deblndw011x.ad001.siemens.net.50041: Flags [.], ack 2, win 2081, options [nop,nop,TS val 3232851031 ecr 3896082572], length 0
16:51:51.182003 IP de2.coia.siemens.net.9400 > deblndw011x.ad001.siemens.net.50041: Flags [F.], seq 1, ack 2, win 2081, options [nop,nop,TS val 3232851031 ecr 3896082572], length 0
16:51:51.182011 IP deblndw011x.ad001.siemens.net.50041 > de2.coia.siemens.net.9400: Flags [.], ack 2, win 1026, options [nop,nop,TS val 3896082582 ecr 3232851031], length 0
^C
7 packets captured
380 packets received by filter
0 packets dropped by kernel

lldb says:

osipovmi@deblndw011x:~/var/Projekte/gitup (main *%=)
$ lldb --core gitup.core gitup
(lldb) target create "gitup" --core "gitup.core"
Core file '/net/home/osipovmi/var/Projekte/gitup/gitup.core' (x86_64) was loaded.
(lldb) bt all
* thread #1, name = 'gitup', stop reason = signal SIGSEGV
  * frame #0: 0x00000008005a5865 libcrypto.so.111`OPENSSL_cleanse at x86_64cpuid.S:231
    frame #1: 0x00000008004a31fb libcrypto.so.111`rand_drbg_cleanup_additional_data [inlined] rand_pool_reattach(pool=0x0000000800257980, buffer=<unavailable>) at rand_lib.c:571:5
    frame #2: 0x00000008004a31ec libcrypto.so.111`rand_drbg_cleanup_additional_data(pool=0x0000000800257980, out=<unavailable>) at rand_lib.c:301
    frame #3: 0x00000008004a49d6 libcrypto.so.111`RAND_DRBG_bytes(drbg=0x0000000800af7480, out="", outlen=0) at drbg_lib.c:686:9
    frame #4: 0x00000008002cfe88 libssl.so.111`SSL_CTX_new(meth=0x00000008002e2810) at ssl_lib.c:3063:13
    frame #5: 0x000000000020c915 gitup`process_command [inlined] ssl_connect(connection=0x00007fffffffa710) at gitup.c:757:20
    frame #6: 0x000000000020c7b9 gitup`process_command(connection=0x00007fffffffa710, command="GET https://git.freebsd.org:443/src.git/info/refs?service=git-upload-pack HTTP/1.1\r\nHost: git.freebsd.org:443\r\nUser-Agent: gitup/0.91\r\nGit-Protocol: version=2\r\n\r\n") at gitup.c:815
    frame #7: 0x0000000000208d87 gitup`main [inlined] get_commit_details(connection=0x00007fffffffa710) at gitup.c:1138:2
    frame #8: 0x0000000000208d33 gitup`main(argc=<unavailable>, argv=0x0000000800acf1c0) at gitup.c:2423
    frame #9: 0x0000000000205280 gitup`_start(ap=<unavailable>, cleanup=<unavailable>) at crt1.c:76:7

truss says:

socket(PF_INET,SOCK_STREAM,IPPROTO_TCP)      = 3 (0x3)
connect(3,{ AF_INET 194.145.60.253:9400 },16)    = 0 (0x0)
mmap(0x0,131072,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371457024 (0x800b2d000)
__sysctl("kern.ostype",2,0x800235c21,0x7fffffff7c30,0x0,0) = 0 (0x0)
__sysctl("kern.hostname",2,0x800235d21,0x7fffffff7c30,0x0,0) = 0 (0x0)
__sysctl("kern.osrelease",2,0x800235e21,0x7fffffff7c30,0x0,0) = 0 (0x0)
__sysctl("kern.version",2,0x800235f21,0x7fffffff7c30,0x0,0) = 0 (0x0)
__sysctl("hw.machine",2,0x800236021,0x7fffffff7c30,0x0,0) = 0 (0x0)
openat(AT_FDCWD,"/lib/libcrypto.so.111",O_RDONLY|O_CLOEXEC|O_VERIFY,00) = 4 (0x4)
fstat(4,{ mode=-r--r--r-- ,inode=197317,size=3062984,blksize=32768 }) = 0 (0x0)
close(4)                     = 0 (0x0)
fstatat(AT_FDCWD,"/usr/share/nls/C/libc.cat",0x7fffffff9110,0x0) ERR#2 'No such file or directory'
fstatat(AT_FDCWD,"/usr/share/nls/libc/C",0x7fffffff9110,0x0) ERR#2 'No such file or directory'
fstatat(AT_FDCWD,"/usr/local/share/nls/C/libc.cat",0x7fffffff9110,0x0) ERR#2 'No such file or directory'
fstatat(AT_FDCWD,"/usr/local/share/nls/libc/C",0x7fffffff9110,0x0) ERR#2 'No such file or directory'
mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371588096 (0x800b4d000)
mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371596288 (0x800b4f000)
mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371604480 (0x800b51000)
mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371612672 (0x800b53000)
mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371620864 (0x800b55000)
mmap(0x0,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371629056 (0x800b57000)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371076096 (0x800ad0000)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371637248 (0x800b59000)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371641344 (0x800b5a000)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371645440 (0x800b5b000)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371649536 (0x800b5c000)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371653632 (0x800b5d000)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371657728 (0x800b5e000)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371661824 (0x800b5f000)
mmap(0x0,20480,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371665920 (0x800b60000)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371686400 (0x800b65000)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371690496 (0x800b66000)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371694592 (0x800b67000)
issetugid()                  = 0 (0x0)
open("/etc/ssl/openssl.cnf",O_RDONLY,0666)   = 4 (0x4)
fstat(4,{ mode=-rw-r--r-- ,inode=197155,size=11136,blksize=32768 }) = 0 (0x0)
read(4,"# $FreeBSD: stable/12/crypto/ope"...,32768) = 11136 (0x2b80)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371698688 (0x800b68000)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371702784 (0x800b69000)
read(4,0x800ae6e00,32768)            = 0 (0x0)
close(4)                     = 0 (0x0)
openat(AT_FDCWD,"/usr/lib/libssl.so.111",O_RDONLY|O_CLOEXEC|O_VERIFY,00) = 4 (0x4)
fstat(4,{ mode=-r--r--r-- ,inode=963344,size=609472,blksize=32768 }) = 0 (0x0)
close(4)                     = 0 (0x0)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371706880 (0x800b6a000)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371710976 (0x800b6b000)
mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371715072 (0x800b6c000)
mmap(0x0,12288,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371719168 (0x800b6d000)
mmap(0x0,12288,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371731456 (0x800b70000)
mmap(0x0,12288,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371743744 (0x800b73000)
mmap(0x0,12288,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371756032 (0x800b76000)
mmap(0x0,12288,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371768320 (0x800b79000)
mmap(0x0,28672,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34371780608 (0x800b7c000)
getpid()                     = 64147 (0xfa93)
getrandom("\M^Y\^E\M-F}\M-s\^PJ\^R\^? \M^W"...,32,0) = 32 (0x20)
getpid()                     = 64147 (0xfa93)
getpid()                     = 64147 (0xfa93)
getpid()                     = 64147 (0xfa93)
getpid()                     = 64147 (0xfa93)
getpid()                     = 64147 (0xfa93)
getpid()                     = 64147 (0xfa93)
getpid()                     = 64147 (0xfa93)
getpid()                     = 64147 (0xfa93)
getpid()                     = 64147 (0xfa93)
SIGNAL 11 (SIGSEGV) code=SEGV_MAPERR trapno=12 addr=0x800b83000
process killed, signal = 11 (core dumped)

The code runs flawlessly w/o the proxy...I don't understand why this happens yet.

michael-o commented 3 years ago

Although this should not crash, the implementation is logically wrong. If not using HTTP -- which we aren't -- one must implement the CONNECT method to tunnel arbitrary traffic because the proxy does not know that the TLS traffic is HTTP (regardless of the port). An HTTP proxy which is the case here will never response to the TLS handshake.

michael-o commented 3 years ago

From OpenSSL:

$ openssl s_client -debug  de.coia.siemens.net:9400
CONNECTED(00000003)
write to 0x800bd1f00 [0x801258000] (323 bytes => 323 (0x143))
0000 - 16 03 01 01 3e 01 00 01-3a 03 03 62 06 d0 b4 65   ....>...:..b...e
0010 - d3 fd 4e 41 bc 39 17 63-97 25 49 80 63 f3 f3 62   ..NA.9.c.%I.c..b
0020 - 28 f4 7a 6b 8e 88 01 59-57 56 cb 20 ae 05 48 0d   (.zk...YWV. ..H.
0030 - 43 bd eb d0 f2 22 62 fc-96 d4 f1 3e 6b 1f 55 ee   C...."b....>k.U.
0040 - b4 e8 b2 72 3c 01 7e 49-02 bb b4 7a 00 3e 13 02   ...r<.~I...z.>..
0050 - 13 03 13 01 c0 2c c0 30-00 9f cc a9 cc a8 cc aa   .....,.0........
0060 - c0 2b c0 2f 00 9e c0 24-c0 28 00 6b c0 23 c0 27   .+./...$.(.k.#.'
0070 - 00 67 c0 0a c0 14 00 39-c0 09 c0 13 00 33 00 9d   .g.....9.....3..
0080 - 00 9c 00 3d 00 3c 00 35-00 2f 00 ff 01 00 00 b3   ...=.<.5./......
0090 - 00 00 00 18 00 16 00 00-13 64 65 2e 63 6f 69 61   .........de.coia
00a0 - 2e 73 69 65 6d 65 6e 73-2e 6e 65 74 00 0b 00 04   .siemens.net....
00b0 - 03 00 01 02 00 0a 00 0c-00 0a 00 1d 00 17 00 1e   ................
00c0 - 00 19 00 18 00 23 00 00-00 16 00 00 00 17 00 00   .....#..........
00d0 - 00 0d 00 30 00 2e 04 03-05 03 06 03 08 07 08 08   ...0............
00e0 - 08 09 08 0a 08 0b 08 04-08 05 08 06 04 01 05 01   ................
00f0 - 06 01 03 03 02 03 03 01-02 01 03 02 02 02 04 02   ................
0100 - 05 02 06 02 00 2b 00 0b-0a 03 04 03 03 03 02 03   .....+..........
0110 - 01 03 00 00 2d 00 02 01-01 00 33 00 26 00 24 00   ....-.....3.&.$.
0120 - 1d 00 20 42 1b c9 16 29-83 1b 17 92 77 2c c3 58   .. B...)....w,.X
0130 - 65 45 6c f0 73 3f 4a 86-1e e8 f3 a2 b7 0f 12 c4   eEl.s?J.........
0140 - 57 d6 69                                          W.i
read from 0x800bd1f00 [0x80124c743] (5 bytes => -1 (0xFFFFFFFFFFFFFFFF))
write:errno=54
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 323 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
read from 0x800bd1f00 [0x800b3a000] (8192 bytes => 0 (0x0))
johnmehr commented 3 years ago

Sorry for the delay (I missed the comments in the last commit). Proxy connections are now being transmitted over HTTP and support for the HTTP_PROXY environment variable has been added. How does it look? Are you still getting SIGSEGVs?

michael-o commented 3 years ago

Testing now....

michael-o commented 3 years ago

No segmentation fault and the tunnel is established, but several issues are still very fishy here..I will go through:

$ truss -o out ./gitup -C gitup.conf stable -v2
# Configuration file: gitup.conf
# Host: git.freebsd.org
# Port: 443
# Proxy Host: de.coia.siemens.net
# Proxy Port: 9400
# Repository Path: https://git.freebsd.org:443/src.git
# Target Directory: /var/osipovmi/freebsd-src
CONNECT git.freebsd.org:443 HTTP/1.1
Host: git.freebsd.org:443

==> bytes sent: 67
==> bytes read: 65      bytes_expected: 0       total_bytes_read: 65
GET https://git.freebsd.org:443/src.git/info/refs?service=git-upload-pack HTTP/1                                                                 .1
Host: git.freebsd.org:443
User-Agent: gitup/0.91
Git-Protocol: version=2

Here is a misconception: Although you now have established a tunnel, you are still treating it as if your are accessing an HTTP sever, not HTTPS. The entire thing hangs. Let's look at at the truss output:

socket(PF_INET,SOCK_STREAM,IPPROTO_TCP)      = 3 (0x3)
connect(3,{ AF_INET 194.145.60.253:9400 },16)    = 0 (0x0)
setsockopt(3,SOL_SOCKET,SO_KEEPALIVE,0x7fffffff779c,4) = 0 (0x0)
setsockopt(3,SOL_SOCKET,SO_SNDBUF,0x7fffffff779c,4) = 0 (0x0)
setsockopt(3,SOL_SOCKET,SO_RCVBUF,0x7fffffff779c,4) = 0 (0x0)
setsockopt(3,SOL_SOCKET,SO_RCVTIMEO,0x7fffffff77a0,16) = 0 (0x0)
setsockopt(3,SOL_SOCKET,SO_SNDTIMEO,0x7fffffff77a0,16) = 0 (0x0)
write(3,"CONNECT git.freebsd.org:443 HTTP"...,67) = 67 (0x43)
write(2,"\r==> bytes sent: 67",19)       = 19 (0x13)
write(2,"\n",1)                  = 1 (0x1)
read(3,"HTTP/1.1 200 Connection Establis"...,4096) = 65 (0x41)
mmap(0x0,1052672,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34378612736 (0x801200000)
write(2,"\r==> bytes read: 65\tbytes_expe"...,58) = 58 (0x3a)
write(2,"\n",1)                  = 1 (0x1)
write(2,"GET https://git.freebsd.org:443/"...,164) = 164 (0xa4)
close(3)                     = 0 (0x0)

The proxy has truly established the connection, but verbose output does not print that. Not helpful for the user. After that you must now initiate the TLS session on that socket because the proxy will now act as a socket proxy simply forward TCP packets to the target on your behalf. The HTTP server at the other end expects a Client Hello instead of a GET.

See:

$ truss -o out openssl s_client -connect git.freebsd.org:443 -proxy de.coia.siemens.net:9400
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = git.freebsd.org
verify return:1
---
Certificate chain
 0 s:CN = git.freebsd.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGJTCCBQ2gAwIBAgISA1l9rrZXZTwRYB5AuyOPqrKUMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTAzMTMyMDE2MDhaFw0yMTA2MTEyMDE2MDhaMBoxGDAWBgNVBAMT
D2dpdC5mcmVlYnNkLm9yZzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIB
ALvQdnD+ylYtU+i0OuB+2lBhYAR2oOfioWn5xXLlybYlZmC6FpkonBeiqzpJa7c8
Nb+WA3R8BuMfBb4ggeKiZctwjKj45ZpdVLJ2EMNQe63WNL9D1ubzRaTMcL6iVbTI
nsEXSkVutb6m7yePXgCkmG6O6KG2eYeStgTRN5KqpBF2CfE/X1iP/G0Cn+mbFiHC
2enetsyUluaPhaZjLnfvOND/nnfEao3ldKiO0/1PY/DzzSyLze7hB1d2TBTmEAsC
wiNPN6w+BjMiGgl1XUoubiJeFNHyoEJX0iHlpNcbK9OQQvWb/X+JB10lRAvdYhp3
MHH1PM8Gm9/NCoOu+qooVgwIGjO8bd5qPRYN32kAvaaXWRuOtO4HDqE8rPyn2rKA
qNdmLdO/rC9mB4xHcCmQL2/SEk08l1cTYaTvLUuOe2NcTbPnAAdTqsDpBy+edbEe
Ti90msj+RyH6pEyzI3PAYO8UOGK0XwEA/OYkMdz9NH1vVgmIrpDDDPc5/lK6z+Hg
ZwClesD0Hr6rHIqFWzCZML6UXOceTn0DuU6n4mNSoQGWNayzPlsiEQsVTr7FX246
FnHI5Ya31qfQn5YzpxOkGSDj0VEY2F5aOicxrKPZ8h/OSE671V1W8s4pebcvEay6
ET5L7e3fb01E4HaoQ34E5kf+XxEwA47Q7KXmlsQQjWDVAgMBAAGjggJLMIICRzAO
BgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwG
A1UdEwEB/wQCMAAwHQYDVR0OBBYEFFQQfp2s44PayawEFI7lZu3eRf2aMB8GA1Ud
IwQYMBaAFBQusxe3WFbLrlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggr
BgEFBQcwAYYVaHR0cDovL3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRw
Oi8vcjMuaS5sZW5jci5vcmcvMBoGA1UdEQQTMBGCD2dpdC5mcmVlYnNkLm9yZzBM
BgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIB
FhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQUGCisGAQQB1nkCBAIEgfYE
gfMA8QB3AJQgvB6O1Y1siHMfgosiLA3R2k1ebE+UPWHbTi9YTaLCAAABeC1yTAYA
AAQDAEgwRgIhAINqupzNuQpa/D67mj6UY+idC9ibei08JfNnP/nD4eZBAiEAsvRT
CjHDtXOCzTgVMYFOKroImffqNcwsUi34EfJSWFAAdgB9PvL4j/+IVWgkwsDKnlKJ
eSvFDngJfy5ql2iZfiLw1wAAAXgtckxAAAAEAwBHMEUCIQDTuuOzDl4tFoFbdeE7
b3fIhfb5n3ZHoXPvGJ1s/P1tCgIgXMkCxuQqk1BkMcu13LPyLlmX5VSZO3K/DAsS
ZV6cvx4wDQYJKoZIhvcNAQELBQADggEBADAcR9zUhnlLMIQpESym7FclEuWmwPzL
D1wXaWaHGoMdbVx+0OEl41Di1IOBB2VUpSygZ+FtiPH4DFNQ6dYBAdCNAZqpl6+4
dJVmA/yimoxHfSDnTu9qlmxdYstpW91/G+zXYWl+faj8Kf6QZ7yq9hpbBLU412+7
pROLJ27dEEEdy5mGaVFlagPUiN8j1wzGW70fIEMKAl7BMom1MZfaKulR45jNYj9b
XZ4+O3TmBXd1UIAljPAZPelYflNJ/P3XHqsxYMze7EVG13Gk361DhnhwwD71DGD0
G4A3d4FPlpV26YA0TiklH9EShXcxduUBdBnNU0IOj70Twb+sa2j6EAs=
-----END CERTIFICATE-----
subject=CN = git.freebsd.org

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3707 bytes and written 456 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 57B57A1B14EBC052DDE869B93A2EBBD9CD670161469C657C1E9F2BEAA5EF464C
    Session-ID-ctx:
    Master-Key: 570A7A256514164078B61A1AF013A056FE2BF3F8BF2F4F45CB445F4E30F12AC35BA432EAFC2E7B3AF86AF8FE52E0C328
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - d3 59 4d 40 a6 73 00 b0-7c ef 95 c1 49 fa 2d a0   .YM@.s..|...I.-.
    0010 - 8a 75 ec a6 5c 5f 73 a4-30 e8 3c af 04 f2 83 8e   .u..\_s.0.<.....
    0020 - 6c 3b b6 f9 dd e5 bd ca-06 53 6c 22 e8 95 d3 9a   l;.......Sl"....
    0030 - 59 25 68 56 5c 8a 51 fc-b8 2d 0b 8b e4 f4 3e 25   Y%hV\.Q..-....>%
    0040 - c6 5c 3b 7c 8f 25 1a 9b-a5 b7 17 fd 02 84 cd 59   .\;|.%.........Y
    0050 - 90 c3 dd 76 5c 01 52 43-3e 10 47 54 e0 ff 65 38   ...v\.RC>.GT..e8
    0060 - 8c 11 73 9b 08 d7 84 6a-14 bc b3 fd 4f a0 c0 ff   ..s....j....O...
    0070 - 89 2d 95 34 38 9f 94 ce-78 20 f9 ec 86 58 06 68   .-.48...x ...X.h
    0080 - 9c 28 cb 34 18 b2 43 67-52 a9 5e e1 22 2c ea 93   .(.4..CgR.^.",..
    0090 - ac 65 c3 79 f7 92 e1 4d-62 d0 7e 10 43 5b c7 19   .e.y...Mb.~.C[..
    00a0 - af 00 89 77 33 1f 42 f1-2e c4 04 f5 73 17 94 62   ...w3.B.....s..b
    00b0 - 4c b1 a1 6b 5b b0 aa 91-46 70 e4 cd 67 ac 5b c1   L..k[...Fp..g.[.
    00c0 - 14 cb 01 01 c5 60 eb 49-cd 28 a6 8b 78 52 ac c5   .....`.I.(..xR..

    Start Time: 1616274270
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

out:

write(1,"CONNECTED(00000003)\n",20)      = 20 (0x14)
write(3,"CONNECT git.freebsd.org:443 HTTP/1.0\r\n\r\n",40) = 40 (0x28)
read(3,"HTTP/1.1 200 Connection Established\r\nProxy-Agent: Zscaler/6.0\r\n\r\n",4096) = 65 (0x41)
select(4,{ 3 },{ 3 },0x0,0x0)            = 1 (0x1)
getpid()                     = 50231 (0xc437)
getpid()                     = 50231 (0xc437)
getpid()                     = 50231 (0xc437)
getpid()                     = 50231 (0xc437)
getpid()                     = 50231 (0xc437)
getpid()                     = 50231 (0xc437)
write(3,"\^V\^C\^A\^A>\^A\0\^A:\^C\^Cz\M-p\^V\^N\M-g\aY\M-Dl\M-W\M-C\M^?\^F\M^^';\^O\M^A:\M^?\M^J\M-#\M-"`\M-S\\k\M-?\M-9\M^X\M^YP 1\0"...,323) = 323 (0x143)
read(3,"\^V\^C\^C\0E",5)             = 5 (0x5)
read(3,"\^B\0\0A\^C\^C\M-^\^O\M^E\M-C\M-z\M-)c\M-%\M-((\M^N\M-f\M^R)F~O-P\M-M\M-p%`]\M-z\M^S\a\M^I\M-F\M-fI'\0\M-@0\0\0\^Y\M^?\^A\0\^A\0"...,69) = 69 (0x45)
read(3,"\^V\^C\^C\n\M^_",5)          = 5 (0x5)
read(3,"\v\0\n\M^[\0\n\M^X\0\^F)0\M^B\^F%0\M^B\^E\r\240\^C\^B\^A\^B\^B\^R\^CY}\M-.\M-6We<\^Q`\^^@\M-;#\M^O\M-*\M-2\M^T0\r\^F\t*\M^FH\M^F"...,2719) = 2719 (0xa9f)
johnmehr commented 3 years ago

How is it working for you now?

michael-o commented 3 years ago

Will test again today or tomorrow.

michael-o commented 3 years ago

It crashes again with a SIGSEGV, we are back to the actual issue. The code still have some issues, I will go through. get_commit_details: |-- CONNECT => process_command() |---- server_connect()

You must read the response, don't expect a body, inspect status code for 2xx and fail if not 2xx.

The course must be like this (if a proxy is used) in pseudo code:

connect(proxy_host:proxy_port)
send("CONNECT")
read_response()
if (status >=200 && status< 300)
  we have a tunnel
else
  fail and break
setup_openssl(): create context, set flags, associate socket FD
perform_commands()

You are setting up OpenSSL and associating way too early. This might be the reason for the crash.

michael-o commented 3 years ago

I would recommend to decouple handling. This means that you should create a create_tunnel() function and the rest of the code shall operate on that new tunneled socket.

johnmehr commented 3 years ago

How does it look now?

pstef commented 3 years ago

I think it's not right to free like that within a loop; freeaddrinfo is supposed to do that itself. So perhaps you want to do it like this:

    temp = start;
    while (temp) {
        if (connection->socket_descriptor < 0) {
            if ((connection->socket_descriptor = socket(temp->ai_family, temp->ai_socktype, temp->ai_protocol)) < 0)
                err(EXIT_FAILURE, "connect_server: socket failure");

            if (connect(connection->socket_descriptor, temp->ai_addr, temp->ai_addrlen) < 0)
                err(EXIT_FAILURE, "connect_server: connect failure (%d)", errno);
        }
        temp = temp->ai_next;
    }
    freeaddrinfo(start);
michael-o commented 3 years ago

Will test..

johnmehr commented 3 years ago

I think it's not right to free like that within a loop; freeaddrinfo is supposed to do that itself. So perhaps you want to do it like this:

  temp = start;
  while (temp) {
      if (connection->socket_descriptor < 0) {
          if ((connection->socket_descriptor = socket(temp->ai_family, temp->ai_socktype, temp->ai_protocol)) < 0)
              err(EXIT_FAILURE, "connect_server: socket failure");

          if (connect(connection->socket_descriptor, temp->ai_addr, temp->ai_addrlen) < 0)
              err(EXIT_FAILURE, "connect_server: connect failure (%d)", errno);
      }
      temp = temp->ai_next;
  }
  freeaddrinfo(start);

Fixed. Thank you!

michael-o commented 3 years ago

It does not work, OpenSSL exits with an error and truss clearly shows why, the code performs a connect to the proxy, the next call closes the socket FD, reopens and does the TLS setup.

The code does a double connect:

    if (connection.proxy_host) {
        connect_server(&connection);
        create_tunnel(&connection);
    }

I would really cleanly decouple the connect(), create_tunnel(), setup_ssl() and process_command(), no nesting because this makes it hard to trace.

johnmehr commented 3 years ago

The four functions have been decoupled (thank you!). How does it look now?

michael-o commented 3 years ago

This looks much much better now. It just works. It has easily cloned src through our proxy. I have identified some other issues, but not necessary related to this which I will report separately. Though, one thing: https://github.com/johnmehr/gitup/blob/2547cfcaf9ddf20b4ebb9b6aac6b7b1c0fc7d56d/gitup.c#L985-L988

This is not neccessarily correct. There is no response body >= 200 && < 300, but with >= 300 there can be chunked response body to denote an error. I have a proxy which rejects connections, luckily it uses Content-Length and not TE: chunked:

$ truss -o out ./gitup -C gitup.conf -v0 stable
# Configuration file: gitup.conf
gitup: process_command: read failure:
HTTP/1.1 403 Forbidden
Content-Type: text/html
Server: Zscaler/6.0
Cache-Control: no-cache
Content-length: 15219

<!--# Id: internetaccess.html 115044 2016-03-18 17:59:46Z szhang -->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3c.org/TR/1999/REC-html401-19991224/loose.dtd">
<html>
<head>
<meta name="description" content="Zscaler makes the internet safe for businesses by protecting their employees from malware, viruses, and other security threats.">
...

It make sense to process response line, decide and then move on?

johnmehr commented 3 years ago

How does it look now?

michael-o commented 3 years ago

Looks good enough now for me. Closing this. Thank you!

johnmehr commented 3 years ago

Thank you for all of your help with this and your patience helping me understand. It is very much appreciated!

michael-o commented 3 years ago

Thank you for all of your help with this and your patience helping me understand. It is very much appreciated!

Very much welcome. Expect more to come!

I have now installed gitup on a few hosts and ist works just fine with the proxy support for the entire ports tree from main.