search

sagemath / binary-pkg

Tools for creating binary tarballs
Other
14 stars 11 forks source link

getting stuck at downloading git #8

Closed dimpase closed 5 years ago

dimpase commented 7 years ago

Not sure why this happens - I can this tarfile with the browser on this machine just fine. By the way, 2.4.0 is very old, the latest in 2.4 series is 2.4.11

Downloading https://www.kernel.org/pub/software/scm/git/git-2.4.0.tar.gz... [ERROR] urllib failed to download (reason: [Errno 54] Connection reset by peer): https://www.kernel.org/pub/software/scm/git/git-2.4.0.tar.gz Downloading https://www.kernel.org/pub/software/scm/git/git-2.4.0.tar.gz... [ERROR] urllib failed to download (reason: [Errno 54] Connection reset by peer): https://www.kernel.org/pub/software/scm/git/git-2.4.0.tar.gz Downloading https://www.kernel.org/pub/software/scm/git/git-2.4.0.tar.gz... [ERROR] urllib failed to download (reason: [Errno 54] Connection reset by peer): https://www.kernel.org/pub/software/scm/git/git-2.4.0.tar.gz [CRITICAL] You may wish to check your Internet connection or the remote server make[1]: *** [/Volumes/Sage/binary-pkg/tools/binary-pkg/activate] Error 127

vbraun commented 7 years ago

Is this on OSX? You need python/openssl with TLS 1.2 support, which Apple has failed to provide

osx:~ vbraun$ openssl s_client -connect www.kernel.org:443
CONNECTED(00000003)
write:errno=54

vs. expected

$ openssl s_client -connect www.kernel.org:443
CONNECTED(00000003)
depth=3 C = SE, O = AddTrust AB, OU = AddTrust External TTP Network, CN = AddTrust External CA Root
verify return:1
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
verify return:1
depth=1 C = FR, ST = Paris, L = Paris, O = Gandi, CN = Gandi Standard SSL CA 2
verify return:1
depth=0 OU = Domain Control Validated, OU = PositiveSSL Multi-Domain, CN = kernel.org
verify return:1

---
Certificate chain
 0 s:/OU=Domain Control Validated/OU=PositiveSSL Multi-Domain/CN=kernel.org
   i:/C=FR/ST=Paris/L=Paris/O=Gandi/CN=Gandi Standard SSL CA 2
 1 s:/C=FR/ST=Paris/L=Paris/O=Gandi/CN=Gandi Standard SSL CA 2
   i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
 2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification Authority
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFrDCCBJSgAwIBAgIRAJ+Fv2ZLDN2vylCGeVAbK+QwDQYJKoZIhvcNAQELBQAw
XzELMAkGA1UEBhMCRlIxDjAMBgNVBAgTBVBhcmlzMQ4wDAYDVQQHEwVQYXJpczEO
MAwGA1UEChMFR2FuZGkxIDAeBgNVBAMTF0dhbmRpIFN0YW5kYXJkIFNTTCBDQSAy
MB4XDTE2MTAxMTAwMDAwMFoXDTE5MTAxMTIzNTk1OVowWzEhMB8GA1UECxMYRG9t
YWluIENvbnRyb2wgVmFsaWRhdGVkMSEwHwYDVQQLExhQb3NpdGl2ZVNTTCBNdWx0
aS1Eb21haW4xEzARBgNVBAMTCmtlcm5lbC5vcmcwggGiMA0GCSqGSIb3DQEBAQUA
A4IBjwAwggGKAoIBgQCxqy/HJ6O+92eAyTSb871SgeGnmof5O3s/VIHMy12aoLJn
+dFu/Q7POBewQcqKSHxL+nteAjuKqEKuTkqm4NvhxY/112cCfymXMwwpfkmNVcr6
orw4bCsjkvsWBrw/uZNhr8ghgkK9SaU7NMgusVJ0Fu9dOu+qDlN2CkqBBz+5Bvdk
nxkSX8GpkNRYvdeloZEUXmPaNpZs5Tau/U5so1oQAarUSEqrcdjriXfV3cGcweBn
5KUldlB26t54Do1wYLAenYo9x4SBiQ15C55xIQVqErQrgukXlBaTU3fVZNXziGul
d5Ql/bYMBuU0U9oF7M8nNoXMvQ8wJisDlPDM8kFiQeHbvR3TTimpUKpJT1axwwV1
U0q71rCSzeGqlcUJTVen76Y3MvldgcnitTqFaQHIz76jqxthr06cN8S36VvMKb+R
fF3IDK03qvvjdnrPGAsFQuXmoKlBV+zWtPsHsZ7MxDlL62yPxqh3ZO3/dxk7yrhi
IeF9Y5DolSkca8hpO2UCAwEAAaOCAeUwggHhMB8GA1UdIwQYMBaAFLOQp9jJr07N
YTyffK1df0H9aTDqMB0GA1UdDgQWBBSzPZhuYWFOGfHdcTYTXTuexlV59jAOBgNV
HQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADAdBgNVHSUEFjAUBggrBgEFBQcDAQYI
KwYBBQUHAwIwSwYDVR0gBEQwQjA2BgsrBgEEAbIxAQICGjAnMCUGCCsGAQUFBwIB
FhlodHRwczovL2Nwcy51c2VydHJ1c3QuY29tMAgGBmeBDAECATBBBgNVHR8EOjA4
MDagNKAyhjBodHRwOi8vY3JsLnVzZXJ0cnVzdC5jb20vR2FuZGlTdGFuZGFyZFNT
TENBMi5jcmwwcwYIKwYBBQUHAQEEZzBlMDwGCCsGAQUFBzAChjBodHRwOi8vY3J0
LnVzZXJ0cnVzdC5jb20vR2FuZGlTdGFuZGFyZFNTTENBMi5jcnQwJQYIKwYBBQUH
MAGGGWh0dHA6Ly9vY3NwLnVzZXJ0cnVzdC5jb20wXQYDVR0RBFYwVIIKa2VybmVs
Lm9yZ4ISYXJjaGl2ZS5rZXJuZWwub3Jngg5naXQua2VybmVsLm9yZ4ISbWlycm9y
cy5rZXJuZWwub3Jngg53d3cua2VybmVsLm9yZzANBgkqhkiG9w0BAQsFAAOCAQEA
j4rwc5ecNh/sjPBNfPykgMMvCqlTtjvTtIOE4PRNn0C/lO6XFZdR23ze+OVcyAwM
Q2+sltel0i0XQ4ta75b+J7momrW4WCx0vWyGIi5VHrdMN4zqMlQ3DljJEyKbTT4u
GmATIN166ldJuu2grAmKAYA8hY6MD+2PSbgxLILTnqHsVHLEAvI81f1pLj1wNOW+
908gAan3FbNlaRNz6q+eQ1xhTVNaO3yJv8HooFifbEg/5oZcktXzKSMUmkgRUUsq
eEDxaDov7umHc+1dYxk2ooN2zKb9VoirAF0bh2s2UHUuJ6BAXtsBX0WdRiK9rsb+
QGEwGHMNc6oRvUYCvEWgKA==
-----END CERTIFICATE-----
subject=/OU=Domain Control Validated/OU=PositiveSSL Multi-Domain/CN=kernel.org
issuer=/C=FR/ST=Paris/L=Paris/O=Gandi/CN=Gandi Standard SSL CA 2
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5022 bytes and written 327 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 3072 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: D5B9CA2186316931C7CFAA8130063A5AB9A42460A7C581D15054892C6C7A7DE7
    Session-ID-ctx: 
    Master-Key: 94A25540B4A5B0BD1811DE461E7CBD5C928A672E9E1782F8FCC57D07ACAB6B50740293411E725671694FBC72342C0A54
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1478716088
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
read:errno=0
dimpase commented 7 years ago

indeed, it's OSX. Would the script run in sage -sh shell?

dimpase commented 7 years ago

OK, after rebuilding Sage's python with openssl and installing pyopnessl I'm getting

This is Mac OS X
Downloading https://www.kernel.org/pub/software/scm/git/git-2.4.0.tar.gz...
[ERROR] urllib failed to download (reason: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)): https://www.kernel.org/pub/software/scm/git/git-2.4.0.tar.gz

How does one get a working setup on OSX, I wonder...

vbraun commented 7 years ago

You also need the root certificates, which is what the errors says... And yes the OSX TSL misery sucks really hard.

You can try 

hit fetch http://www.kernel.org/pub/software/scm/git/git-2.4.0.tar.gz

to download over http, this should work and once the archive is stored in hashdist's cache it'll be used later instead of downloading.

dimpase commented 7 years ago

Unfortunately hit fetch ... ends with the same SSL error, whether it is http or https :-(

dimpase commented 7 years ago

Worked around by adding

import ssl
ssl._create_default_https_context = ssl._create_unverified_context

into hit script :-)

dimpase commented 7 years ago

Is it known to work on OSX 10.12? I get

...
Getting hashstack from github
Already up-to-date.
This is Mac OS X
[git] Building git/5bizhfr6sxp5, follow log with:
[git]   tail -f /Users/dima/.hashdist/tmp/git-5bizhfr6sxp5/_hashdist/build.log
[git|ERROR] Command '[u'/bin/bash', '_hashdist/build.sh']' returned non-zero exit status 2
[git|ERROR] command failed (code=2); raising
make[1]: *** [/Volumes/Sage/binary-pkg/tools/binary-pkg/activate] Error 127
make: *** [package-sage] Error 2
vbraun commented 7 years ago

Don't know, can you post the build.log?

dimpase commented 7 years ago

Away from the machine now, but I also tried to just build git on it, several versions, at sage -sh prompt, all failed. I suppose we should look at what homebrew does to build git... Probably building sage's git on OSX is broken for a while already. (unless it is yet another ObjectC-ism in the headers).

On 10 Nov 2016 9:35 am, "Volker Braun" notifications@github.com wrote:

Don't know, can you post the build.log?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/sagemath/binary-pkg/issues/8#issuecomment-259643449, or mute the thread https://github.com/notifications/unsubscribe-auth/ABN8HLzpV-HpyC_K3FQbsWknXY4r3Z-pks5q8uVRgaJpZM4KtzKf .

dimpase commented 7 years ago

in its infinite wisdom, Apple has /usr/bin/CC pointing to clang. To undo this, I'll need to disable SIP, etc, etc... So the building happens with clang instead of Sage's gcc, and fails with 

...
2016/11/10 18:03:40 - INFO: [package:run_job]     CC xdiff/xhistogram.o
2016/11/10 18:03:40 - INFO: [package:run_job]     AR xdiff/lib.a
2016/11/10 18:03:40 - INFO: [package:run_job]     LINK git-credential-store
2016/11/10 18:03:40 - INFO: [package:run_job] ld: library not found for -lrt
2016/11/10 18:03:40 - INFO: [package:run_job] clang: error: linker command failed with exit code 1 (use -v to see invocation)
dimpase commented 7 years ago

Well, I don't know the build system you use here, and don't see how to fix this, I can only say that in Sage we disable git's openssl : https://trac.sagemath.org/ticket/17091

dimpase commented 7 years ago

OK, installed openssl from source in /usr/local, with this, git must be configured with ./configure --with-openssl=/usr/local/

I still cannot build git-2.4.0, it end with the same error ( ld: library not found for -lrt)

However, I can build got 2.10.2. That is, the next question is how to change the configs to add --with-openssl=/usr/local/ to the git flags, and bump up git version. I dug it down to tools/toolaid/bootstrap-files/hashstack/pkgs/git.yaml, and I gather that the flag can be added to extra: list in name: configure record.

But I don;t understand how to change sources: record. While url: is clear, what the hell is key: and how do I get the right value? (Internet does not seem to know this...)

vbraun commented 7 years ago 
hit fetch http://host/filename

downloads and prints the new hash

dimpase commented 7 years ago

Do you know where in hasdist(?) source one can bump up git's version?

dimpase commented 7 years ago

ok, so it is 

sources:
- key: tar.gz:hv7pe5oybol2vjq7hnv6tu64kfrafzxw
  url: https://www.kernel.org/pub/software/scm/git/git-2.10.2.tar.gz

(wasted 20 minutes trying to make hashdist run on a system where python==python3...)

dimpase commented 7 years ago

Modified git.yaml to use git 2.10.2. Still it's not going to fly easily. Note that the default OSX python is built with broken openssl support, and so even though we have good openssl pre-installed, python is not using it, and one gets

Downloading https://www.kernel.org/pub/software/scm/git/git-2.10.2.tar.gz...
[ERROR] urllib failed to download (reason: [Errno 54] Connection reset by peer): https://www.kernel.org/pub/software/scm/git/git-2.10.2.tar.gz
Downloading https://www.kernel.org/pub/software/scm/git/git-2.10.2.tar.gz...

Switching to python built with good openssl results in already reported above SSL cert problem. Using the dodgy workaround I described above, modifying body of hit script, I am able to get past the initial steps, and to building of Sage proper. Will report the result.

dimpase commented 7 years ago

Sage build failed at Singular, with 

 ...
  CXXLD    syzextra.la
Making all in pyobject
  CXX      pyobject_la-pyobject.lo
g++: error: unrecognized command line option '-Wshorten-64-to-32'
make[10]: *** [pyobject_la-pyobject.lo] Error 1
make[9]: *** [all-recursive] Error 1

I don't know where that weird flag -Wshorten-64-to-32 came from .

dimpase commented 7 years ago

the latter is binary-pkg specific, I don't see this while building "normal" sage on this OSX 10.12 machine.

vbraun commented 5 years ago

This shouldn't be an issue any more, e.g. Apple finally updated their openssl.