OpenSSL@1.1 Migration [Tracking]

DomT4 commented 8 years ago

Borrowing my own comment from another thread:

I don't have a lot of interest in carrying around what is essentially a feature patch (support for 1.1.0) given OpenSSL are supporting 1.0.2 as an LTS branch. Not planning to leap over upstream releases unless absolutely necessary for whatever reason.

Transition is also complicated by potentially overlapping linkage, so for example, if you brew install curl --with-openssl against OpenSSL 1.1.0 but you add --with-libssh2 as well which remains linked against OpenSSL 1.0.2, we're not really sure how predictably that'll behave. Ideally, the plan is:

Anything standalone (i.e. nothing in brew uses) that supports OpenSSL 1.1.0 can use this new formula as the dependency.

Anything else needs to be upgraded by tree, so to update curl you'd want libssh2 to be using OpenSSL 1.1.0 first, etc.

We may well change that plan as time passes; If I could I'd have the entire ecosystem using OpenSSL 1.1.0 tomorrow but it really depends on how quickly we can get this done without breaking significant things for users.

Immediate potential deps tree:

brew deps --include-build --include-optional --1 <formula>

Usage or potential usage list:

brew uses <formula> --include-build --include-optional

Generate a build report for addition to the list:

brew gist-logs <formula>

Fair warning: This is going to be a messy list.

[ ] activemq-cpp - Not used by anything else - Fails to build - https://gist.github.com/anonymous/55392af913363942490f5df2f29fc6ce
[ ] afflib - Used optionally by bulk_extractor, sleuthkit & autopsy - Fails to build, fixed upstream already, will be in 3.7.9 (?) release.
[ ] aircrack-ng - Not used by anything else - Fails to build - https://gist.github.com/anonymous/241dee104589d7a37a4ed3861622f0a9
[ ] alpine - Not used by anything else - Fails to build - https://gist.github.com/anonymous/d8fe6343d990cad18a988efa708fc666
[ ] amap - Not used by anything else - Builds/runs fine, seemingly, once appdefs files are installed.
[ ] ansible - Not used by anything else - Fails to build, pyOpenSSL - https://gist.github.com/anonymous/d691b6d13da3a8ee264b4f84c77ef3f0
[ ] apib - Not used by anything else - Fails to build - https://gist.github.com/anonymous/845ffcb3f795d69c7662e0528bc4e2c2
[ ] apr-util - Used by subversion, apib - Builds fine, no idea how to test properly.
[ ] arangodb - Not used by anything else - Fails to build - https://gist.github.com/anonymous/060fa5ca54329bca6713ffc98b6261de
[ ] asio - Used by cpp-netlib - Fails to build in a big way - https://gist.github.com/anonymous/47f8889e59d9194fd12f875540e7ab11
[ ] axel - Not used by anything else - Fails to build, fixed upstream already, should be in 2.12 (?).
[ ] bacula-fd - Not used by anything else - Fails to build - https://gist.github.com/anonymous/8780cbc5dbf28e25d2db7429ab689a74
[ ] bbftp-client - Not used by anything else - Fails to build - https://gist.github.com/anonymous/1bbe2a9e25e38605fb6657066aaef693
[ ] bibtexconv - Not used by anything else - Builds/runs fine (Only needs MD5 crypto stuff).
[ ] bigloo - Not used by anything else - Fails to build - https://gist.github.com/anonymous/161b8539d919e2984c669df34e62c5a0
[ ] bind - Used by dnsperf - Fails to build - https://gist.github.com/anonymous/68c8cf065be6d48a408126b1ccd864e4
[ ] bip - Not used by anything else - Fails to build - https://gist.github.com/anonymous/4216da32770ca385e32e5c6a69a65ca0
[ ] bitchx - Not used by anything else - Fails to build - https://gist.github.com/anonymous/eb52052681e870c52a8438d8dfc1eb3c
[ ] botan - Used by monotone, qca & softhsm - Fails to build - https://gist.github.com/anonymous/2dd98429848c11f28898517e6196e6c9
[ ] bro - Not used by anything else - Fails to build, looks unrelated - https://gist.github.com/anonymous/a7e496d762267fff39dd2292c33a5a15
[ ] btpd - Not used by anything else - Builds fine, looks like only uses SHA1 from Crypto.
[ ] bulk_extractor - Not used by anything else - Fails to build - https://gist.github.com/anonymous/8dc86676f8ef80008a9c21fa78d644da
[ ] burp - Not used by anything else - Looks like it fails silently - https://gist.github.com/anonymous/8f9e2429168ceb4a81ee598e2a301948
[ ] cadaver
[ ] center-im
[ ] certbot
[ ] cfengine
[ ] charm-tools
[ ] cherokee
[ ] clamav
[ ] conan
[ ] cpp-netlib
[ ] cppcms
[ ] cpprestsdk
[ ] csup
[ ] ctorrent
[ ] ctunnel
[ ] curl
[ ] cvsync
[ ] davix
[ ] dcmtk
[ ] dovecot
[ ] duo_unix
[ ] duplicity
[ ] efl
[ ] ejabberd
[ ] ekg2
[ ] elinks
[ ] engine_pkcs11
[ ] epic5
[ ] eralchemy
[ ] erlang
[ ] ettercap
[ ] exim
[ ] fastd
[ ] fetchmail
[ ] ffmpeg
[ ] folly
[ ] fossil
[ ] fq
[ ] frag_find
[ ] freeradius-server
[ ] freeswitch
[ ] freetds
[ ] ftimes
[ ] gammu
[ ] gearman
[ ] getdns
[ ] getxbook
[ ] git-crypt
[ ] gkrellm
[ ] globus-toolkit
[ ] gloox
[ ] gpac
[ ] groonga
[ ] gsoap
[ ] gst-plugins-bad
[ ] gtmess
[ ] gwenhywfar
[ ] h2o
[ ] haproxy
[ ] hashpump
[ ] http_load
[ ] httperf
[ ] httping
[ ] httrack
[ ] hydra
[ ] icecast
[ ] ike-scan
[ ] imap-uw
[ ] imapfilter
[ ] innotop
[ ] inspircd
[ ] ipmitool
[ ] ipmiutil
[ ] ircd-hybrid
[ ] ircii
[ ] irods
[ ] irssi
[ ] isync
[ ] john-jumbo
[ ] knot
[ ] kore
[ ] lastpass-cli
[ ] ld64
[ ] ldapvi
[ ] ldid
[ ] ldns
[ ] lftp
[ ] libcapn
[ ] libcouchbase
[ ] libdap
[ ] libevent
[ ] libewf
[ ] libexosip
[ ] libfreefare
[ ] libgda
[ ] libimobiledevice
[ ] liblacewing
[ ] libmowgli
[ ] liboauth
[ ] libopendkim
[ ] libopkele
[ ] libp11
[ ] librdkafka
[ ] libre
[ ] libslax
[ ] libssh
[ ] libssh2
[ ] libstrophe
[ ] libtins
[ ] libtorrent-rasterbar
[ ] libu2f-server
[ ] libwebsockets
[ ] libxmlsec1
[ ] libzdb
[ ] lighttpd
[ ] links
[ ] lrdf
[ ] luvit
[ ] lynx
[ ] makepkg
[ ] mariadb-connector-c
[ ] mariadb
[ ] md5sha1sum
[ ] megatools
[ ] midnight-commander
[ ] mit-scheme
[ ] mitmproxy
[ ] mktorrent
[ ] monetdb
[ ] mongo-c
[ ] mongodb
[ ] mongoose
[ ] monit
[ ] monkeysphere
[ ] mosquitto
[ ] mpop
[ ] mpw
[ ] msmtp
[ ] mutt
[ ] mycli
[ ] mydumper
[ ] mysql-cluster
[ ] mysql-connector-c++
[ ] mysql
[ ] mytop
[ ] nagios-plugins
[ ] ncrack
[ ] neko
[ ] neon
[ ] net-snmp
[ ] nghttp2
[ ] nginx
[ ] ngircd
[ ] nmap
[ ] node-build
[ ] node
[ ] nrpe
[ ] nsd
[ ] nut
[ ] nzbget
[ ] onepass
[ ] ooniprobe
[ ] opensaml
[ ] opensc
[ ] openvpn
[ ] ophcrack
[ ] opusfile
[ ] osc
[ ] osslsigncode
[ ] passenger
[ ] pdftoedn
[ ] pdns
[ ] pdnsrec
[ ] percona-server
[ ] percona-toolkit
[ ] percona-xtrabackup
[ ] pev
[ ] pgcli
[ ] phantomjs
[ ] pincaster
[ ] pjproject
[ ] pkcs11-helper
[ ] poco
[ ] podofo
[ ] postgres-xc
[ ] postgresql
[ ] pound
[ ] profanity
[ ] proxytunnel
[ ] psqlodbc
[ ] pulledpork
[ ] pulseaudio
[ ] pure-ftpd
[ ] pwntools
[ ] pwsafe
[ ] pyenv
[ ] pypy
[ ] pypy3
[ ] python
[ ] python3
[ ] qca
[ ] qt
[ ] rabbitmq-c
[ ] radare2
[ ] rethinkdb
[ ] rtags
[ ] rtmpdump
[ ] ruby-build
[ ] ruby
[ ] rust
[ ] saltstack
[ ] sane-backends
[ ] sblim-sfcc
[ ] scamper
[ ] scrypt
[ ] sdhash
[ ] sgfutils
[ ] shadowsocks-libev
[ ] shairport-sync
[ ] shairport
[ ] siege
[ ] silc-client
[ ] sipp
[ ] sipsak
[ ] skipfish
[ ] slowhttptest
[ ] slrn
[ ] sngrep
[ ] snort
[ ] snownews
[ ] socat
[ ] sofia-sip
[ ] sops
[ ] spdylay
[ ] spiped
[ ] sqlcipher
[ ] squid
[ ] srtp
[ ] ssldump
[ ] sslscan
[ ] sslsplit
[ ] sstp-client
[ ] strongswan
[ ] stunnel
[ ] stuntman
[ ] subversion
[ ] swi-prolog
[ ] sylpheed
[ ] sysbench
[ ] szl
[ ] tarantool
[ ] tarsnap
[ ] tcpflow
[ ] testssl
[ ] thc-pptp-bruter
[ ] thrift
[ ] tinc
[ ] tiny-fugue
[ ] tlsdate
[ ] tn5250
[ ] tomcat-native
[ ] tor
[ ] trafficserver
[ ] u-boot-tools
[ ] unbound
[ ] unshield
[ ] upscaledb
[ ] urweb
[ ] uwsgi
[ ] virtuoso
[ ] voms
[ ] vsftpd
[ ] w3m
[ ] web100clt
[ ] webfs
[ ] wget
[ ] wimlib
[ ] wrk
[ ] x11vnc
[ ] xar-mackyle
[ ] xaric
[ ] xml-security-c
[ ] xml-tooling-c
[ ] xrootd
[ ] yara
[ ] yubico-piv-tool
[ ] zbackup
[ ] znc - Fails to build but some upstream activity https://github.com/jimloco/Csocket/pull/64 https://github.com/znc/znc/issues/1310 https://github.com/znc/znc/pull/1311

alex commented 8 years ago

Patches have landed in Python upstream, next releases on all branches will support 1.1.0.

JLHwung commented 8 years ago

@DomT4 How could one generate logs like this? Are there any commands or scripts? If so we can crowdsourcing the build logs on other formulas multiplying other platforms.

DomT4 commented 8 years ago

@JLHwung brew gist-logs <formula>.

JLHwung commented 8 years ago

@DomT4 How could one contribute on this list? For example

[ ] cadaver Not used by anything else, builds fine, no idea how to test properly.

DomT4 commented 8 years ago

Maintainers have direct access. Everyone else will need to leave a comment & wait for a maintainer to incorporate it into the master list. Generally the test process is something like:

Change any mention of depends_on "openssl" or Formula["openssl"] in formula to openssl@1.1.
brew install -s <formula>
brew test -v <formula> (If formula defines a test).
brew gist-logs <formula>
Dump comment here with the formula name, say build failed/succeeded, provide gist-logs.

JLHwung commented 8 years ago

@DomT4 don't forget to mention that use brew uses <formula> --include-build --include-optional to detect whether this formula is dependent of other formulas.

It is also worthy noting that some formula depends on OpenSSL optionally, so one needs go through the option to make formula building against openssl@1.1

Here is a list of c initials.

[ ] cadaver Not used by anything else, builds fine, no idea how to test properly. https://gist.github.com/anonymous/d74839b0aaf7a0d995ac0a0f92c86f09
[ ] center-im Not used by anything else, builds fine, test passed https://gist.github.com/anonymous/e93d431c89f47b9c91620d0ac474d30d
[ ] certbot Not used by anything else, builds failed, may be related with Python upstream https://gist.github.com/JLHwung/ec8cb634aa0e74dceee076e86a180055
[ ] cfengine Not used by anything else, builds failed https://gist.github.com/JLHwung/dbd4d57be2cb1a01d0bcfbf2d5d58561
[ ] charm-tools Not used by anything else, builds failed, seems same error with certbot https://gist.github.com/JLHwung/1d87435c2234bd0d17b59f449e418315
[ ] cherokee Not used by anything else, builds failed https://gist.github.com/79a4dce42d7eab3c7b8b0ffe6f332307
[ ] clamav Not used by anything else, builds failed on configuration, complained does not found OpenSSL even if openssl@1.1 is installed https://gist.github.com/JLHwung/ff173f97a66b089639c4af88604e4a73
[ ] conan Not used by anything else, builds failed https://gist.github.com/b9b2f51f62f0bc60d7875592273caf5d
[ ] cpp-netlib Used by osquery, builds failed https://gist.github.com/289b9eb3e6c97c9cb72d00796149b498
[ ] cppcms Not used by anything else, builds fine, test passed https://gist.github.com/3dba3e7934463ec2848baa7d702d3612
[ ] cpprestsdk Not used by anything else, builds failed https://gist.github.com/a31ee7e2370d0c384b8ec8c03dbde009
[ ] csup Not used by anything else, builds fine, test passed https://gist.github.com/71799fd0b3fe007360587b18b8504b52
[ ] ctorrent Not used by anything else, builds fine, test passed, no logs
[ ] ctunnel Not used by anything else, builds failed https://gist.github.com/896ab02c880146cc6fec3a239544705a
[ ] curl Used by freeswitch,lnav,saldl,strongswan,xml-tooling-c,homebrew/php/php54-http,homebrew/php/php56-http,git-ftp,powerman,shibboleth-sp,weechat,homebrew/php/php53-http,homebrew/php/php55-http,homebrew/php/phpbrew builds fine, test passed, but we knew libssh2 and nghttp2, etc does not support 1.1 now. https://gist.github.com/c85e2a1524e05cafe758bc0b54bae742
[ ] cvsync Not used by anything else, builds fine, test passed https://gist.github.com/7ea3b039da854ec5a7bb3da8f5deeb11

chdiza commented 8 years ago

mutt just officially gained support for building openssl 1.1 in HEAD.

vszakats commented 8 years ago

libssh2 does support openssl@1.1 in HEAD now.

JLHwung commented 8 years ago

@vszakats Thank you so much for reminding people of the progress. I raise pull request #4734 to have libssh2 HEAD depends on openssl@1.1.

DomT4 commented 8 years ago

Closing this because of https://github.com/Homebrew/brew/pull/1283. Thank you to those who helped work on it a little so far.

MikeMcQuaid commented 8 years ago

Let's leave it open, others will need to continue this work.

DomT4 commented 8 years ago

I was mostly hoping to avoid the notifications. I'll try the unsubscribe notifications button instead.

MikeMcQuaid commented 8 years ago

@DomT4 Thanks!

MikeMcQuaid commented 8 years ago

Based on https://github.com/Homebrew/homebrew-core/issues/6828 I think we should change approach here and stop migrating things to use openssl@1.1 until we can do it all in a large revision PR i.e. all the projects using OpenSSL 1.0 currently work with OpenSSL 1.1. Migrating things one-by-one is going to result in mixed dependency trees and that's not going to be good. CC @tdsmith and generally @Homebrew/maintainers for this.

imax9000 commented 7 years ago

Please consider rolling this back for broken packages. For example, ansible has been broken for 3 months now.

ilovezfs commented 7 years ago

@gelraen what issue(s) are you referring to specifically?

imax9000 commented 7 years ago

@ilovezfs brew install ansible chokes on installing openssl-1.1. I've worked it around by manually editing ansible.rb locally to use any version of openssl, not 1.1 specifically.

ilovezfs commented 7 years ago

@gelraen that's an issue specific to your system. openssl@1.1 is a non-optional dependency of ansible, and we have pre-built binary bottles for Sierra, El Capitan, and Yosemite, which were all built against 1.1. Please open an issue.

imax9000 commented 7 years ago

Sorry, I was indeed too quick to judge. It was broken for me because I have TOP env var set in my shell for completely unrelated reason (see gist for the full error message). Should I open an issue for general env sanitizing or just this specific instance?

ilovezfs commented 7 years ago

@gelraen you can make a note of the situation here: https://github.com/Homebrew/brew/issues/932

MikeMcQuaid commented 7 years ago

Closing as we're not actively migrating these formulae now.

Homebrew / homebrew-core

OpenSSL@1.1 Migration [Tracking] #4543