sqitchers / homebrew-sqitch

Homebrew Formulas for Sqitch
22 stars 10 forks source link

MacOS dependency error --with-mysql-support: Module 'DBD::mysql' is not installed #54

Closed smichel17 closed 2 years ago

smichel17 commented 2 years ago

This looks similar to https://github.com/sqitchers/homebrew-sqitch/issues/44. However, I did not use --without-mysql and the error isn't exactly the same.

This produced the error: Installing the dependencies failed: Module 'DBD::mysql' is not installed

Full log ``` ==> Downloading https://ghcr.io/v2/homebrew/core/lz4/manifests/1.9.3 ######################################################################## 100.0% ==> Downloading https://ghcr.io/v2/homebrew/core/lz4/blobs/sha256:5ed09cfe61aa92ae7e28a5a9 ==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:5ed09 ######################################################################## 100.0% ==> Downloading https://ghcr.io/v2/homebrew/core/protobuf/manifests/3.19.4 ######################################################################## 100.0% ==> Downloading https://ghcr.io/v2/homebrew/core/protobuf/blobs/sha256:ca9840b58a314543c0f ==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:ca984 ######################################################################## 100.0% ==> Downloading https://ghcr.io/v2/homebrew/core/zstd/manifests/1.5.2 ######################################################################## 100.0% ==> Downloading https://ghcr.io/v2/homebrew/core/zstd/blobs/sha256:92089ac665de71072f944a1 ==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:92089 ######################################################################## 100.0% ==> Downloading https://ghcr.io/v2/homebrew/core/mysql/manifests/8.0.28 ######################################################################## 100.0% ==> Downloading https://ghcr.io/v2/homebrew/core/mysql/blobs/sha256:723dee93398d4790ba6f54 ==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:723de ######################################################################## 100.0% ==> Downloading https://www.cpan.org/authors/id/D/DW/DWHEELER/App-Sqitch-v1.2.1.tar.gz Already downloaded: /Users/smichel/Library/Caches/Homebrew/downloads/343224499909ee312b73c86433aa92f25a15090e2cbae945c835c6674b616461--App-Sqitch-v1.2.1.tar.gz ==> Reinstalling sqitchers/sqitch/sqitch --with-postgres-support --with-sqlite-s ==> Installing dependencies for sqitchers/sqitch/sqitch: lz4, protobuf, zstd and mysql ==> Installing sqitchers/sqitch/sqitch dependency: lz4 ==> Pouring lz4--1.9.3.monterey.bottle.tar.gz 🍺 /usr/local/Cellar/lz4/1.9.3: 22 files, 647.8KB ==> Installing sqitchers/sqitch/sqitch dependency: protobuf ==> Pouring protobuf--3.19.4.monterey.bottle.tar.gz 🍺 /usr/local/Cellar/protobuf/3.19.4: 270 files, 19.6MB ==> Installing sqitchers/sqitch/sqitch dependency: zstd ==> Pouring zstd--1.5.2.monterey.bottle.tar.gz 🍺 /usr/local/Cellar/zstd/1.5.2: 31 files, 2.7MB ==> Installing sqitchers/sqitch/sqitch dependency: mysql ==> Pouring mysql--8.0.28.monterey.bottle.tar.gz ==> /usr/local/Cellar/mysql/8.0.28/bin/mysqld --initialize-insecure --user=smichel --based 🍺 /usr/local/Cellar/mysql/8.0.28: 304 files, 294MB ==> Installing sqitchers/sqitch/sqitch --with-postgres-support --with-sqlite-sup ==> cpanm --local-lib instutil --notest Menlo::CLI::Compat Module::Build ==> perl Build.PL --install_base /usr/local/Cellar/sqitch/v1.2.1 --etcdir /usr/local/etc/s ==> ./Build bundle Last 15 lines from /Users/smichel/Library/Logs/Homebrew/sqitch/03.Build: --> Working on Template Fetching http://www.cpan.org/authors/id/A/AT/ATOOMIC/Template-Toolkit-3.010.tar.gz ... OK Configuring Template-Toolkit-3.010 ... OK ==> Found dependencies: AppConfig --> Working on AppConfig Fetching http://www.cpan.org/authors/id/N/NE/NEILB/AppConfig-1.71.tar.gz ... OK Configuring AppConfig-1.71 ... OK Building AppConfig-1.71 ... OK Successfully installed AppConfig-1.71 Building Template-Toolkit-3.010 ... OK Successfully installed Template-Toolkit-3.010 ! Installing the dependencies failed: Module 'DBD::mysql' is not installed ! Bailing out the installation for App-Sqitch-v1.2.1. Error installing modules: 93 distributions installed If reporting this issue please do so at (not Homebrew/brew or Homebrew/core): https://github.com/sqitchers/homebrew-sqitch/issues These open issues may also help: After brew install sqitch --with-postgres-support, error deploying https://github.com/sqitchers/homebrew-sqitch/issues/46 ```

I tried uninstalling and then using the install command instead of reinstall and that didn't change anything. The log above is from the original reinstall command because it has the most complete output (includes mysql dependency installation).

theory commented 2 years ago

Ugh, cpam is not showing you where the build log file is :-(. Run it with --verbose and you should get more details on why DBD::mysql failed to install.

smichel17 commented 2 years ago

I tried running cpanm install DBD::mysql and got Can't link/include C library 'ssl', 'crypto', aborting

Full log ``` cpanm (App::cpanminus) 1.9018 on perl 5.034000 built for darwin-thread-multi-2level Work directory is /Users/smichel/.cpanm/work/1646269683.3508 You have make /usr/bin/make You have /usr/bin/curl: curl 7.77.0 (x86_64-apple-darwin21.0) libcurl/7.77.0 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.42.0 You have /usr/bin/tar: bsdtar 3.5.1 - libarchive 3.5.1 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.8 You have /usr/bin/unzip Searching install () on cpanmetadb ... install is up to date. (0.01) Searching DBD::mysql () on cpanmetadb ... --> Working on DBD::mysql Fetching http://www.cpan.org/authors/id/D/DV/DVEEDEN/DBD-mysql-4.050.tar.gz -> OK Unpacking DBD-mysql-4.050.tar.gz Entering DBD-mysql-4.050 Checking configure dependencies from META.json Checking if you have Data::Dumper 0 ... Yes (2.179) Checking if you have ExtUtils::MakeMaker 6.58 ... Yes (7.62) Checking if you have DBI 1.609 ... Yes (1.643) Checking if you have Devel::CheckLib 1.09 ... Yes (1.14) Configuring DBD-mysql-4.050 Running Makefile.PL PLEASE NOTE: For 'make test' to run properly, you must ensure that the database user 'smichel' can connect to your MySQL server and has the proper privileges that these tests require such as 'drop table', 'create table', 'drop procedure', 'create procedure' as well as others. mysql> grant all privileges on test.* to 'smichel'@'localhost' identified by 's3kr1t'; You can also optionally set the user to run 'make test' with: perl Makefile.PL --testuser=username I will use the following settings for compiling and testing: cflags (mysql_config) = -I/usr/local/opt/mysql/include/mysql embedded (guessed ) = ldflags (guessed ) = libs (mysql_config) = -L/usr/local/opt/mysql/lib -lmysqlclient -lz -lzstd -lssl -lcrypto -lresolv mysql_config (guessed ) = mysql_config nocatchstderr (default ) = 0 nofoundrows (default ) = 0 nossl (default ) = 0 testdb (default ) = test testhost (default ) = testpassword (default ) = testport (default ) = testsocket (default ) = testuser (guessed ) = smichel To change these settings, see 'perl Makefile.PL --help' and 'perldoc DBD::mysql::INSTALL'. Checking if libs are available for compiling... Can't link/include C library 'ssl', 'crypto', aborting. -> N/A Checking dependencies from META.json ... Checking if you have Test::Simple 0.90 ... Yes (1.302183) Checking if you have DBI 1.609 ... Yes (1.643) Checking if you have Test::Deep 0 ... Yes (1.130) Checking if you have Time::HiRes 0 ... Yes (1.9767) Checking if you have bigint 0 ... Yes (0.51) Checking if you have ExtUtils::MakeMaker 0 ... Yes (7.62) -> FAIL Configure failed for DBD-mysql-4.050. See /Users/smichel/.cpanm/work/1646269683.3508/build.log for details. ```

I also tried brew install openssl mysql-connector-c at the recommendation of https://metacpan.org/dist/DBD-mysql/view/lib/DBD/mysql/INSTALL.pod#macOS, which changed nothing.

smichel17 commented 2 years ago

So the --verbose option produced a LOT of output. Full 29k lines: sqitch-homebrew.log

Is there anything in particular I should keep an eye out for?

The perl part referenced at the bottom is missing a bunch of dependencies (looks like everything). Maybe this is a library path not being correctly specified?

~/Library/Logs/Homebrew/sqitch/02.perl ``` 2022-03-03 17:34:08 +0000 perl Build.PL --install_base /usr/local/Cellar/sqitch/v1.2.1 --etcdir /usr/local/etc/sqitch --verbose --with postgres --with sqlite --with mysql Checking prerequisites... requires: ! Algorithm::Backoff::Exponential is not installed ! Clone is not installed ! Config::GitLike is not installed ! DateTime is not installed ! DateTime::TimeZone is not installed ! Devel::StackTrace is not installed ! Encode::Locale is not installed ! Hash::Merge is not installed ! IO::Pager is not installed ! IPC::System::Simple is not installed ! List::MoreUtils is not installed ! Locale::Messages is not installed ! Locale::TextDomain is not installed ! Moo is not installed ! Moo::Role is not installed ! Path::Class is not installed ! PerlIO::utf8_strict is not installed ! Pod::Find is not installed ! StackTrace::Auto is not installed ! String::Formatter is not installed ! Sub::Exporter is not installed ! Sub::Exporter::Util is not installed ! Template::Tiny is not installed ! Throwable is not installed ! Try::Tiny is not installed ! Type::Library is not installed ! Type::Utils is not installed ! Types::Standard is not installed ! URI::db is not installed ! namespace::autoclean is not installed test_requires: ! IO::Pager is not installed ! Module::Runtime is not installed ! Path::Class is not installed ! Test::Dir is not installed ! Test::Exception is not installed ! Test::File is not installed ! Test::File::Contents is not installed ! Test::MockModule is not installed ! Test::MockObject::Extends is not installed ! Test::NoWarnings is not installed ! Test::Warn is not installed recommends: * Class::XSAccessor is not installed * Template is not installed * Type::Tiny::XS is not installed ERRORS/WARNINGS FOUND IN PREREQUISITES. You may wish to install the versions of the modules indicated above before proceeding with this installation Run 'Build installdeps' to install missing prerequisites. Checking whether your kit is complete... Looks good Created MYMETA.yml and MYMETA.json Creating new 'Build' script for 'App-Sqitch' version 'v1.2.1' ```
smichel17 commented 2 years ago

I also started mysql, connected and granted the permissions suggested in https://github.com/sqitchers/homebrew-sqitch/issues/44#issuecomment-528654627. Or rather, ran the equivalent, since that syntax is no longer valid in mysql 8:

$ brew service start mysql
$ mysql -uroot
mysql> create user 'smichel'@'localhost' identified by 's3kr1t';
mysql> grant all on test.* to 'smichel'@'localhost';

No effect that I could tell.

smichel17 commented 2 years ago

For the record: I expect that this is probably a missing dependency or variable on my machine, and I have people at my company I can ask for help getting sqitch installed. I'm mainly posting this to help the next person, since #44 wasn't enough to get me past it. I expect the outcome will be some extra documentation in the mysql section of the readme.

theory commented 2 years ago

Is there anything in particular I should keep an eye out for?

Yeah, this is the bit:

! Configure failed for DBD-mysql-4.050. See /private/tmp/sqitch-20220303-14394-vczjc4/App-Sqitch-v1.2.1/.brew_home/.cpanm/work/1646328849.27634/build.log for details.

Which is annoying tbh; --verbose should have emitted the error. I suspect the error you encountered here is the issue. macOS 12 has made some of the crypto stuff a bit trickier; I made this function to set up variables so things can find OpenSSL:

ssl_init () {
    v=$1 
    export PATH=$brew/opt/openssl$v/bin:$PATH 
    export OPENSSL_PREFIX=$brew/opt/openssl$v 
    export LDFLAGS=-L$brew/opt/openssl$v/lib 
    export CPPFLAGS=-I$brew/opt/openss$v/include 
}

Which, given that Homebrew doesn't pass environment variables through to builds for security reasons I'm not sure what to do about. I think you might need to ask your coworkers for some pointers, as this is about the limit of my knowledge of MySQL with Homebrew. :-(

smichel17 commented 2 years ago

Extra annoyance: it seems homebrew (or something) cleans up the /private/tmp files after it finishes the build, whether successful or not.

Maybe there's a way to disable this, but I hacked around it with this script ```sh #!/usr/bin/env bash while : do for logfile in /private/tmp/sqitch-*/App-Sqitch-v1.2.1/.brew_home/.cpanm/work/**/build.log do if test -f "$logfile" then echo logfile is: "$logfile" logdir="$HOME/sqitch-logs/$(basename $(dirname $(realpath $logfile)))" mkdir -p "$logdir" cp "$logfile" "$logdir/build.log" else echo skipping, no file "$logfile" fi done sleep 1 done ``` Start it, run the build command in a different terminal, then kill it with ctrl-c once the build finishes
smichel17 commented 2 years ago

Here is the full log, but the part that seems relevant doesn't have any information, it just points me back to the very same file I'm already looking at :/ So I suspect it's the same error I saw when trying to install DBD::mysql manually, but am not sure. Note: this time I tried installing with --with-std-env, since that worked for a coworker at some point.

Searching DBD::mysql (4.018) on cpanmetadb ...
--> Working on DBD::mysql
Fetching http://www.cpan.org/authors/id/D/DV/DVEEDEN/DBD-mysql-4.050.tar.gz
-> OK
Unpacking DBD-mysql-4.050.tar.gz
Entering DBD-mysql-4.050
Checking configure dependencies from META.json
Checking if you have ExtUtils::MakeMaker 6.58 ... Yes (7.62)
Checking if you have DBI 1.609 ... Yes (1.643)
Checking if you have Devel::CheckLib 1.09 ... No
Checking if you have Data::Dumper 0 ... Yes (2.179)
==> Found dependencies: Devel::CheckLib
Searching Devel::CheckLib (1.09) on cpanmetadb ...
--> Working on Devel::CheckLib
Fetching http://www.cpan.org/authors/id/M/MA/MATTN/Devel-CheckLib-1.14.tar.gz
-> OK
Unpacking Devel-CheckLib-1.14.tar.gz
Entering Devel-CheckLib-1.14
Checking configure dependencies from META.json
Checking if you have ExtUtils::MakeMaker 6.58 ... Yes (7.62)
Configuring Devel-CheckLib-1.14
Running Makefile.PL
-> OK
Checking dependencies from MYMETA.json ...
Checking if you have File::Spec 0 ... Yes (3.80)
Checking if you have Exporter 0 ... Yes (5.76)
Checking if you have File::Temp 0.16 ... Yes (0.2311)
Building Devel-CheckLib-1.14
-> OK
Successfully installed Devel-CheckLib-1.14
Configuring DBD-mysql-4.050
Running Makefile.PL
-> N/A
Checking dependencies from META.json ...
Checking if you have DBI 1.609 ... Yes (1.643)
-> FAIL Configure failed for DBD-mysql-4.050. See /private/tmp/sqitch-20220304-17454-hx3gps/App-Sqitch-v1.2.1/.brew_home/.cpanm/work/1646410808.29427/build.log for details.
theory commented 2 years ago

Very weird; --verbose should be passed to cpanm as configured here. But it seems like a safe assumption that it's something to do with the environment. I would have expected the --with-std-env to work, though maybe you need to make sure that Homebrew's OpenSSL is in your path (not /usr/bin/openssl).

Pass --keep-tmp to get Homebrew to keep the tmp dir, then you should be able to look at the file that cpanm tells you to look in.

theory commented 2 years ago

Try this patch, which will install openssl as a prerequisite for MySQL.

--- a/Formula/sqitch.rb
+++ b/Formula/sqitch.rb
@@ -44,6 +44,7 @@ class Sqitch < Formula

   if build.with? "mysql-support"
     depends_on 'mysql' => :recommended
+    depends_on "openssl" => :build
   end

   if build.with? "firebird-support"
smichel17 commented 2 years ago

Okay, the patch did not work, but I did finally get this working. I confirmed that it was reproducible by uninstalling everything, checking that it didn't work, and installing again.

I had to do a combination of the solutions from this stackoverflow question AND use --with-std-env:

brew unlink mysql
brew install mysql-client
PATH="$(brew --prefix mysql-client)/bin:$PATH"
export LIBRARY_PATH="$(brew --prefix openssl)/lib:$LIBRARY_PATH"
brew install sqitch --with-postgres-support --with-sqlite-support --with-mysql-support --with-std-env

If anyone is struggling with this in the future: I advise trying different things running regular cpanm DBD::mysql, because that produced better error messages. Once I got that working, it worked in homebrew too, but I don't think I would have figured it out if I was only using debug output from homebrew.


@theory If you want to try to automate those steps in the homebrew installer, I'm happy to test. Otherwise, we can just add a note to the readme.

theory commented 2 years ago

There really ought to be a a way in brew to add openssl to the path and library path without --with-std-dev, no? Seems weird that it doesn't trust its own installed libraries.

smichel17 commented 2 years ago

I was just kind of trying things until I found something that works. I never saw the SSL error (or really anything useful) in the logs when compiling with homebrew, so maybe that isn't a problem?

I'm new to Mac and homebrew (longtime Linux user), so I'm flying blind here. With that context, I notice:

theory commented 2 years ago

I just spent some time trying to replicate this, but failed. --with-mysql-support works fine for me. Fort the DBD::mysql build, it emitted this config:

  cflags        (mysql_config) = -I/opt/homebrew/opt/mysql/include/mysql
  embedded      (guessed     ) = 
  ldflags       (guessed     ) = 
  libs          (mysql_config) = -L/opt/homebrew/opt/mysql/lib -lmysqlclient -lz -L/opt/homebrew/lib -lzstd -L/opt/homebrew/opt/openssl@1.1/lib -lssl -lcrypto -lresolv
  mysql_config  (guessed     ) = mysql_config
  nocatchstderr (default     ) = 0
  nofoundrows   (default     ) = 0
  nossl         (default     ) = 0
  testdb        (default     ) = test
  testhost      (default     ) = 
  testpassword  (default     ) = 
  testport      (default     ) = 
  testsocket    (default     ) = 
  testuser      (guessed     ) = david

So it knows about the openssl library location. The mysql-client formula is not installed:

$ brew info mysql-client
mysql-client: stable 8.0.29 (bottled) [keg-only]
Open source relational database management system
https://dev.mysql.com/doc/refman/8.0/en/
Not installed
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/mysql-client.rb
License: GPL-2.0-only with Universal-FOSS-exception-1.0
==> Dependencies
Build: cmake ✘, pkg-config ✔
Required: libevent ✔, libfido2 ✔, openssl@1.1 ✔, zstd ✔
==> Caveats
mysql-client is keg-only, which means it was not symlinked into /opt/homebrew,
because it conflicts with mysql (which contains client libraries).

==> Analytics
install: 36,388 (30 days), 84,235 (90 days), 310,764 (365 days)
install-on-request: 30,625 (30 days), 72,437 (90 days), 270,801 (365 days)
build-error: 76 (30 days)

Note the bit under caveats: the mysql formula includes the client libraries.

Neither $OPENSSL_PREFIX nor $LIBRARY_PATH is set in my environment.

I have to admit I'm stumped as to what caused the issue for you. I'm glad you go it worked out, though. Mind if I close this issue, @smichel17?

smichel17 commented 2 years ago

Sure. If I find more information later I can re-open, and at the very least now there's a record that might be useful to someone searching in the future.

theory commented 2 years ago

Totally, thank you!