search

dimitri / pgloader

Migrate to PostgreSQL in a single command!
http://pgloader.io
Other
5.33k stars 540 forks source link

Unable to install pgloader on macOS (Big Sur, M1) #1312

Open Wilson13 opened 3 years ago

Wilson13 commented 3 years ago

Hi,

I was trying to install pgloader via this command: brew install --HEAD pgloader

This is the output and error reported:

==> Cloning https://github.com/dimitri/pgloader.git
Updating /Users/wilson/Library/Caches/Homebrew/pgloader--git
==> Checking out branch master
Already on 'master'
Your branch is up to date with 'origin/master'.
HEAD is now at 3047c9a Typo fix in MS SQL documentation (#1242)
==> Patching
Error: Failure while executing; `patch -g 0 -f -p1` exited with 1. Here's the output:
can't find file to patch at input line 3
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/local-projects/cl-csv/parser.lisp
|+++ b/local-projects/cl-csv/parser.lisp
--------------------------
No file to patch.  Skipping patch.
1 out of 1 hunk ignored

Environment

MacBook Pro (13-inch, M1, 2020) macOS Big Sur v11.5.2 (20G95)

scrobot commented 3 years ago

The same problem. I've caught exception when I tried to install pgloader on my mac(M1) with brew

brew install pgloader
Error: pgloader: no bottle available!
You can try to install from source with:
  brew install --build-from-source pgloader
Please note building from source is unsupported. You will encounter build
failures with some formulae. If you experience any issues please create pull
requests instead of asking for help on Homebrew's GitHub, Twitter or any other
official channels.

And also when I tried to build it from sources

brew install --build-from-source pgloader
==> Downloading https://ghcr.io/v2/homebrew/core/sbcl/manifests/2.1.8
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/sbcl/blobs/sha256:df642cb657479a6ae84a6adfc22cafbcb876542fb9b69527b4ca2755ffc15e25
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:df642cb657479a6ae84a6adfc22cafbcb876542fb9b69527b4ca2755ffc15e25?se=2021-09-01T11%3A05%3A00Z&sig=PRsr08H9%2BDUlBx%2FU63PTpQiPMQgbju3I
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/buildapp/manifests/1.5.6_3
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/buildapp/blobs/sha256:3a4d0ac54096f853b8b271e02369dd2666e5b69c975d3b3f18ae6e54b4adf966
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:3a4d0ac54096f853b8b271e02369dd2666e5b69c975d3b3f18ae6e54b4adf966?se=2021-09-01T11%3A05%3A00Z&sig=9VlHNiuNx%2Bbb5%2BRioAZYiIjflXOfsV5O
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/m4/manifests/1.4.19
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/m4/blobs/sha256:ea1be04e51645f9e31c8d2cab6d144bb7f47efb1f35214b9f1014e27db5a3bc1
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:ea1be04e51645f9e31c8d2cab6d144bb7f47efb1f35214b9f1014e27db5a3bc1?se=2021-09-01T11%3A05%3A00Z&sig=hVtRGKxtmCfglpaOvqRC1XKyMvvGHqMgGFPI
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/libtool/manifests/2.4.6_4
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/libtool/blobs/sha256:a41a4872cdfaa34bb4723e728b73dd8c7a05725501a262bb41ad9af4e2fcd1d6
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:a41a4872cdfaa34bb4723e728b73dd8c7a05725501a262bb41ad9af4e2fcd1d6?se=2021-09-01T11%3A05%3A00Z&sig=WaStvjRIFMgQGjKclvKcX169YFmx8IHSjoXv
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/unixodbc/manifests/2.3.9_1
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/unixodbc/blobs/sha256:66e4b186a19526e02782557afe6926d2cfb9f372e94cbcc387f531b122f510e0
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:66e4b186a19526e02782557afe6926d2cfb9f372e94cbcc387f531b122f510e0?se=2021-09-01T11%3A05%3A00Z&sig=8gpUHK35tO%2Ba0RMi88ZmjufjVxLHHJIpfS
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/freetds/manifests/1.3.3
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/freetds/blobs/sha256:95f5ec9318ef76fc7fd19e1e4bfbb651ce0f70c0a445734ea2eb52c9431b5300
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:95f5ec9318ef76fc7fd19e1e4bfbb651ce0f70c0a445734ea2eb52c9431b5300?se=2021-09-01T11%3A05%3A00Z&sig=cfKiL1UHLyRHRYB7UKJtRwIqVOYKN9A%2FdW
######################################################################## 100.0%
==> Downloading https://github.com/dimitri/pgloader/releases/download/v3.6.2/pgloader-bundle-3.6.2.tgz
==> Downloading from https://github-releases.githubusercontent.com/594087/a86b1500-6c8d-11ea-86c4-5dd3e9eae544?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210901%2Fus-east-1%2Fs3%2Faws4_r
######################################################################## 100.0%
==> Installing dependencies for pgloader: sbcl, buildapp, m4, libtool, unixodbc and freetds
==> Installing pgloader dependency: sbcl
==> Pouring sbcl--2.1.8.arm64_big_sur.bottle.tar.gz
🍺  /opt/homebrew/Cellar/sbcl/2.1.8: 1,325 files, 60MB
==> Installing pgloader dependency: buildapp
==> Pouring buildapp--1.5.6_3.arm64_big_sur.bottle.tar.gz
🍺  /opt/homebrew/Cellar/buildapp/1.5.6_3: 4 files, 39.9MB
==> Installing pgloader dependency: m4
==> Pouring m4--1.4.19.arm64_big_sur.bottle.tar.gz
🍺  /opt/homebrew/Cellar/m4/1.4.19: 13 files, 726.3KB
==> Installing pgloader dependency: libtool
==> Pouring libtool--2.4.6_4.arm64_big_sur.bottle.tar.gz
🍺  /opt/homebrew/Cellar/libtool/2.4.6_4: 75 files, 3.7MB
==> Installing pgloader dependency: unixodbc
==> Pouring unixodbc--2.3.9_1.arm64_big_sur.bottle.tar.gz
🍺  /opt/homebrew/Cellar/unixodbc/2.3.9_1: 46 files, 2.3MB
==> Installing pgloader dependency: freetds
==> Pouring freetds--1.3.3.arm64_big_sur.bottle.tar.gz
🍺  /opt/homebrew/Cellar/freetds/1.3.3: 1,262 files, 14.2MB
==> Installing pgloader
==> Patching
==> make
Last 15 lines from /Users/aleksejskrobot/Library/Logs/Homebrew/pgloader/01.make:
                         --load local-projects/pgloader-3.6.2/src/hooks.lisp              \
                         --load-system pgloader                    \
                         --eval '(setf pgloader.params::*version-string* "3.6.2")' \
                         --entry pgloader:main                        \
                         --dynamic-space-size 4096              \
                         --compress-core                         \
                         --output bin/pgloader.tmp
;; loading system "cffi"
;; loading system "cl+ssl"
WARNING: /opt/homebrew/Cellar/sbcl/2.1.8/libexec/bin/sbcl is loading libcrypto in an unsafe way
fatal error encountered in SBCL pid 13689 pthread 0x1002d7d40:
SIGABRT received.

   0:     0x16fd43d18 pc=0x18cccce60
make: *** [bin/pgloader] Error 1

Do not report this issue to Homebrew/brew or Homebrew/core!
made-indrayana commented 3 years ago

I have the same problem with @Wilson13, I'm on Bigs Sur 11.6 but on Intel i7.

jhult commented 2 years ago

Same issue:

csadai commented 2 years ago

Same problem with Big Sur (11.5) and Apple M1 chip

erika-e commented 2 years ago

I don't get the exact same error. I have Rosetta2 enabled on all of my terminal windows and installed brew from a Rosetta2 terminal.

I can run brew install pgloader but trying to run any pgloader commands e.g. pgloader --help results in the libcrytpo error message:

WARNING: /usr/local/bin/pgloader is loading libcrypto in an unsafe way
fatal error encountered in SBCL pid 53056 pthread 0x1006fb600:
SIGABRT received.

Anyone know of a workaround? I am on an M1 chip and Monterey.

chrise86 commented 2 years ago

I don't get the exact same error. I have Rosetta2 enabled on all of my terminal windows and installed brew from a Rosetta2 terminal.

I can run brew install pgloader but trying to run any pgloader commands e.g. pgloader --help results in the libcrytpo error message:

WARNING: /usr/local/bin/pgloader is loading libcrypto in an unsafe way
fatal error encountered in SBCL pid 53056 pthread 0x1006fb600:
SIGABRT received.

Anyone know of a workaround? I am on an M1 chip and Monterey.

Also getting this on a regular Intel based Mac mini.

ScottSmudger commented 2 years ago

Same issue

brew install pgloader brew install --HEAD pgloader and building from source doesnt work

Monterey 12.0.1 / Intel i7

Any workarounds or fixes?

zmitja commented 2 years ago

This fixes the issue:

brew install openssl sudo ln -s /opt/homebrew/Cellar/openssl\@3/3.0.0_1/lib/libcrypto.dylib /usr/local/lib/libcrypto.dylib sudo ln -s /opt/homebrew/Cellar/openssl\@3/3.0.0_1/lib/libssl.dylib /usr/local/lib/libssl.dylib

Found a solution here: https://github.com/cl-plus-ssl/cl-plus-ssl/issues/114

Note, I already had pgloader installed and working, upgrade to Monterey broke it, so I had the issue with running pgloader, not installing it, but I think it's the same issue so this fix should fix the installation problem as well.

tuanalumi commented 2 years ago

I am on an Intel, and I have to use Docker version to overcome this.

yveshield commented 2 years ago

brew install openssl@3 brew link --force openssl@3

then its works again

nbsoftware commented 2 years ago

Still not working here.

Had first to update the links to the current version of openssl:

sudo ln -s /opt/homebrew/Cellar/openssl@3/3.0.1/lib/libcrypto.dylib /usr/local/lib/libcrypto.dylib
sudo ln -s /opt/homebrew/Cellar/openssl@3/3.0.1/lib/libssl.dylib /usr/local/lib/libssl.dylib

as well as

sudo ln -s /opt/homebrew/lib/libsybdb.dylib /usr/lib/libsybdb.dylib

but then it hung shortly after:

==> Patching
==> make
Last 15 lines from /Users/fmeyer/Library/Logs/Homebrew/pgloader/01.make:
;; loading system "cffi"
;; loading system "cl+ssl"
;; loading system "mssql"
;; loading file #P"/private/tmp/pgloader-20211223-23208-t6qbpi/pgloader-bundle-3.6.2/local-projects/pgloader-3.6.2/src/hooks.lisp"
;; loading system "pgloader"
Fatal TYPE-ERROR:
  The values
    (#<FUNCTION (LAMBDA (ESRAP::PRODUCTION)
                  :IN
                  "/private/tmp/pgloader-20211223-23208-t6qbpi/pgloader-bundle-3.6.2/.brew_home/.cache/common-lisp/sbcl-2.1.9-macosx-arm64/private/tmp/pgloader-20211223-23208-t6qbpi/pgloader-bundle-3.6.2/software/cl-abnf-20190521-git/abnf.fasl") {7008B7C14B}>
     NIL NIL)

  are not of type
    (VALUES FUNCTION &OPTIONAL)
make: *** [bin/pgloader] Error 1

This is where I'm stuck. Any ideas?

Meanwhile I'll give Ubuntu arm64 a try...

yveshield commented 2 years ago

@nbsoftware

I'm wondering if you completely uninstalled pgloader and openssl@3 before reinstalling? If not, please try

nbsoftware commented 2 years ago

well I did try to remove the links, brew remove both, brew clean, and reinstalled, still same error :/

nyudenkov commented 2 years ago

Same issue here.

MacOS 12.0.1 / M1

codr36 commented 2 years ago

Same error on Monterey 12.1 (21C52) M1 Air... FYI installing and linking openssl@3 did not work 😕

sawirricardo commented 2 years ago

Any workaround? Also on monterey. No bottle available!

nbsoftware commented 2 years ago

My workaround was to install an ARM Ubuntu VM and get the package running there... not exactly the most trivial setup, but at least it worked

dbouman commented 2 years ago

I'm on macOS Monterey 12.2 M1 and getting exact same issue, installing and linking openssl did not work.

Shantarli commented 2 years ago

Same here with Monterey 12.3 on Intel.

surjithctly commented 2 years ago

Same issue with Mac Monterey with M1 Chip. What's the best solution so far?

ifree-zm commented 2 years ago

I have the same problem with @scrobot ,how to fix it?

smilenannan commented 2 years ago

might be the easiest way around.

ScottSmudger commented 2 years ago

I circumvented it for now by using the docker image, saves the headache

liujchen commented 2 years ago

might be the easiest way around.

Yes, it works

mohammed7aafar commented 2 years ago

You can try to install from source with:

$ brew install --build-from-source pgloader

nbsoftware commented 2 years ago 
➜  ~ brew install --build-from-source pgloader
==> Downloading https://github.com/dimitri/pgloader/releases/download/v3.6.2/pgloader-bundle-3.6.2.tgz
Already downloaded: /Users/fmeyer/Library/Caches/Homebrew/downloads/2a1b2485b3a52a8aa90117254bcc5b1701cd81cc53b0e6f1c16900b1d967f948--pgloader-bundle-3.6.2.tgz
==> Patching
==> make
Last 15 lines from /Users/fmeyer/Library/Logs/Homebrew/pgloader/01.make:
;; loading system "cffi"
;; loading system "cl+ssl"
;; loading system "mssql"
;; loading file #P"/private/tmp/pgloader-20220309-61828-1p1i4fg/pgloader-bundle-3.6.2/local-projects/pgloader-3.6.2/src/hooks.lisp"
;; loading system "pgloader"
Fatal TYPE-ERROR:
  The values
    (#<FUNCTION (LAMBDA (ESRAP::PRODUCTION)
                  :IN
                  "/private/tmp/pgloader-20220309-61828-1p1i4fg/pgloader-bundle-3.6.2/.brew_home/.cache/common-lisp/sbcl-2.1.9-macosx-arm64/private/tmp/pgloader-20220309-61828-1p1i4fg/pgloader-bundle-3.6.2/software/cl-abnf-20190521-git/abnf.fasl") {7008DBC14B}>
     NIL NIL)

  are not of type
    (VALUES FUNCTION &OPTIONAL)
make: *** [bin/pgloader] Error 1
rgossiaux commented 2 years ago

might be the easiest way around.

Just going to bump this comment because this actually works, if you are running into this.

plus- commented 2 years ago

I know it's not ideal but for people using docker I've built the image on my M1 and it's available here

szynkowskij commented 2 years ago

Deleted my comment, but the make did end up working just fine once I found out my work laptop has an enforced proxy server on it 🙃

Building 3.6.3 is definitely the way right now. I think I used brew to install sbcl, clisp openssl, freetds and I'm not sure I even needed those, but part of the problem I think I suffered a lot from was having an x86_64 version of brew running through rosetta as well and I think just incorrect x86 libs were being picked up instead of arm64.

Hopefully we'll see a brew build get updated. I think the main issue with it is that it's still on 3.6.2 and not utilizing newer versions of pgloader lisp dependencies from quicklisp.

adimyth commented 2 years ago

Same issue

brew install pgloader brew install --HEAD pgloader and building from source doesnt work

Monterey 12.0.1 / Intel i7

Any workarounds or fixes?

This doesn't seem to be working for me. I'm on M1 Pro & using version 3.6.3

The make pgloader command is stuck forever & doesn't make any progress

szynkowskij commented 2 years ago

Same issue brew install pgloader brew install --HEAD pgloader and building from source doesnt work Monterey 12.0.1 / Intel i7 Any workarounds or fixes?

This doesn't seem to be working for me. I'm on M1 Pro & using version 3.6.3

The make pgloader command is stuck forever & doesn't make any progress

It making and getting stuck with no feedback seems odd if you just skim through the makefile. Do you have some of the additional requirements? Like sbcl?

adimyth commented 2 years ago

Same issue brew install pgloader brew install --HEAD pgloader and building from source doesnt work Monterey 12.0.1 / Intel i7 Any workarounds or fixes?

This doesn't seem to be working for me. I'm on M1 Pro & using version 3.6.3 The make pgloader command is stuck forever & doesn't make any progress

It making and getting stuck with no feedback seems odd if you just skim through the makefile. Do you have some of the additional requirements? Like sbcl?

Oh thanks! I missed a couple of dependencies, the installation happened correctly! 👍

However, running ./build/bin/pgloader --help is stuck forever

szynkowskij commented 2 years ago

Same issue brew install pgloader brew install --HEAD pgloader and building from source doesnt work Monterey 12.0.1 / Intel i7 Any workarounds or fixes?

This doesn't seem to be working for me. I'm on M1 Pro & using version 3.6.3 The make pgloader command is stuck forever & doesn't make any progress

It making and getting stuck with no feedback seems odd if you just skim through the makefile. Do you have some of the additional requirements? Like sbcl?

Oh thanks! I missed a couple of dependencies, the installation happened correctly! 👍

However, running ./build/bin/pgloader --help is stuck forever

The only time I had some similar problems of it working very odd was when I was able to make it and create the binary, but I had a mixture of x86 and aarch64 required libs. I made sure to remove my x86 version of homebrew and all of its libs that came from a timemachine backup when I upgraded to my M1. That would be my only other guess at your problems without it giving any feedback.

pgloader does hang every once and a while for me on my mac, but that's usually somewhere in the process of loading large files.

adimyth commented 2 years ago

Same issue brew install pgloader brew install --HEAD pgloader and building from source doesnt work Monterey 12.0.1 / Intel i7 Any workarounds or fixes?

This doesn't seem to be working for me. I'm on M1 Pro & using version 3.6.3 The make pgloader command is stuck forever & doesn't make any progress

It making and getting stuck with no feedback seems odd if you just skim through the makefile. Do you have some of the additional requirements? Like sbcl?

Oh thanks! I missed a couple of dependencies, the installation happened correctly! 👍 However, running ./build/bin/pgloader --help is stuck forever

The only time I had some similar problems of it working very odd was when I was able to make it and create the binary, but I had a mixture of x86 and aarch64 required libs. I made sure to remove my x86 version of homebrew and all of its libs that came from a timemachine backup when I upgraded to my M1. That would be my only other guess at your problems without it giving any feedback.

pgloader does hang every once and a while for me on my mac, but that's usually somewhere in the process of loading large files.

It is acting in a non-deterministic fashion. I tried building again from source, however running make pgloader gets stuck forever

panayao commented 1 year ago

pgloader.zip For M1/M2 compiled on macOS Monterey 12.6.3

panayao commented 1 year ago 
LOAD DATABASE
     FROM      mysql://username:password@mysql.db.url/mydb
     INTO postgresql://postgres:password@postgress.db.url/postgres

 WITH include drop, create tables, create indexes, reset sequences,
      workers = 1, concurrency = 1,
      multiple readers per thread, rows per range = 50000

  SET PostgreSQL PARAMETERS
      maintenance_work_mem to '128MB',
      work_mem to '12MB',
      search_path to 'postgres, public, "$user"'

  SET MySQL PARAMETERS
      net_read_timeout  = '120',
      net_write_timeout = '120'

 CAST type bigint when (= precision 20) to bigserial drop typemod,
      type date drop not null drop default using zero-dates-to-null,
      type year to integer;

I was able to get it to work but only if workers = 1 otherwise seems to crash

philharmonie commented 1 year ago

I'm stuck as well on the same topic:

debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {700B600003}>:
  Problem running initialization hook #<FUNCTION OPEN-FOREIGN-LIBS>:
  Unable to load foreign library (LIBCRYPTO).
  Error opening shared object "/usr/local/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.1.1.dylib":
  dlopen(/usr/local/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.1.1.dylib, 0x000A): tried: '/usr/local/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.1.1.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.1.1.dylib' (no such file), '/usr/local/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.1.1.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')).

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [CONTINUE] Skip this initialization hook.
  1: [ABORT   ] Exit from the current thread.

(SB-INT:CALL-HOOKS "initialization" (#<FUNCTION OPEN-FOREIGN-LIBS>) :ON-ERROR :ERROR)

I tried to compile it myself (without success) and I tried to use the build provieded by @panayao without success.

I am using openssl@3 and I don't know why it is trying so use 1.1 and how to work around that.

I tied 

brew unlink openssl@3
Unlinking /opt/homebrew/Cellar/openssl@3/3.1.1_1... 5548 symlinks removed.
❯ brew link openssl@1.1
Linking /opt/homebrew/Cellar/openssl@1.1/1.1.1u... 3999 symlinks created.
❯ echo 'export PATH="/opt/homebrew/opt/openssl@1.1/bin:$PATH"' >> ~/.zshrc