search

dimitri / pgloader

Migrate to PostgreSQL in a single command!
http://pgloader.io
Other
5.33k stars 541 forks source link

windows installation #652

Closed krisgoks closed 3 years ago

krisgoks commented 6 years ago

Has anyone successfully installed pgloader on windows ? If yes would you please provide step by step instructions

dimitri commented 6 years ago

It seems to be problematic to do so. Other issues are open with the Windows support label, that you can read to meet with windows users.

I don't know windows for not having used it ever, but I could see about building a pgloader.exe with some sponsoring for the time it will take. See about buying a pgloader Moral License at http://pgloader.io/pgloader-moral-license.html (the « Custom Development » one, or if other windows users also chime in a bunch of Partner level ones) so that I can see about that. Thanks.

philCryoport commented 6 years ago

FYI to anyone who is trying to build in Windows 10, here's how far I've gotten:

  1. Download and install msys2 binary for your Windows architecture x86 (32-bit) or x86_64 (64-bit) from: https://www.msys2.org/
  2. Verify that you have a recent version of make by typing make --version and hitting enter
    1. I'm using: 
      $ make --version
GNU Make 4.2.1
Built for x86_64-pc-msys
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
  3. Download and install sbcl binary for your Windows architecture from: http://www.sbcl.org/platform-table.html
  4. Open msys2 console for your Windows architecture (if you have 64-bit Windows, it'll install both 32-bit and 64-bit consoles if memory serves me)
  5. Edit your ~/.bashrc with vim to add the installed SBCL binaries to the bash path -- basically:
    1. scroll to the very last line of the file
    2. hit the i key to enter insert mode
    3. scroll to the end of the last line
    4. hit the enter key
    5. for 64-bit Windows with SBCL 1.4.2, paste this: export PATH=$PATH:/c/progra~1/steelb~1/1.4.2
      1. adjust accordingly for 32-bit Windows and other SBCL versions -- dir /x in the normal Windows Command Prompt is your friend...
    6. hit the Esc key to exit insert mode
    7. hit the : key to enter a command
    8. hit the x key to save and exit
  6. verify that sbcl is working by closing out of the msys2 console, re-opening it, and then running sbcl --version: 
    $ sbcl --version
SBCL 1.4.2
  7. Download the latest bundle release of pgloader from: https://github.com/dimitri/pgloader/releases
  8. Unpack it (I use 7-Zip for Windows: https://www.7-zip.org/download.html )
  9. cd to the directory
  10. make and enter
philCryoport commented 6 years ago

My build fails at:

;; loading system "buildapp"
bin/buildapp.exe      --logfile /tmp/pgloader-bundle-build.log     \
                         --require sb-posix --require sb-bsd-sockets --require sb-rotate-byte                             \
                         --sbcl sbcl                                 \
                         --asdf-tree .                                \
                         --load-system pgloader                    \
                         --eval '(setf pgloader.params::*version-string* "3.4.1")' \
                         --load local-projects/pgloader-3.4.1/src/hooks.lisp              \
                         --entry pgloader:main                        \
                         --dynamic-space-size 1024                            \
                                                  \
                         --output bin/pgloader.tmp
;; loading system "pgloader"
Fatal CIRCULAR-DEPENDENCY:
  Circular dependency:
     ((#<ASDF/LISP-ACTION:LOAD-OP > . #<ASDF/SYSTEM:SYSTEM "simple-date">)
      (#<ASDF/LISP-ACTION:LOAD-OP >
       . #<ASDF/SYSTEM:SYSTEM "simple-date-postgres-glue">)
      (#<ASDF/LISP-ACTION:LOAD-OP >
       . #<ASDF/COMPONENT:MODULE "simple-date-postgres-glue" "simple-date">)
      (#<ASDF/LISP-ACTION:LOAD-OP >
       . #<ASDF/LISP-ACTION:CL-SOURCE-FILE "simple-date-postgres-glue" "simple-date" "cl-postgres-glue">)
      (#<ASDF/LISP-ACTION:PREPARE-OP >
       . #<ASDF/LISP-ACTION:CL-SOURCE-FILE "simple-date-postgres-glue" "simple-date" "cl-postgres-glue">)
      (#<ASDF/LISP-ACTION:PREPARE-OP >
       . #<ASDF/COMPONENT:MODULE "simple-date-postgres-glue" "simple-date">)
      (#<ASDF/LISP-ACTION:PREPARE-OP >
       . #<ASDF/SYSTEM:SYSTEM "simple-date-postgres-glue">))
make: *** [Makefile:47: bin/pgloader] Error 1

Full log starting with make: foo2.txt

philCryoport commented 6 years ago

BTW make clean isn't working with make 4.2.1 on msys2 64-bit:

 ~/downloads/pgloader-bundle-3.4.1
$ make clean
make: *** No rule to make target 'clean'.  Stop.
philCryoport commented 6 years ago

In case it's of any use, here's /tmp/pgloader-bundle-build.log:

pgloader-bundle-build.log

dimitri commented 6 years ago

The problem you have with ASDF circular dependency isn't limited to windows in any ways, to my knowledge. You need to use more recent version of pgloader dependencies. Normally make clean all will trash your local copy for Quicklisp packages and download them again, solving the problem you have here.

philCryoport commented 6 years ago

Hi Dimitri,

make clean all isn't working with make 4.2.1 on msys2 64-bit:

 ~/downloads/pgloader-bundle-3.4.1
$ make clean all
make: *** No rule to make target 'clean'.  Stop.

Help?

klyhne commented 6 years ago

Hey @philCryoport I'm trying to follow the steps you posted here. Have you managed to make a succesful build on windows?

I am getting the following error:

; caught ERROR:
;   READ error during COMPILE-FILE:
;
;     Symbol "LOCAL-SOCKET" not found in the SB-BSD-SOCKETS package.
;
;       Line: 77, Column: -1, File-Position: 3973
;
;       Stream: #<SB-INT:FORM-TRACKING-STREAM for "file C:\\msys64\\home\\KristianLyhne\\pgloader\\build\\quicklisp\\local-projects\\qmynd\\src\\api.lisp" {1007C98B83}>

debugger invoked on a UIOP/LISP-BUILD:COMPILE-FILE-ERROR in thread
#<THREAD "main thread" RUNNING {10027900C3}>:
  COMPILE-FILE-ERROR while compiling #<CL-SOURCE-FILE "qmynd" "src" "api">

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY                        ] Retry
                                     compiling #<CL-SOURCE-FILE "qmynd" "src" "api">.
  1: [ACCEPT                       ] Continue, treating
                                     compiling #<CL-SOURCE-FILE "qmynd" "src" "api">
                                     as having been successful.
  2:                                 Retry ASDF operation.
  3: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the
                                     configuration.
  4:                                 Retry ASDF operation.
  5:                                 Retry ASDF operation after resetting the
                                     configuration.
  6: [ABORT                        ] Give up on "pgloader"
  7: [CONTINUE                     ] Ignore runtime option --eval "(ql:quickload \"pgloader\")".
  8:                                 Skip rest of --eval and --load options.
  9:                                 Skip to toplevel READ/EVAL/PRINT loop.
 10: [EXIT                         ] Exit SBCL (calling #'EXIT, killing the process).

(UIOP/LISP-BUILD:CHECK-LISP-COMPILE-RESULTS NIL T T "~/asdf-action::format-action/" ((#<ASDF/LISP-ACTION:COMPILE-OP > . #<ASDF/LISP-ACTION:CL-SOURCE-FILE "qmynd" "src" "api">)))
   error finding frame source: Bogus form-number: the source file has probably
                               changed too much to cope with.
   source: NIL

I tried with make clean all, which seems to work for me with make 4.2.1 on msys2 64-bit. It gives me the same error.

Help is highly appreciated.

Best,

Kristian

dimitri commented 6 years ago

@klyhne which version of SBCL are you using?

philCryoport commented 6 years ago

Hey @philCryoport I'm trying to follow the steps you posted here. Have you managed to make a succesful build on windows?

Apologies it was so long ago I don't remember. I think in the end I just used Linux.

jreisam commented 6 years ago

Hey guys, trying to "make" it work on windows too, but considering this comment session I'm now choosing linux to do so. thank you anyway

Godwottery commented 5 years ago

I am having the same type of problems

..................................................
[package qmynd]...................................
[package qmynd-impl]..............................
..;
; caught ERROR:
;   READ error during COMPILE-FILE:
;
;     Symbol "LOCAL-SOCKET" not found in the SB-BSD-SOCKETS package.
;
;       Line: 77, Column: -1, File-Position: 3973
;
;       Stream: #<SB-INT:FORM-TRACKING-STREAM for "file C:\\Data\\software-misc\\pgloader-master\\build\\quicklisp\\local-projects\\qmynd\\src\\api.lisp" {1009F031D3}>

debugger invoked on a UIOP/LISP-BUILD:COMPILE-FILE-ERROR in thread
#<THREAD "main thread" RUNNING {10027900C3}>:
  COMPILE-FILE-ERROR while compiling #<CL-SOURCE-FILE "qmynd" "src" "api">

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY                        ] Retry
                                     compiling #<CL-SOURCE-FILE "qmynd" "src" "api">.
  1: [ACCEPT                       ] Continue, treating
                                     compiling #<CL-SOURCE-FILE "qmynd" "src" "api">
                                     as having been successful.
  2:                                 Retry ASDF operation.
  3: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the
                                     configuration.
  4:                                 Retry ASDF operation.
  5:                                 Retry ASDF operation after resetting the
                                     configuration.
  6: [ABORT                        ] Give up on "pgloader"
  7: [CONTINUE                     ] Ignore runtime option --eval "(ql:quickload \"pgloader\")".
  8:                                 Skip rest of --eval and --load options.
  9:                                 Skip to toplevel READ/EVAL/PRINT loop.
 10: [EXIT                         ] Exit SBCL (calling #'EXIT, killing the process).

(UIOP/LISP-BUILD:CHECK-LISP-COMPILE-RESULTS NIL T T "~/asdf-action::format-action/" ((#<ASDF/LISP-ACTION:COMPILE-OP > . #<ASDF/LISP-ACTION:CL-SOURCE-FILE "qmynd" "src" "api">)))
   error finding frame source: Bogus form-number: the source file has probably
                               changed too much to cope with.
   source: NIL
0]

sbcl - 1.4.2

Godwottery commented 5 years ago

I also run into the following problem with SSLeay32.dll 

$ make
sbcl --noinform --no-sysinit --no-userinit --load build/quicklisp/setup.lisp                   \
             --eval '(push "/c/data/software-misc/pgloader-master/" ql:*local-project-directories*)' \
             --eval '(ql:quickload "pgloader")'                       \
             --eval '(quit)'
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
.....
;;; Checking for wide character support... WARNING: Lisp implementation doesn't use UTF-16, but accepts surrogate code points.
 yes, using code points.
.
;;; Checking for wide character support... WARNING: Lisp implementation doesn't use UTF-16, but accepts surrogate code points.
 yes, using code points.
;;; Building Closure with CHARACTER RUNES
..
debugger invoked on a CFFI:LOAD-FOREIGN-LIBRARY-ERROR in thread
#<THREAD "main thread" RUNNING {10027900C3}>:
  Unable to load any of the alternatives:
   ("libssl32.dll" "ssleay32.dll")

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY                        ] Try loading the foreign library again.
  1: [USE-VALUE                    ] Use another library instead.
  2: [TRY-RECOMPILING              ] Recompile reload and try loading it again
  3: [RETRY                        ] Retry
                                     loading FASL for #<CL-SOURCE-FILE "cl+ssl" "src" "reload">.
  4: [ACCEPT                       ] Continue, treating
                                     loading FASL for #<CL-SOURCE-FILE "cl+ssl" "src" "reload">
                                     as having been successful.
  5:                                 Retry ASDF operation.
  6: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the
                                     configuration.
  7:                                 Retry ASDF operation.
  8:                                 Retry ASDF operation after resetting the
                                     configuration.
  9: [ABORT                        ] Give up on "pgloader"
 10: [CONTINUE                     ] Ignore runtime option --eval "(ql:quickload \"pgloader\")".
 11:                                 Skip rest of --eval and --load options.
 12:                                 Skip to toplevel READ/EVAL/PRINT loop.
 13: [EXIT                         ] Exit SBCL (calling #'EXIT, killing the process).

(CFFI::FL-ERROR "Unable to load any of the alternatives:~%   ~S" ("libssl32.dll" "ssleay32.dll"))
   source: (ERROR 'LOAD-FOREIGN-LIBRARY-ERROR :FORMAT-CONTROL CONTROL
                  :FORMAT-ARGUMENTS ARGUMENTS)
0]

The above was when building from master, the following when building from a bundle: 

/c/users/$SUERNAME/Downloads/pgloader-bundle-3.5.2
$ make pgloader
bin/buildapp.exe      --logfile /tmp/pgloader-bundle-build.log     \
                         --require sb-posix --require sb-bsd-sockets --require sb-rotate-byte                             \
                         --sbcl sbcl                                 \
                         --asdf-tree .                                \
                         --load-system pgloader                    \
                         --eval '(setf pgloader.params::*version-string* "3.5.2")' \
                         --load local-projects/pgloader-3.5.2/src/hooks.lisp              \
                         --entry pgloader:main                        \
                         --dynamic-space-size 1024                            \
                                                  \
                         --output bin/pgloader.tmp
;; loading system "pgloader"
Fatal LOAD-FOREIGN-LIBRARY-ERROR:
  Unable to load any of the alternatives:
   ("libssl32.dll" "ssleay32.dll")
Fatal LOAD-FOREIGN-LIBRARY-ERROR:
  Unable to load any of the alternatives:
   ("libssl32.dll" "ssleay32.dll")
make: *** [Makefile:47: bin/pgloader] Error 1
Godwottery commented 5 years ago

The more I look into the issues above, it seems that there is something amiss with the libraries. Probably the issue is that I am trying to build on a 64-bit machine. It seems that this is upstream rather than with pgloader. See also https://github.com/cl-plus-ssl/cl-plus-ssl/issues/53

Edit: I installed Shining Light OpenSSL for Win64 and fixed my path to include this one. That solved the SSL include problem. Now working on the local-socket.

MRigal commented 5 years ago

Hi @dimitri First thanks for the work done on the package and also for trying to support more or less Windows. After many hours, I'm also blocked at the same point as @Godwottery and @philCryoport .

Here again the steps and the versions. On Windows 2012 R2 64 bits:

  1. Install msys 64 bits
  2. Get ready for DEV on msys, using pacman and setting environment variables, similar to: https://github.com/orlp/dev-on-windows/wiki/Installing-GCC--&-MSYS2
  3. Get latest SBCL (1.4.14 as compiled)
  4. Add libssl.dll from one of https://wiki.openssl.org/index.php/Binaries

All works, but I'm stucked while compiling pgloader (many versions, including v3.6.0 pre-release) with the error:

Fatal COMPILE-FILE-ERROR:
  COMPILE-FILE-ERROR while compiling #<CL-SOURCE-FILE "qmynd" "src" "api">

And looking in the log I get the following:

; caught ERROR:
;   READ error during COMPILE-FILE:
;
;     Symbol "LOCAL-SOCKET" not found in the SB-BSD-SOCKETS package.
;
;       Line: 77, Column: -1, File-Position: 3973
;
;       Stream: #<SB-INT:FORM-TRACKING-STREAM for "file C:\\pgloader-bundle-3.6.0\\local-projects\\qmynd\\src\\api.lisp" {10036AD133}>

The problem is in how qymnd is accesing the sockets. And the root of the error might be in this commit introduced in sbcl v 1.4.2, which changes the behaviour for windows platforms only: https://github.com/sbcl/sbcl/commit/6eb5fcbeb890182d3f03c02edb430ec52d72bcfc

I've tried to find olders sbcl libraries without success and I've tried to compile by myself a patched version of the 1.4.15 source also without success.

I hope this helps in case you want to dig deeper to fix the issue.

Best, Matt

dimitri commented 5 years ago

Hi @MRigal ; thanks for the status update! As I'm not using windows, progress in compatibility with this OS is very slow, at best, i.e. when it happens. Can you please try compiling with Clozure-CL (as per the INSTALL file), which for sure will not suffer from the problem you mention with SB-BSD-SOCKETS, the SBCL specific implementation of sockets.

MRigal commented 5 years ago

Hi @dimitri I've triedto disable all sbcl specific settings, I had to create a symlink for the name, but I'm also stucked here with 

Fatal SIMPLE-ERROR:
  Compilation failed: In INITIATE-CONNECTION: Undeclared free variable *UNIX-SOCKET-DIR* in C:/pgloader-bundle-3.6.0/software/postmodern-20181018-git/cl-postgres/public.lisp

Unfortunately, I have only very little knowledge about lisp

w93163red commented 5 years ago

Just for another update, I got stuck at this point:

[package mssql]..Help! 11 nested errors. SB-KERNEL:*MAXIMUM-ERROR-DEPTH* exceeded.
Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {10012E0613}>
0: (SB-KERNEL::%SIGNAL #<TYPE-ERROR expected-type: LIST datum: 0>)
1: (ERROR TYPE-ERROR :DATUM 0 :EXPECTED-TYPE LIST :CONTEXT NIL)
2: (SB-KERNEL:INTERNAL-ERROR #.(SB-SYS:INT-SAP #X009A32E0) #<unused argument>)
3: ("foreign function: #x43101B")
4: ("foreign function: #x403791")
Help! ERROR-ERROR is 2 levels deep. Will try to reset the IO streams by calling STREAM-COLD-INIT-OR-RESET.
Help! 11 nested errors. SB-KERNEL:*MAXIMUM-ERROR-DEPTH* exceeded.
Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {10012E0613}>
0: (SB-KERNEL::MAP-RESTARTS #<CLOSURE (LAMBDA (RESTART) :IN COMPUTE-RESTARTS) {10036CBB3B}> #<SIMPLE-ERROR "Maximum error nesting depth exceeded" {10036CBA13}> T)
1: (COMPUTE-RESTARTS #<SIMPLE-ERROR "Maximum error nesting depth exceeded" {10036CBA13}>)
2: (SB-DEBUG::%INVOKE-DEBUGGER #<SIMPLE-ERROR "Maximum error nesting depth exceeded" {10036CBA13}>)
3: ((FLET "LAMBDA0" :IN "SYS:SRC;CODE;DEBUG.LISP"))
4: (SB-IMPL::CALL-WITH-SANE-IO-SYNTAX #<CLOSURE (FLET "LAMBDA0" :IN "SYS:SRC;CODE;DEBUG.LISP") {9A1D3B}>)
5: ((FLET "THUNK" :IN SB-DEBUG::FUNCALL-WITH-DEBUG-IO-SYNTAX))
6: (SB-IMPL::%WITH-STANDARD-IO-SYNTAX #<CLOSURE (FLET "THUNK" :IN SB-DEBUG::FUNCALL-WITH-DEBUG-IO-SYNTAX) {9A1E0B}>)
7: (SB-DEBUG::FUNCALL-WITH-DEBUG-IO-SYNTAX #<FUNCTION SB-DEBUG::%INVOKE-DEBUGGER> #<SIMPLE-ERROR "Maximum error nesting depth exceeded" {10036CBA13}>)
8: (INVOKE-DEBUGGER #<SIMPLE-ERROR "Maximum error nesting depth exceeded" {10036CBA13}>)
9: ((FLET SB-IMPL::TRY-TO-INVOKE-DEBUGGER :IN SB-IMPL::ERROR-ERROR))
10: ((FLET "THUNK" :IN SB-IMPL::ERROR-ERROR))
11: (SB-IMPL::%WITH-STANDARD-IO-SYNTAX #<CLOSURE (FLET "THUNK" :IN SB-IMPL::ERROR-ERROR) {9A205B}>)
12: (SB-IMPL::ERROR-ERROR "Help! " 11 " nested errors. " "SB-KERNEL:*MAXIMUM-ERROR-DEPTH* exceeded.")
13: (SB-KERNEL:INTERNAL-ERROR #.(SB-SYS:INT-SAP #X009A2420) #<unused argument>)
14: ("foreign function: #x43101B")
15: ("foreign function: #x403791")
Help! ERROR-ERROR is 3 levels deep. Will try to THROW this thread to the toplevel.

debugger invoked on a SB-SYS:MEMORY-FAULT-ERROR in thread
#<THREAD "main thread" RUNNING {10012E0613}>:
  Unhandled memory fault at #xFFFFFFFFFFFFFFFF.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [ABORT] Exit from the current thread.

("bogus stack frame")
0]

debugger invoked on a SB-INT:SIMPLE-CONTROL-ERROR in thread
#<THREAD "main thread" RUNNING {10012E0613}>:
  attempt to THROW to a tag that does not exist: SB-IMPL::%END-OF-THE-WORLD

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [ABORT] Exit from the current thread.

(THROW)
0]
Unable to recommit addr 000000000308FFF8 eip 000000002000090C
Backtrace: BT (thread 00000000034A2000)
[#01]: ebp = 000000000099EBF0, ret = 0000000001346E80
[#02]: ebp = 000000002000090C, ret = 2914894F168B4C07
make: *** [Makefile:103: build/libs.stamp] Error 1

The previous steps I have taken:

w93163red commented 5 years ago

Another quick update on building from the bundle:

$ make pgloader
mkdir -p bin
sbcl --noinform --no-sysinit --no-userinit --load bundle.lisp                           \
             --eval '(asdf:load-system :buildapp)'                    \
             --eval '(buildapp:build-buildapp "bin/buildapp.exe")'                  \
             --eval '(quit)'
WARNING:
   Reference to deprecated function (SB-DEBUG:BACKTRACE-AS-LIST) from COMMAND-LINE-DEBUGGER
;; loading system "buildapp"
bin/buildapp.exe      --logfile /tmp/pgloader-bundle-build.log     \
                         --require sb-posix --require sb-bsd-sockets --require sb-rotate-byte                             \
                         --sbcl sbcl                                 \
                         --asdf-tree .                                \
                         --load-system cffi                           \
                         --load-system cl+ssl                         \
                         --load local-projects/pgloader-3.6.1/src/hooks.lisp              \
                         --load-system pgloader                    \
                         --eval '(setf pgloader.params::*version-string* "3.6.1")' \
                         --entry pgloader:main                        \
                         --dynamic-space-size 1024                            \
                                                  \
                         --output bin/pgloader.tmp
;; loading system "cffi"
;; loading system "cl+ssl"
;; loading file #P"D:/pgloader-bundle-3.6.1/local-projects/pgloader-3.6.1/src/hooks.lisp"
;; loading system "pgloader"
Fatal COMPILE-FILE-ERROR:
  COMPILE-FILE-ERROR while compiling #<CL-SOURCE-FILE "qmynd" "src" "api">
make: *** [Makefile:47: bin/pgloader] Error 1

The SBCL version is 1.4.16.

w93163red commented 5 years ago

Another test on CCL: I get the sybdb.dll file from here: https://github.com/ramiro/freetds/releases then use CCL to build the pgloader. However, it hangs on here:


$ make CL=D:\\ccl\\wx86cl64.exe pgloader
D:\ccl\wx86cl64.exe --no-init --load build/quicklisp/setup.lisp                   \
             --eval '(push :pgloader-image *features*)'               \
             --eval '(setf *print-circle* t *print-pretty* t)'        \
             --eval '(ql:quickload "pgloader")'                       \
             --eval '(push "/cygdrive/d/pgloader-1/" ql:*local-project-directories*)' \
             --eval '(ql:quickload "pgloader")'                       \
             --eval '(quit)'
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
...
;;; Checking for wide character support... yes, using code points.
;;; Checking for wide character support... yes, using code points.
;;; Building Closure with CHARACTER RUNES

To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"

wlingxiang@TYCW16177 /cygdrive/d/pgloader-1
$ make CL=D:\\ccl\\wx86cl64.exe pgloader
mkdir -p build/bin
D:\ccl\wx86cl64.exe --no-init --load build/quicklisp/setup.lisp               \
             --eval '(ql:quickload "buildapp")'                   \
             --eval '(buildapp:build-buildapp "build/bin/buildapp.ccl.exe")'              \
             --eval '(quit)'
To load "buildapp":
  Install 1 Quicklisp release:
    buildapp
; Fetching #<URL "http://beta.quicklisp.org/archive/buildapp/2015-12-18/buildapp-1.5.6.tgz">
; 16.00KB
==================================================
16,389 bytes in 0.01 seconds (2667.48KB/sec)
; Loading "buildapp"
[package buildapp]...........
> Error: Error #<SILENT-EXIT-ERROR #x2100A1343D>
> While executing: (:INTERNAL MAIN), in process listener(1).
> Type :GO to continue, :POP to abort, :R for a list of available restarts.
> If continued: Skip evaluation of (buildapp:build-buildapp "build/bin/buildapp.ccl.exe")
> Type :? for other options.
1 >
w93163red commented 5 years ago

All, I might figure out a way to compile pgloader on windows. For sybdb.dll drive, I use dblib.dll and rename it to sybdb.dll. Here is the link: ftp://ftp.freepascal.org/fpc/contrib/windows/dblib_0.95.zip For lisp, I use CCL. Somehow, sbcl did not work on my computer.

Another thing is when you compile pgloader on windows, do not use the makefile... The environment I used is cygwin64. I will try to reproduce these steps on cmd again. If it works too, I will share the binary file so that other one did not need to go through this painful process again...

Go to the pgloader folder, then execute: ../ccl/wx86cl64.exe --no-init --load ./src/save.lisp https://github.com/dimitri/pgloader/issues/671 If you have installed all your necessary dependencies, you will see the binary file: pgloader.

However, I still encountered this problem: $ ./pgloader

Error: Undefined foreign library: CL+SSL::LIBEAY32 While executing: CFFI::GET-FOREIGN-LIBRARY, in process Initial(0). Type :GO to continue, :POP to abort, :R for a list of available restarts. If continued: Skip (possibly crucial) startup function OPEN-FOREIGN-LIBS. Type :? for other options.

w93163red commented 5 years ago

Here is the version I have compiled. It is compiled using CCL 1.11.5. pgloader.zip For running it, make sure you have the openssl lib on your windows PC. https://slproweb.com/products/Win32OpenSSL.html I have also tested the SBCL one. But it is not stable as CCL one as I tested so far.

The screenshot: image

The steps I have using:

  1. Install cygwin64, when asking installing packages, choose the package I have selected: image

Make sure the verison is the same as mine.

  1. Download CCL. https://ccl.clozure.com/. And compress the zip in somewhere.
  2. Download dblib_0.95.zip. Copy all the files to cygwin64/bin. Duplicate dblib.dll and rename it to sybdb.dll
  3. Clone the pgloader repo git clone https://github.com/dimitri/pgloader.git
  4. Open cygwin64. Locate to the parent folder of pgloader. Without using the Make, type in the following command instead: [your ccl.exe location] --no-init --load ./src/save.lisp For example : ../ccl/wx86cl64.exe --no-init --load ./src/save.lisp The above command will generate a x64 version pgloader.
  5. Wait for few minutes
  6. In build/bin, there will be a pgloader. Rename it to pgloader.exe
  7. run pgloader.exe, when it popped out the error, type :GO

dblib_0.95.zip

samuei commented 4 years ago

If somebody can verify and duplicate @w93163red's work, should we consider reopening #385 to automate binary releases?

samuei commented 4 years ago

Windows users can also install the Linux Subsystem for Windows and use apt to install the latest release version.

bburns commented 4 years ago

Thanks for the tip, @samuei - posted here https://stackoverflow.com/questions/38750898/build-or-install-pg-loader-on-windows/59370143

takato1314 commented 3 years ago

Hi i tried to build the source using make command following instructions from this way but i ran into this error.

  0: [CONTINUE                     ] Return from SB-UNIX:SIGINT.
;   (FORMATTER "~
;   BEGIN
;      ~{NEW.~a = now();~^~%   ~}
;      RETURN NEW;
;   END;").
;   Use *BREAK-ON-SIGNALS* to intercept.
;
;    error in FORMAT: Unknown directive (character: Return)
;     ~
;   BEGIN
;      ~{NEW.~a = now();~^~%   ~}
;      RETURN NEW;
;   END;
;      ^

debugger invoked on a UIOP/LISP-BUILD:COMPILE-FILE-ERROR in thread
#<THREAD "main thread" RUNNING {1000560083}>:
  COMPILE-FILE-ERROR while
  compiling #<CL-SOURCE-FILE "pgloader" "src" "pgsql" "pgsql-trigger">

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY                        ] Retry
                                     compiling #<CL-SOURCE-FILE "pgloader" "src" "pgsql" "pgsql-trigger">.
  1: [ACCEPT                       ] Continue, treating
                                     compiling #<CL-SOURCE-FILE "pgloader" "src" "pgsql" "pgsql-trigger">
                                     as having been successful.
  2:                                 Retry ASDF operation.
  3: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the
                                     configuration.
  4:                                 Retry ASDF operation.
  5:                                 Retry ASDF operation after resetting the
                                     configuration.
  6: [ABORT                        ] Give up on "pgloader"
  7: [REGISTER-LOCAL-PROJECTS      ] Register local projects and try again.
  8: [CONTINUE                     ] Ignore runtime option --eval "(ql:quickload \"pgloader\")".
  9:                                 Skip rest of --eval and --load options.
 10:                                 Skip to toplevel READ/EVAL/PRINT loop.
 11: [EXIT                         ] Exit SBCL (calling #'EXIT, killing the process).

(UIOP/LISP-BUILD:CHECK-LISP-COMPILE-RESULTS NIL T T "~/asdf-action::format-action/" ((#<ASDF/LISP-ACTION:COMPILE-OP > . #<ASDF/LISP-ACTION:CL-SOURCE-FILE "pgloader" "src" "pgsql" "pgsql-trigger">)))
   error finding frame source: Bogus form-number: the source file has probably
                               changed too much to cope with.
   source: NIL

Any idea?

phoe commented 3 years ago

Convert all source files to Unix line endings and try again - it should resolve this.

takato1314 commented 3 years ago

Thank you @phoe , I forgot to change my git settings to checkout as unix line endings.

phoe commented 3 years ago

This is a known issue between Windows and Lisp code in general. format's Tilde Newline directive interacts with Windows line endings, since on Windows it becomes Tilde Linefeed rather than Tilde Newline - and the former is undefined. I don't think that any Lisp implementations provide workarounds for it, since it would change other Lisp semantics.

So, basically - Lisp code on Windows needs to have Unix line endings on it, or you can't guarantee that it will compile.

jouvin commented 2 years ago

Hi,

I'm trying to build pgloader on Windows 10 but I don't succeed weither I use SBCL or CCL. With SBCL, the error seems to happen when building the pgloader binary but I don't have any clue why. The error is:

;; loading system "buildapp"
mkdir -p build/bin
build/bin/buildapp.sbcl.exe      --logfile /tmp/build.log                \
                         --require sb-posix --require sb-bsd-sockets --require sb-rotate-byte                        \
                         --sbcl sbcl                            \
                         --asdf-path .                           \
                         --asdf-tree build/quicklisp/local-projects     \
                         --manifest-file build/manifest.ql             \
                         --asdf-tree build/quicklisp/dists              \
                         --asdf-path .                           \
                         --load-system cffi                      \
                         --load-system cl+ssl                    \
                         --load-system mssql                     \
                         --load src/hooks.lisp                   \
                         --load-system pgloader               \
                         --entry pgloader:main                   \
                         --dynamic-space-size 1024                       \
                                             \
                         --output build/bin/pgloader.exe.tmp
;; loading system "cffi"
;; loading system "cl+ssl"
;; loading system "mssql"
;; loading file #P"C:/Users/jouvin/PycharmProjects/Postgresql/pgloader/src/hooks.lisp"
;; loading system "pgloader"
Fatal COMPILE-FILE-ERROR:
  COMPILE-FILE-ERROR while
  compiling #<CL-SOURCE-FILE "pgloader" "src" "pgsql" "pgsql-trigger">
make: *** [Makefile:138: build/bin/pgloader.exe] Error 1

(no mention of the "bogus form numer" like in https://github.com/takato1314, I don't think I have a line ending problem).

With CCL, the error is:

[package qmynd-impl]..............................
..............
> Error: No such file or directory : #P"C:/Users/jouvin/AppData/Local/cache/common-lisp/ccl-1.12-f98-win-x64/C/Users/jouvin/PycharmProjects/Postgresql/pgloader/build/quicklisp/dists/quicklisp/software/qmynd-20190710-git/src/mysql-protocol/text-protocol/command-initialize-database-tmpJ1D66IEV.wx64fsl"
> While executing: CCL::%CREATE-FILE, in process listener(1).

Thanks in advance for any help!

Anat56 commented 2 years ago

Thanks for the instruction I had an error that there is no sybdb.dll library, I downloaded the latest one: https://sourceforge.net/projects/zeoslib/files/3rd%20party/FreeTDS/

FYI to anyone who is trying to build in Windows 10, here's how far I've gotten:

  1. Download and install msys2 binary for your Windows architecture x86 (32-bit) or x86_64 (64-bit) from: https://www.msys2.org/

  2. Verify that you have a recent version of make by typing make --version and hitting enter

    1. I'm using:
    $ make --version
GNU Make 4.2.1
Built for x86_64-pc-msys
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
  3. Download and install sbcl binary for your Windows architecture from: http://www.sbcl.org/platform-table.html
  4. Open msys2 console for your Windows architecture (if you have 64-bit Windows, it'll install both 32-bit and 64-bit consoles if memory serves me)

  5. Edit your ~/.bashrc with vim to add the installed SBCL binaries to the bash path -- basically:

    1. scroll to the very last line of the file
    2. hit the i key to enter insert mode
    3. scroll to the end of the last line
    4. hit the enter key

    5. for 64-bit Windows with SBCL 1.4.2, paste this: export PATH=$PATH:/c/progra~1/steelb~1/1.4.2

      1. adjust accordingly for 32-bit Windows and other SBCL versions -- dir /x in the normal Windows Command Prompt is your friend...
    6. hit the Esc key to exit insert mode
    7. hit the : key to enter a command
    8. hit the x key to save and exit
  6. verify that sbcl is working by closing out of the msys2 console, re-opening it, and then running sbcl --version: 
    $ sbcl --version
SBCL 1.4.2
  7. Download the latest bundle release of pgloader from: https://github.com/dimitri/pgloader/releases
  8. Unpack it (I use 7-Zip for Windows: https://www.7-zip.org/download.html )
  9. cd to the directory
  10. make and enter
Anat56 commented 2 years ago

result:

URL "/mysql/list-all-columns.sql" not found!

Zastai commented 1 year ago

For reference: the build script I'm using (to avoid having to use a make from WSL/MSYS/...).

This expects to be in a directory containing:

It also works around an issue with two source files that will fail to compile (during the final buildapp step) when they have CRLF line endings.

param (
  [string] $Version = '3.6.9',
  # pfloader defaults to 16GB, except on windows where it uses 1GB. We default to 8GB as a compromise.
  [int] $ImageMemorySize = 8192,
  [switch] $NoGit
)

$ErrorActionPreference = 'Stop'

$dep_dir = (Resolve-Path dependencies)
$sbcl_home = (Resolve-Path sbcl-2.3.1)
$sbcl = (Resolve-Path (Join-Path $sbcl_home sbcl.exe))

# FIXME: This naming is intended to match what GitHub supplies as top-level folder in its sources.zip; not sure if that's safe.
$src_dir = "pgloader-$Version"

# 0. Get the source code for the requested version
if (-not (Test-Path -PathType Container $src_dir)) {
  if ($NoGit) {
    Invoke-WebRequest https://github.com/dimitri/pgloader/archive/refs/tags/v$Version.zip -OutFile pgloader-sources.zip
    Expand-Archive -Path pgloader-sources.zip -DestinationPath . -Force
    # Without a wait, Remove-Item will fail with "file in use"
    Start-Sleep -Seconds 3
    Remove-Item pgloader-sources.zip
  }
  else {
    git clone --depth 1 --branch v$Version https://github.com/dimitri/pgloader $src_dir
  }
  # Some files need to have LF line endings
  'src/pgsql/pgsql-trigger.lisp', 'src/sources/csv/csv.lisp' | ForEach-Object {
    $file = (Resolve-Path (Join-Path $src_dir $_))
    $contents = (Get-Content -Raw $file) -replace "`r`n","`n"
    Set-Content -Path $file -Value $contents -NoNewline
  }
}

$bin_dir = 'binaries'

if (-not (Test-Path -PathType Container -Path $bin_dir)) {
  New-Item -ItemType Directory $bin_dir | Out-Null
}

# Now that they exist, validate them and make them absolute
$bin_dir = (Resolve-Path $bin_dir)
$src_dir = (Resolve-Path $src_dir)

Push-Location $src_dir
try {

  # FIXME: These steps assume the build steps from the Makefile of version 3.6.9.

  # 1. Set Up QuickLisp if needed
  if (-not (Test-Path -PathType Container -Path build\quicklisp)) {
    Invoke-WebRequest http://beta.quicklisp.org/quicklisp.lisp -OutFile build\quicklisp.lisp
    &$sbcl --noinform --no-sysinit --no-userinit `
           --load build\quicklisp.lisp `
           --load src/getenv.lisp `
           --eval '(quicklisp-quickstart:install :path "build/quicklisp" :proxy (getenv "http_proxy"))' `
           --eval '(quit)'
  }

  # 2. Set Up Dependencies
  if (-not (Test-Path -PathType Leaf -Path build\libs.stamp)) {
    Copy-Item -Destination $sbcl_home $dep_dir\*.dll
    &$sbcl --noinform --no-sysinit --no-userinit `
           --load build\quicklisp\setup.lisp `
           --eval '(push :pgloader-image *features*)' `
           --eval '(setf *print-circle* t *print-pretty* t)' `
           --eval ('(push "{0}" ql:*local-project-directories*)' -f (Resolve-Path .)) `
           --eval '(ql:quickload "pgloader")' `
           --eval '(quit)'
    New-Item -ItemType File build\libs.stamp | Out-Null
  }

  # 3. Write Manifest
  if (-not (Test-Path -PathType Leaf -Path build\manifest.ql)) {
    &$sbcl --noinform --no-sysinit --no-userinit `
           --load build\quicklisp\setup.lisp `
           --eval '(ql:write-asdf-manifest-file "build/manifest.ql")' `
           --eval '(quit)'
  }

  # 4. Create buildapp.exe
  if (-not (Test-Path -PathType Leaf -Path build\bin\buildapp.exe)) {
    &$sbcl --noinform --no-sysinit --no-userinit `
           --load build\quicklisp\setup.lisp `
           --eval '(ql:quickload "buildapp")' `
           --eval '(buildapp:build-buildapp "build/bin/buildapp.exe")' `
           --eval '(quit)'
  }

  # 5. Create pgloader.exe
  if (-not (Test-Path -PathType Leaf -Path $bin_dir\pgloader.exe)) {
    &.\build\bin\buildapp.exe --logfile build/bin/buildapp.log `
                              --require sb-posix `
                              --require sb-bsd-sockets `
                              --require sb-rotate-byte `
                              --sbcl $sbcl `
                              --asdf-path . `
                              --asdf-tree build/quicklisp/local-projects `
                              --manifest-file build/manifest.ql `
                              --asdf-tree build/quicklisp/dists `
                              --asdf-path . `
                              --load-system cffi `
                              --load-system cl+ssl `
                              --load-system mssql `
                              --load src/hooks.lisp `
                              --load-system pgloader `
                              --entry pgloader:main `
                              --dynamic-space-size $ImageMemorySize `
                              --output build/bin/pgloader.exe
    Copy-Item -Destination $bin_dir $dep_dir\*.dll
    Copy-Item -Destination $bin_dir build\bin\pgloader.exe
  }
}
finally {
  Pop-Location
}
Zastai commented 1 year ago

One thing to note is that the resulting binary seems to get tied to your machine and user. Not sure which component it is (perhaps OpenSSL trying to access the .ssh folder?) but if moved to another machine, the executable will error out unless your windows user folder (e.g. "C:\Users\foo") exists there.

ladudu commented 6 months ago

I tried compiling on win10 and spent all day without success. Has anyone succeeded? I need to use pgloader on win server, does anyone have any good suggestions?

ladudu commented 6 months ago

For reference: the build script I'm using (to avoid having to use a make from WSL/MSYS/...).

This expects to be in a directory containing:

  • a folder with the SBCL binaries

    • the current SBCL works fine without being installed, so to avoid messing with PATH I would recommend

    • install the MSI to some location

    • copy the contents of that location into a sbcl-<version> folder

    • uninstall the MSI

  • a "dependencies" folder containing the DLLs needed. Currently, these are:

    • libcrypto-1_1-x64.dll and libssl-1_1-x64.dll (from OpenSSL)

    • Lite MSI installer, put DLLs in bin dir instead of system, copy them from that bin dir, uninstall MSI

    • libiconv-2.dll and sybdb.dll (renamed from libsybdb-5.dll) from FreeTDS

    • sqlite3.dll (from SQLite)

It also works around an issue with two source files that will fail to compile (during the final buildapp step) when they have CRLF line endings.

param (
  [string] $Version = '3.6.9',
  # pfloader defaults to 16GB, except on windows where it uses 1GB. We default to 8GB as a compromise.
  [int] $ImageMemorySize = 8192,
  [switch] $NoGit
)

$ErrorActionPreference = 'Stop'

$dep_dir = (Resolve-Path dependencies)
$sbcl_home = (Resolve-Path sbcl-2.3.1)
$sbcl = (Resolve-Path (Join-Path $sbcl_home sbcl.exe))

# FIXME: This naming is intended to match what GitHub supplies as top-level folder in its sources.zip; not sure if that's safe.
$src_dir = "pgloader-$Version"

# 0. Get the source code for the requested version
if (-not (Test-Path -PathType Container $src_dir)) {
  if ($NoGit) {
    Invoke-WebRequest https://github.com/dimitri/pgloader/archive/refs/tags/v$Version.zip -OutFile pgloader-sources.zip
    Expand-Archive -Path pgloader-sources.zip -DestinationPath . -Force
    # Without a wait, Remove-Item will fail with "file in use"
    Start-Sleep -Seconds 3
    Remove-Item pgloader-sources.zip
  }
  else {
    git clone --depth 1 --branch v$Version https://github.com/dimitri/pgloader $src_dir
  }
  # Some files need to have LF line endings
  'src/pgsql/pgsql-trigger.lisp', 'src/sources/csv/csv.lisp' | ForEach-Object {
    $file = (Resolve-Path (Join-Path $src_dir $_))
    $contents = (Get-Content -Raw $file) -replace "`r`n","`n"
    Set-Content -Path $file -Value $contents -NoNewline
  }
}

$bin_dir = 'binaries'

if (-not (Test-Path -PathType Container -Path $bin_dir)) {
  New-Item -ItemType Directory $bin_dir | Out-Null
}

# Now that they exist, validate them and make them absolute
$bin_dir = (Resolve-Path $bin_dir)
$src_dir = (Resolve-Path $src_dir)

Push-Location $src_dir
try {

  # FIXME: These steps assume the build steps from the Makefile of version 3.6.9.

  # 1. Set Up QuickLisp if needed
  if (-not (Test-Path -PathType Container -Path build\quicklisp)) {
    Invoke-WebRequest http://beta.quicklisp.org/quicklisp.lisp -OutFile build\quicklisp.lisp
    &$sbcl --noinform --no-sysinit --no-userinit `
           --load build\quicklisp.lisp `
           --load src/getenv.lisp `
           --eval '(quicklisp-quickstart:install :path "build/quicklisp" :proxy (getenv "http_proxy"))' `
           --eval '(quit)'
  }

  # 2. Set Up Dependencies
  if (-not (Test-Path -PathType Leaf -Path build\libs.stamp)) {
    Copy-Item -Destination $sbcl_home $dep_dir\*.dll
    &$sbcl --noinform --no-sysinit --no-userinit `
           --load build\quicklisp\setup.lisp `
           --eval '(push :pgloader-image *features*)' `
           --eval '(setf *print-circle* t *print-pretty* t)' `
           --eval ('(push "{0}" ql:*local-project-directories*)' -f (Resolve-Path .)) `
           --eval '(ql:quickload "pgloader")' `
           --eval '(quit)'
    New-Item -ItemType File build\libs.stamp | Out-Null
  }

  # 3. Write Manifest
  if (-not (Test-Path -PathType Leaf -Path build\manifest.ql)) {
    &$sbcl --noinform --no-sysinit --no-userinit `
           --load build\quicklisp\setup.lisp `
           --eval '(ql:write-asdf-manifest-file "build/manifest.ql")' `
           --eval '(quit)'
  }

  # 4. Create buildapp.exe
  if (-not (Test-Path -PathType Leaf -Path build\bin\buildapp.exe)) {
    &$sbcl --noinform --no-sysinit --no-userinit `
           --load build\quicklisp\setup.lisp `
           --eval '(ql:quickload "buildapp")' `
           --eval '(buildapp:build-buildapp "build/bin/buildapp.exe")' `
           --eval '(quit)'
  }

  # 5. Create pgloader.exe
  if (-not (Test-Path -PathType Leaf -Path $bin_dir\pgloader.exe)) {
    &.\build\bin\buildapp.exe --logfile build/bin/buildapp.log `
                              --require sb-posix `
                              --require sb-bsd-sockets `
                              --require sb-rotate-byte `
                              --sbcl $sbcl `
                              --asdf-path . `
                              --asdf-tree build/quicklisp/local-projects `
                              --manifest-file build/manifest.ql `
                              --asdf-tree build/quicklisp/dists `
                              --asdf-path . `
                              --load-system cffi `
                              --load-system cl+ssl `
                              --load-system mssql `
                              --load src/hooks.lisp `
                              --load-system pgloader `
                              --entry pgloader:main `
                              --dynamic-space-size $ImageMemorySize `
                              --output build/bin/pgloader.exe
    Copy-Item -Destination $bin_dir $dep_dir\*.dll
    Copy-Item -Destination $bin_dir build\bin\pgloader.exe
  }
}
finally {
  Pop-Location
}

Can you share your project? I wanted to try building on my computer but without success, I'm not familiar with powershell