dimitri / pgloader

Migrate to PostgreSQL in a single command!
http://pgloader.io
Other
5.33k stars 540 forks source link

KABOOM! FATAL error: There is no thread support in this instance. #1101

Open rotdrop opened 4 years ago

rotdrop commented 4 years ago

Sorry, the bug report below probably does not meet the minimal requirements. However, /usr/bin/pgloade is a binary and the error message even with debug just does not give me a clue. Can be that I should file a bug report to the Raspbian or Debian maintainers.

No

Tried, no results.

pgloader  mysql://homeassistant:onohLAH4feHeJS6j@localhost/homeassistant?charset=utf8mb4 postgres://@/hass
KABOOM!
FATAL error: There is no thread support in this instance.
An unhandled error condition has been signalled:
   There is no thread support in this instance.

What I am doing here?

There is no thread support in this instance.

With debug:

homeassistant@raspberrypi:~/.homeassistant $ pgloader -d mysql://homeassistant:onohLAH4feHeJS6j@localhost/homeassistant?charset=utf8mb4 postgres://@/hass
mkdir -p #P"/tmp/pgloader/"
pgloader version 3.6.1
sb-impl::*default-external-format* :UTF-8
tmpdir: #P"/tmp/pgloader/"
KABOOM!
FATAL error: There is no thread support in this instance.
Date/time: 2020-03-15-00:33An unhandled error condition has been signalled:
                              There is no thread support in this instance.

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {504E7001}>
0: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE-TO-STREAM #<SB-IMPL::STRING-OUTPUT-STREAM {5489E7E9}>)
1: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE #<BORDEAUX-THREADS::BORDEAUX-MP-CONDITION {5489E6C1}> :OUTPUT NIL :IF-EXISTS :APPEND :VERBOSE NIL)
2: ((FLET "H0" :IN PGLOADER::MAIN) #<BORDEAUX-THREADS::BORDEAUX-MP-CONDITION {5489E6C1}>)
3: (SB-KERNEL::%SIGNAL #<BORDEAUX-THREADS::BORDEAUX-MP-CONDITION {5489E6C1}>)
4: (ERROR #<BORDEAUX-THREADS::BORDEAUX-MP-CONDITION {5489E6C1}>)
5: (BORDEAUX-THREADS::%MAKE-THREAD #<unused argument> #<unused argument>)
6: (LPARALLEL.KERNEL::MAKE-WORKER-THREAD #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 1 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {5489E4F9}> #S(LPARALLEL.KERNEL::WORKER :HANDSHAKE/FROM-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK (NIL) :CVAR NIL) :HANDSHAKE/TO-WORKER #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK (NIL) :CVAR NIL) :EXIT-NOTIFICATION #S(LPARALLEL.CONS-QUEUE:CONS-QUEUE :IMPL #S(LPARALLEL.RAW-QUEUE:RAW-QUEUE :HEAD NIL :TAIL NIL) :LOCK (NIL) :CVAR NIL) :THREAD NIL :RUNNING-CATEGORY NIL :INDEX 0 :TASKS #S(LPARALLEL.SPIN-QUEUE:SPIN-QUEUE :HEAD #1=(LPARALLEL.SPIN-QUEUE::DUMMY) :TAIL #1#)) "lparallel" ((PGLOADER.PARAMS:*LOG-FILENAME* . #P"/tmp/pgloader/pgloader.log") (PGLOADER.PARAMS:*LOG-MIN-MESSAGES* . :DEBUG) (PGLOADER.PARAMS:*CLIENT-MIN-MESSAGES* . :DEBUG) (PGLOADER.MONITOR:*MONITORING-QUEUE*) (*ERROR-OUTPUT* . #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDERR* {4F07D8B1}>) (PGLOADER.PARAMS:*ROOT-DIR* . #P"/tmp/pgloader/") (*STANDARD-OUTPUT* . #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDOUT* {4F203C29}>) (PGLOADER.PARAMS:*SUMMARY-PATHNAME*) (PGLOADER.MONITOR::*SECTIONS* QUOTE #S(PGLOADER.STATE::STATE :PRELOAD #S(PGLOADER.STATE::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 0 {5489DDF9}> :TABNAMES NIL :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0 :RS 0.0 :WS 0.0 :BYTES 0) :DATA #S(PGLOADER.STATE::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 0 {5489E061}> :TABNAMES NIL :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0 :RS 0.0 :WS 0.0 :BYTES 0) :POSTLOAD #S(PGLOADER.STATE::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 0 {5489E2C9}> :TABNAMES NIL :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0 :RS 0.0 :WS 0.0 :BYTES 0) :BYTES 0 :SECS 0.0)) (LPARALLEL.KERNEL:*KERNEL* . #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 1 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {5489E4F9}>)))
7: (LPARALLEL.KERNEL::MAKE-WORKER #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 1 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {5489E4F9}> 0 #S(LPARALLEL.SPIN-QUEUE:SPIN-QUEUE :HEAD #1=(LPARALLEL.SPIN-QUEUE::DUMMY) :TAIL #1#))
8: (LPARALLEL.KERNEL::%FILL-WORKERS #(0) #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 1 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {5489E4F9}>)
9: (LPARALLEL.KERNEL::FILL-WORKERS #(0) #<LPARALLEL.KERNEL:KERNEL :NAME "lparallel" :WORKER-COUNT 1 :USE-CALLER NIL :ALIVE T :SPIN-COUNT 2000 {5489E4F9}>)
10: (LPARALLEL.KERNEL:MAKE-KERNEL 1 :NAME "lparallel" :BINDINGS ((PGLOADER.PARAMS:*LOG-FILENAME* . #P"/tmp/pgloader/pgloader.log") (PGLOADER.PARAMS:*LOG-MIN-MESSAGES* . :DEBUG) (PGLOADER.PARAMS:*CLIENT-MIN-MESSAGES* . :DEBUG) (PGLOADER.MONITOR:*MONITORING-QUEUE*) (*ERROR-OUTPUT* . #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDERR* {4F07D8B1}>) (PGLOADER.PARAMS:*ROOT-DIR* . #P"/tmp/pgloader/") (*STANDARD-OUTPUT* . #<SYNONYM-STREAM :SYMBOL SB-SYS:*STDOUT* {4F203C29}>) (PGLOADER.PARAMS:*SUMMARY-PATHNAME*) (PGLOADER.MONITOR::*SECTIONS* QUOTE #S(PGLOADER.STATE::STATE :PRELOAD #S(PGLOADER.STATE::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 0 {5489DDF9}> :TABNAMES NIL :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0 :RS 0.0 :WS 0.0 :BYTES 0) :DATA #S(PGLOADER.STATE::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 0 {5489E061}> :TABNAMES NIL :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0 :RS 0.0 :WS 0.0 :BYTES 0) :POSTLOAD #S(PGLOADER.STATE::PGSTATE :TABLES #<HASH-TABLE :TEST EQUAL :COUNT 0 {5489E2C9}> :TABNAMES NIL :READ 0 :ROWS 0 :ERRS 0 :SECS 0.0 :RS 0.0 :WS 0.0 :BYTES 0) :BYTES 0 :SECS 0.0))) :CONTEXT NIL :SPIN-COUNT NIL :USE-CALLER NIL)
11: (PGLOADER.MONITOR:START-MONITOR :START-LOGGER T :QUEUE NIL :LOG-FILENAME NIL :LOG-MIN-MESSAGES NIL :CLIENT-MIN-MESSAGES NIL)
12: (PGLOADER.MONITOR::CALL-WITH-MONITOR #<CLOSURE (LAMBDA NIL :IN PGLOADER::MAIN) {5489E36D}>)
13: (PGLOADER::MAIN ("pgloader" "-d" "mysql://homeassistant:onohLAH4feHeJS6j@localhost/homeassistant?charset=utf8mb4" "postgres://@/hass"))
14: ((LAMBDA NIL :IN "/build/pgloader-YEgT9v/pgloader-3.6.1/dumper-2SKVI5f7.lisp"))
15: ((FLET SB-UNIX::BODY :IN SAVE-LISP-AND-DIE))
16: ((FLET "WITHOUT-INTERRUPTS-BODY-14" :IN SAVE-LISP-AND-DIE))
17: ((LABELS SB-IMPL::RESTART-LISP :IN SAVE-LISP-AND-DIE))

What I am doing here?

There is no thread support in this instance
dimitri commented 4 years ago

Thread support is provided by the lisp implementation, here SBCL 1.4.15. Which platform are you using, where SBCL does not have threads? Consider using a newer version of SBCL with thread support for your platform (see http://www.sbcl.org) or using Clozure-CL maybe?

eldemcan commented 4 years ago

I have same problem,

pi@raspberrypi:~/TAOP/data/cdstore$ sudo apt show sbcl
Package: sbcl
Version: 2:1.4.16-2
Priority: optional
Section: lisp
Maintainer: Debian Common Lisp Team <debian-common-lisp@lists.debian.org>
Installed-Size: 38.1 MB
Provides: lisp-compiler, sbcl-fasl-loader-78
Depends: libc6 (>= 2.28), zlib1g (>= 1:1.1.4)
Recommends: binfmt-support
Suggests: sbcl-doc, sbcl-source, slime
Breaks: cl-asdf (<< 2:3.1.5)
Homepage: http://www.sbcl.org
Download-Size: 7,798 kB
APT-Sources: http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages
pi@raspberrypi:~/TAOP/data/cdstore$ pgloader  --version
pgloader version "3.6.1"
compiled with SBCL 1.4.15.debian
pi@raspberrypi:~/TAOP/data/cdstore$

Installed pgloader using apt .

phoe commented 4 years ago

1.4.16 is very old and will likely not run modern pgloader. Can you try with a newer SBCL version?

Is your SBCL built with threading support? Run sbcl and type *features*; check if :THREAD-SUPPORT is present within the output.

Ahsoka commented 4 years ago

After looking into this issue, I've stumbled across this source that says that SBCL does not support threading on 32-bit ARM architectures. As far as I can tell from the trace-back above this issue (so far) has only occurred on Raspberry Pis (including mine), which happen to have a 32-bit ARM architecture. Does this mean that pgloader cannot be used on 32-bit ARM architectures? Also some more information I discovered while trying to get this to work is when using apt-get install sbcl, the latest stable version available to the Pi is 2:1.4.16-2 (which can be seen in @eldemcan's traceback). However, it appears that regardless of the version of SBCL, when using pgloader --version it always displays compiled with SBCL 1.4.15.debian, even after reinstalling pgloader . I have no idea why that's happening.

I am not well-versed enough in SPCL or Linux to develop a work-around or to even determine whether or not one is even possible. Would greatly appreciate if someone could tell me whether trying to get pgloader to work on a 32-bit architecture is work the effort or if it is just a lost cause.

phoe commented 4 years ago

Does this mean that pgloader cannot be used on 32-bit ARM architectures?

AFAIK CCL supports threads on arm32.

Ahsoka commented 4 years ago

AFAIK CCL supports threads on arm32.

So what would be the next step in trying to get this to work. As I mentioned above, I am not very well versed with Linux or SPCL. Tbh, I don't even know what AFAIK CCL is.

phoe commented 4 years ago

CCL is Clozure Common Lisp, another implementation of CL.

I think the steps would look like this: grab a copy from https://github.com/Clozure/ccl/releases; make the CCL available in the path symlinked to ccl; edit https://github.com/dimitri/pgloader/blob/master/Makefile#L6 to say ccl instead of sbcl; build pgloader.

Ahsoka commented 4 years ago

CCL is Clozure Common Lisp, another implementation of CL.

I think the steps would look like this: grab a copy from https://github.com/Clozure/ccl/releases; make the CCL available in the path symlinked to ccl; edit https://github.com/dimitri/pgloader/blob/master/Makefile#L6 to say ccl instead of sbcl; build pgloader.

Alright I will try this and report back my findings.

Ahsoka commented 4 years ago

While trying to build I got this message.

Loading "zs3"
> Error: Error while trying to load definition for system ironclad from
>        pathname
>        /home/pi/pgloader/build/quicklisp/dists/quicklisp/software/ironclad-v0.50/ironclad.asd:
>
>           :IF-FEATURE is an invalid initarg to INITIALIZE-INSTANCE for #<STANDARD-CLASS MODULE>.
>        Valid initargs: (:NAME :VERSION :DESCRIPTION :LONG-DESCRIPTION
>                         :IN-ORDER-TO :DO-FIRST :PARENT :PATHNAME
>                         :AROUND-COMPILE :ENCODING :PROPERTIES :COMPONENTS
>                         :IF-COMPONENT-DEP-FAILS :DEFAULT-COMPONENT-CLASS).
> While executing: (:INTERNAL ASDF::LOAD-SYSDEF), in process listener(1).
> Type :GO to continue, :POP to abort, :R for a list of available restarts.
> If continued: Skip evaluation of (ql:quickload "pgloader")
> Type :? for other options.
1 >

862 might be a related issue, but I am not sure.

phoe commented 4 years ago

Which CCL version are you using? (Try running ccl and posting the greeting banner printed by the compiler.)

Ahsoka commented 4 years ago

I ran ccl and here is the output ⬇

Clozure Common Lisp Version 1.12 (v1.12) LinuxARM32

For more information about CCL, please see http://ccl.clozure.com.

CCL is free software.  It is distributed under the terms of the Apache
Licence, Version 2.0.

I am pretty sure this is the latest version.

phoe commented 4 years ago

Yes, CCL 1.12 is the latest. Strange.

Once you get this error message while building, can you issue (asdf:asdf-version)?

Ahsoka commented 4 years ago

Alright while building I got the same error as above. Here's the output from (asdf:asdf-version)

"2.26"
phoe commented 4 years ago

Wow! 2.26 is ancient - no doubt that pgloader doesn't build.

We need to download the recent ASDF from https://common-lisp.net/project/asdf/archives/asdf.lisp and load it before pgloader is attempted to be built.

At https://github.com/dimitri/pgloader/blob/master/Makefile#L120-L125 there is a passage:

$(BUILDAPP_CCL): $(QLDIR)/setup.lisp
    mkdir -p $(BUILDDIR)/bin
    $(CL) $(CL_OPTS) --load $(QLDIR)/setup.lisp               \
             --eval '(ql:quickload "buildapp")'                   \
             --eval '(buildapp:build-buildapp "$@")'              \
             --eval '(quit)'

Let's try modifying that to:

$(BUILDAPP_CCL): $(QLDIR)/setup.lisp
    mkdir -p $(BUILDDIR)/bin
    $(CL) $(CL_OPTS)                                          \
             --load /home/your/path/to/asdf.lisp                  \
             --load $(QLDIR)/setup.lisp                           \
             --eval '(ql:quickload "buildapp")'                   \
             --eval '(buildapp:build-buildapp "$@")'              \
             --eval '(quit)'

Where /home/your/path/to/asdf.lisp points to the ASDF release you have downloaded.


@dimitri: it seems that pgloader has an implicit dependency on ASDF3 - this should be specified somewhere in the code, just like split-sequence does:

#.(unless (or #+asdf3.1 (version<= "3.1" (asdf-version)))
    (error "You need ASDF >= 3.1 to load this system correctly."))
Ahsoka commented 4 years ago

Hmmm... something weird happened. I did as you instructed above, however, the same error appeared. And even more puzzling is when I ran (asdf:asdf-version) it still printed "2.26", even though I installed the new version and added it to the Makefile.

Here's my Makefile for reference:

# pgloader build tool
APP_NAME   = pgloader
VERSION    = 3.6.2

# use either sbcl or ccl
CL         = armcl

# default to 4096 MB of RAM size in the image
DYNSIZE    = 4096

LISP_SRC   = $(wildcard src/*lisp)         \
             $(wildcard src/monkey/*lisp)  \
             $(wildcard src/utils/*lisp)   \
             $(wildcard src/load/*lisp)    \
             $(wildcard src/parsers/*lisp) \
             $(wildcard src/pg-copy/*lisp) \
             $(wildcard src/pgsql/*lisp)   \
             $(wildcard src/sources/*lisp) \
             pgloader.asd

BUILDDIR   = build
LIBS       = $(BUILDDIR)/libs.stamp
QLDIR      = $(BUILDDIR)/quicklisp
MANIFEST   = $(BUILDDIR)/manifest.ql
LATEST     = $(BUILDDIR)/pgloader-latest.tgz

BUNDLEDIST = 2020-02-18
BUNDLENAME = pgloader-bundle-$(VERSION)
BUNDLEDIR  = $(BUILDDIR)/bundle/$(BUNDLENAME)
BUNDLE     = $(BUILDDIR)/$(BUNDLENAME).tgz
BUNDLETESTD= $(BUILDDIR)/bundle/test

ifeq ($(OS),Windows_NT)
EXE           = .exe
COMPRESS_CORE = no
DYNSIZE       = 1024            # support for windows 32 bits
else
EXE =
endif

PGLOADER   = $(BUILDDIR)/bin/$(APP_NAME)$(EXE)
BUILDAPP_CCL  = $(BUILDDIR)/bin/buildapp.ccl$(EXE)
BUILDAPP_SBCL = $(BUILDDIR)/bin/buildapp.sbcl$(EXE)

ifeq ($(CL),sbcl)
BUILDAPP      = $(BUILDAPP_SBCL)
BUILDAPP_OPTS = --require sb-posix                      \
                --require sb-bsd-sockets                \
                --require sb-rotate-byte
CL_OPTS    = --noinform --no-sysinit --no-userinit
else
BUILDAPP   = $(BUILDAPP_CCL)
CL_OPTS    = --no-init
endif

ifeq ($(CL),sbcl)
COMPRESS_CORE ?= $(shell $(CL) --noinform \
                               --quit     \
                               --eval '(when (member :sb-core-compression cl:*features*) (write-string "yes"))')

endif

# note: on Windows_NT, we never core-compress; see above.
ifeq ($(COMPRESS_CORE),yes)
COMPRESS_CORE_OPT = --compress-core
endif

DEBUILD_ROOT = /tmp/pgloader

all: $(PGLOADER)

clean:
        rm -rf $(LIBS) $(QLDIR) $(MANIFEST) $(BUILDAPP) $(PGLOADER) docs/_build

$(QLDIR)/local-projects/qmynd:
        git clone --depth 1 https://github.com/qitab/qmynd.git $@

$(QLDIR)/local-projects/cl-ixf:
        git clone --depth 1 https://github.com/dimitri/cl-ixf.git $@

$(QLDIR)/local-projects/cl-db3:
        git clone --depth 1 https://github.com/dimitri/cl-db3.git $@

$(QLDIR)/local-projects/cl-csv:
        git clone --depth 1 https://github.com/AccelerationNet/cl-csv.git $@

$(QLDIR)/setup.lisp:
        mkdir -p $(BUILDDIR)
        curl -o $(BUILDDIR)/quicklisp.lisp http://beta.quicklisp.org/quicklisp.lisp
        $(CL) $(CL_OPTS) --load $(BUILDDIR)/quicklisp.lisp                        \
             --load src/getenv.lisp                                               \
             --eval '(quicklisp-quickstart:install :path "$(BUILDDIR)/quicklisp" :proxy (getenv "http_proxy"))' \
             --eval '(quit)'

quicklisp: $(QLDIR)/setup.lisp ;

clones: $(QLDIR)/local-projects/cl-ixf \
        $(QLDIR)/local-projects/cl-db3 \
        $(QLDIR)/local-projects/cl-csv \
        $(QLDIR)/local-projects/qmynd ;

$(LIBS): $(QLDIR)/setup.lisp
        $(CL) $(CL_OPTS) --load $(QLDIR)/setup.lisp                   \
             --eval '(push :pgloader-image *features*)'               \
             --eval '(setf *print-circle* t *print-pretty* t)'        \
             --eval '(push "$(PWD)/" ql:*local-project-directories*)' \
             --eval '(ql:quickload "pgloader")'                       \
             --eval '(quit)'
        touch $@

libs: $(LIBS) ;

$(MANIFEST): $(LIBS)
        $(CL) $(CL_OPTS) --load $(QLDIR)/setup.lisp                \
             --eval '(ql:write-asdf-manifest-file "$(MANIFEST)")'  \
             --eval '(quit)'

manifest: $(MANIFEST) ;

$(BUILDAPP_CCL): $(QLDIR)/setup.lisp
        mkdir -p $(BUILDDIR)/bin
        $(CL) $(CL_OPTS)                                          \
             --load /home/pi/pgloader/asdf.lisp                   \
             --load $(QLDIR)/setup.lisp                           \
             --eval '(ql:quickload "buildapp")'                   \
             --eval '(buildapp:build-buildapp "$@")'              \
             --eval '(quit)'

$(BUILDAPP_SBCL): $(QLDIR)/setup.lisp
        mkdir -p $(BUILDDIR)/bin
        $(CL) $(CL_OPTS) --load $(QLDIR)/setup.lisp               \
             --eval '(ql:quickload "buildapp")'                   \
             --eval '(buildapp:build-buildapp "$@")'              \
             --eval '(quit)'

buildapp: $(BUILDAPP) ;

$(PGLOADER): $(MANIFEST) $(BUILDAPP) $(LISP_SRC)
        mkdir -p $(BUILDDIR)/bin
        $(BUILDAPP)      --logfile /tmp/build.log                \
                         $(BUILDAPP_OPTS)                        \
                         --sbcl $(CL)                            \
                         --asdf-path .                           \
                         --asdf-tree $(QLDIR)/local-projects     \
                         --manifest-file $(MANIFEST)             \
                         --asdf-tree $(QLDIR)/dists              \
                         --asdf-path .                           \
                         --load-system cffi                      \
                         --load-system cl+ssl                    \
                         --load-system mssql                     \
                         --load src/hooks.lisp                   \
                         --load-system $(APP_NAME)               \
                         --entry pgloader:main                   \
                         --dynamic-space-size $(DYNSIZE)         \
                         $(COMPRESS_CORE_OPT)                    \
                         --output $@.tmp
        # that's ugly, but necessary when building on Windows :(
        mv $@.tmp $@

pgloader: $(PGLOADER) ;

pgloader-standalone:
        $(BUILDAPP)    $(BUILDAPP_OPTS)                        \
                       --sbcl $(CL)                            \
                       --load-system $(APP_NAME)               \
                       --load src/hooks.lisp                   \
                       --entry pgloader:main                   \
                       --dynamic-space-size $(DYNSIZE)         \
                       $(COMPRESS_CORE_OPT)                    \
                       --output $(PGLOADER)
test: $(PGLOADER)
        $(MAKE) PGLOADER=$(realpath $(PGLOADER)) CL=$(CL) -C test regress

save: ./src/save.lisp $(LISP_SRC)
        $(CL) $(CL_OPTS) --load ./src/save.lisp

check-saved:
        $(MAKE) PGLOADER=$(realpath $(PGLOADER)) CL=$(CL) -C test regress

clean-bundle:
        rm -rf $(BUNDLEDIR)
        rm -rf $(BUNDLETESTD)/$(BUNDLENAME)/*

$(BUNDLETESTD):
        mkdir -p $@

$(BUNDLEDIR):
        mkdir -p $@
        $(CL) $(CL_OPTS) --load $(QLDIR)/setup.lisp      \
             --eval '(defvar *bundle-dir* "$@")'         \
             --eval '(defvar *pwd* "$(PWD)/")'           \
             --eval '(defvar *ql-dist* "$(BUNDLEDIST)")' \
             --load bundle/ql.lisp

$(BUNDLEDIR)/version.sexp: $(BUNDLEDIR)
        echo "\"$(VERSION)\"" > $@

$(BUNDLE): $(BUNDLEDIR) $(BUNDLEDIR)/version.sexp
        cp bundle/README.md $(BUNDLEDIR)
        cp bundle/save.lisp $(BUNDLEDIR)
        sed -e s/%VERSION%/$(VERSION)/ < bundle/Makefile > $(BUNDLEDIR)/Makefile
        git archive --format=tar --prefix=pgloader-$(VERSION)/ master \
             | tar -C $(BUNDLEDIR)/local-projects/ -xf -
        make QLDIR=$(BUNDLEDIR) clones
        tar -C build/bundle                 \
            --exclude bin                   \
            --exclude test/sqlite           \
            -czf $@ $(BUNDLENAME)

bundle: clean-bundle $(BUNDLE) $(BUNDLETESTD)
        tar -C $(BUNDLETESTD) -xf $(BUNDLE)
        make -C $(BUNDLETESTD)/$(BUNDLENAME)
        $(BUNDLETESTD)/$(BUNDLENAME)/bin/pgloader --version

test-bundle:
        $(MAKE) -C $(BUNDLEDIR) test

deb:
        # intended for use on a debian system
        mkdir -p $(DEBUILD_ROOT) && rm -rf $(DEBUILD_ROOT)/*
        rsync -Ca --exclude 'build'                                       \
                  --exclude '.vagrant'                                    \
              ./ $(DEBUILD_ROOT)/
        cd $(DEBUILD_ROOT) && make -f debian/rules orig
        cd $(DEBUILD_ROOT) && debuild -us -uc -sa
        cp -a /tmp/pgloader_* /tmp/cl-pgloader* build/

rpm:
        # intended for use on a CentOS or other RPM based system
        mkdir -p $(DEBUILD_ROOT) && rm -rf $(DEBUILD_ROOT)
        rsync -Ca --exclude=build/* ./ $(DEBUILD_ROOT)/
        cd /tmp && tar czf $(HOME)/rpmbuild/SOURCES/pgloader-$(VERSION).tar.gz pgloader
        cd $(DEBUILD_ROOT) && rpmbuild -ba pgloader.spec
        cp -a $(HOME)/rpmbuild/SRPMS/*rpm build
        cp -a $(HOME)/rpmbuild/RPMS/x86_64/*rpm build

pkg:
        # intended for use on a MacOSX system
        mkdir -p $(DEBUILD_ROOT) && rm -rf $(DEBUILD_ROOT)/*
        mkdir -p $(DEBUILD_ROOT)/usr/local/bin/
        mkdir -p $(DEBUILD_ROOT)/usr/local/share/man/man1/
        cp ./pgloader.1 $(DEBUILD_ROOT)/usr/local/share/man/man1/
        cp ./build/bin/pgloader $(DEBUILD_ROOT)/usr/local/bin/
        pkgbuild --identifier org.tapoueh.pgloader \
                 --root $(DEBUILD_ROOT)            \
                 --version $(VERSION)              \
                 ./build/pgloader-$(VERSION).pkg

latest:
        git archive --format=tar --prefix=pgloader-$(VERSION)/ v$(VERSION) \
        | gzip -9 > $(LATEST)

check: test ;

.PHONY: test pgloader-standalone docs bundle
phoe commented 4 years ago

I might have applied this to just one step but there seem to be more in the makefile.

Basically - look where Quicklisp, which is --load $(QLDIR)/setup.lisp, is loaded - and before each such occurrence, try to load the new ASDF in the same style in which we've edited the above makefile step.

Ahsoka commented 4 years ago

Alright we made progress, but we are not quite there yet. It appears that the fix you suggested worked to fix the error above, however, there is a new error. Also this time around when I ran (asdf:asdf-version) it outputed "3.3.4" instead of "2.26". I think we are close! 🙃

[package usocket]..............
Read error between positions 66 and 163 in /home/pi/pgloader/build/quicklisp/dists/quicklisp/software/usocket-0.8.3/backend/openmcl.lisp.
> Error: Foreign function not found: ARM-LINUX::|gethostname|
> While executing: CCL::LOAD-EXTERNAL-FUNCTION, in process listener(1).
> Type :GO to continue, :POP to abort, :R for a list of available restarts.
> If continued: Skip evaluation of (ql:quickload "pgloader")
> Type :? for other options.
1 >

You can see the Makefile file I used here.

phoe commented 4 years ago

Yes, ASDF 3.3.4 is what we want to have - that's good.

Now we have a different compilation issue - usocket fails to build on CCL for arm32 since the gethostname function was not found, and I do not know why that is the case.

Please file an issue on https://github.com/Clozure/ccl and link me there.

Ahsoka commented 4 years ago

Alright so I was able to solve the issue I mentioned above. See this issue for reference. However, now I run into another (which I believe) is directly related to pgloader. Here is the new error ⬇

;; loading system "cffi"
;; loading system "cl+ssl"
;; loading system "mssql"
;; loading file #P"/home/pi/pgloader/src/hooks.lisp"
;; loading system "pgloader"
Fatal SIMPLE-ERROR:
  Compilation failed: In POSTGRES-ARRAY-STRING-TO-ARRAY: Undeclared free variable WOL=CTC/WOL in /home/pi/pgloader/build/quicklisp/dists/quicklisp/software/postmodern-20200715-git/postmodern/util.lisp
Fatal error in "buildapp" : Error #<SILENT-EXIT-ERROR #x14640BFE>
make: *** [Makefile:146: build/bin/pgloader] Error 255
Ahsoka commented 4 years ago

I tried rebuilding from scratch as well and I got a similar error ⬇

;; loading system "cffi"
;; loading system "cl+ssl"
;; loading system "mssql"
;; loading file #P"/home/pi/pgloader-test/src/hooks.lisp"
;; loading system "pgloader"
Fatal error in "buildapp" : Error #<SILENT-EXIT-ERROR #x1466CE56>
make: *** [Makefile:146: build/bin/pgloader] Error 255
phoe commented 4 years ago

Congratulations - you have found a Postmodern error! :open_mouth:

See the ticket for a workaround in Postmodern code.

phoe commented 4 years ago

https://github.com/marijnh/Postmodern/issues/247 is now fixed. Please try using the newest version from master and let's continue trying to get this to work!

Ahsoka commented 4 years ago

How would go about using the newest version? Do I just rebuild pgloader?

phoe commented 4 years ago

I think pgloader will, by default, pick up the Quicklisp version, which is regenerated monthly. @dimitri - do you have any pointers?

Ahsoka commented 4 years ago

Alright I built from scratch (by deleting the old folder and using git clone again). Looks like pgloader didn't pick up the latest version. 😐 Here's the same error as before ⬇

;; loading system "cffi"
;; loading system "cl+ssl"
;; loading system "mssql"
;; loading file #P"/home/pi/pgloader/src/hooks.lisp"
;; loading system "pgloader"
Fatal error in "buildapp" : Error #<SILENT-EXIT-ERROR #x1463EACE>
make: *** [Makefile:146: build/bin/pgloader] Error 255

[Edit] Here's a longer version of the error ⬇

; Loading "buildapp"
[package buildapp]...........
;; loading system "buildapp"
mkdir -p build/bin
build/bin/buildapp.ccl      --logfile /tmp/build.log                \
                                                 \
                         --sbcl ccl                            \
                         --asdf-path .                           \
                         --asdf-tree build/quicklisp/local-projects     \
                         --manifest-file build/manifest.ql             \
                         --asdf-tree build/quicklisp/dists              \
                         --asdf-path .                           \
                         --load-system cffi                      \
                         --load-system cl+ssl                    \
                         --load-system mssql                     \
                         --load src/hooks.lisp                   \
                         --load-system pgloader               \
                         --entry pgloader:main                   \
                         --dynamic-space-size 4096         \
                                             \
                         --output build/bin/pgloader.tmp
;; loading system "cffi"
;; loading system "cl+ssl"
;; loading system "mssql"
;; loading file #P"/home/pi/pgloader/src/hooks.lisp"
;; loading system "pgloader"
Fatal error in "buildapp" : Error #<SILENT-EXIT-ERROR #x1463EACE>
make: *** [Makefile:146: build/bin/pgloader] Error 255
dimitri commented 4 years ago

The current Makefile organisation and buildapp use can make it hard to debug building pgloader in many cases. That's why there's also the https://github.com/dimitri/pgloader/blob/master/src/save.lisp option, and the make CL=ccl save target. Please try that one, and as all it does is the following, you could also try the following instead:

ccl64 --no-init --load ./src/save.lisp

That way @phoe might have more clues and will be in a better position to help debug the building of pgloader on your architecture.

Ahsoka commented 4 years ago

Error 1

Alright I tried rebuilding with ccl --no-init --load ./src/save.lisp and got this error ⬇

272,874 bytes in 0.03 seconds (8075.11KB/sec)
> Error: Unexpected HTTP status for #<URL "http://beta.quicklisp.org/archive/qmynd/2019-07-10/qmynd-20190710-git.tgz">: 206
> While executing: (:INTERNAL QL-HTTP:HTTP-FETCH), in process listener(1).
> Type :GO to continue, :POP to abort, :R for a list of available restarts.
> If continued: Skip loading "./src/save.lisp"
> Type :? for other options.
1 >

I am using ccl instead of ccl64 because the computer in question is 32bit ARM and I don't have the ccl64 command. Also when I ran (asdf:asdf-version) it outputted "3.3.3" instead of the normal "3.3.4".

Error 2

I tried rebuilding again, just to make the error was reproducible, and the error happened at a different time then the previous error. The errors seem similar in nature though, both of them have Error: Unexpected HTTP status for #<URL ... > in them. Also btw, (asdf:asdf-version) is still "3.3.3".

To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "mssql":
  Load 4 ASDF systems:
    asdf cffi iterate parse-number
  Install 2 Quicklisp releases:
    cl-mssql garbage-pools
> Error: Unexpected HTTP status for #<URL "http://beta.quicklisp.org/archive/garbage-pools/2013-07-20/garbage-pools-20130720-git.tgz">: 206
> While executing: (:INTERNAL QL-HTTP:HTTP-FETCH), in process listener(1).
> Type :GO to continue, :POP to abort, :R for a list of available restarts.
> If continued: Skip loading "./src/save.lisp"
> Type :? for other options.
1 >

Misc

I tried rebuilding again just out of curiosity to see what would happen, and I got the same result as Error 2. Once again, (asdf:asdf-version) is still "3.3.3".

quicklisp commented 4 years ago

The 206 status is generated from some intermediate process - the Quicklisp software does not request ranges. Do you perhaps have a HTTP proxy in your environment?

Ahsoka commented 4 years ago

Alright, it turns out (I think) that the "HTTP proxy" I had was nginx. Once I disabled it, the error above went away. But now I get another error and this one is short and sweet. All it says is Segmentation fault and that's it.

[Edit] I also tried rebuilding with sudo and still got the same simple error Segmentation fault.

phoe commented 4 years ago

Ouch. This is the worst. I hate debugging non-memory-safe code.

Could you post the full console output?

Ahsoka commented 4 years ago

Could you post the full console output?

That is the entire output :/ All it said was Segmentation fault. Looks it happened while it was loading package pgloader.user-symbols. Here's a screenshot of the error. ⬇️

Screen Shot 2020-08-23 at 3 37 12 PM

Is there anyway for me to get more debug info? Also thanks btw for putting so much effort to help me :) I really appreciate it!

phoe commented 4 years ago

No no, by "full output" I meant full output since issuing the command to build pgloader.

This means that either the error happened while pgloader.user-symbols was being loaded or it was happened after pgloader.user-symbols has been loaded successfully, and then stuff happened.

I do not know how to debug this - perhaps the /src/save.lisp file will need to be modified to include some debug information/console printouts. It seems that https://github.com/dimitri/pgloader/blob/master/src/save.lisp already contains some debugging information by means of its (format t ...) calls - it perhaps would be worth to investigate those.

Ahsoka commented 4 years ago

Alright the full output is super long, so here goes nothing. Have fun looking through this :p Super Long Full Output:

pi@raspberrypi:~/pgloader $ sudo ccl --no-init --load ./src/save.lisp
File /home/pi/pgloader/build/quicklisp/setup.lisp is not found, installing Quicklisp from http://beta.quicklisp.org/quicklisp.lisp
Running command: curl -o /home/pi/pgloader/build/quicklisp.lisp http://beta.quicklisp.org/quicklisp.lisp

  ==== quicklisp quickstart 2015-01-28 loaded ====

    To continue with installation, evaluate: (quicklisp-quickstart:install)

    For installation options, evaluate: (quicklisp-quickstart:help)

; Fetching #<URL "http://beta.quicklisp.org/client/quicklisp.sexp">
; 0.82KB
==================================================
838 bytes in 0.00 seconds (409.18KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/client/2020-01-04/quicklisp.tar">
; 250.00KB
==================================================
256,000 bytes in 0.04 seconds (6097.56KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/client/2015-09-24/setup.lisp">
; 4.94KB
==================================================
5,054 bytes in 0.00 seconds (4935.55KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/asdf/2.26/asdf.lisp">
; 194.07KB
==================================================
198,729 bytes in 0.05 seconds (4312.70KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/dist/quicklisp.txt">
; 0.40KB
==================================================
408 bytes in 0.00 seconds (398.44KB/sec)
Installing dist "quicklisp" version "2020-07-15".
; Fetching #<URL "http://beta.quicklisp.org/dist/quicklisp/2020-07-15/releases.txt">
; 458.86KB
==================================================
469,876 bytes in 0.05 seconds (9177.27KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/dist/quicklisp/2020-07-15/systems.txt">
; 336.13KB
==================================================
344,201 bytes in 0.42 seconds (802.23KB/sec)

  ==== quicklisp installed ====

    To load a system, use: (ql:quickload "system-name")

    To find systems, use: (ql:system-apropos "term")

    To load Quicklisp every time you start Lisp, use: (ql:add-to-init-file)

    For more information, see http://www.quicklisp.org/beta/

Loading file /home/pi/pgloader/build/quicklisp/setup.lisp
Loading system pgloader
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "cl-log":
  Install 1 Quicklisp release:
    cl-log
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-log/2013-01-28/cl-log.1.0.1.tgz">
; 18.03KB
==================================================
18,463 bytes in 0.00 seconds (9015.14KB/sec)
; Loading "cl-log"
[package com.ravenbrook.common-lisp-log]........
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "postmodern":
  Load 2 ASDF systems:
    asdf uiop
  Install 13 Quicklisp releases:
    alexandria bordeaux-threads cl-base64 cl-ppcre
    cl-unicode closer-mop global-vars ironclad md5
    postmodern split-sequence uax-15 usocket
; Fetching #<URL "http://beta.quicklisp.org/archive/global-vars/2014-11-06/global-vars-20141106-git.tgz">
; 3.50KB
==================================================
3,581 bytes in 0.00 seconds (3497.07KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/closer-mop/2020-07-15/closer-mop-20200715-git.tgz">
; 22.98KB
==================================================
23,533 bytes in 0.00 seconds (22981.45KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-unicode/2019-05-21/cl-unicode-20190521-git.tgz">
; 576.85KB
==================================================
590,698 bytes in 0.06 seconds (10120.24KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/usocket/2019-12-27/usocket-0.8.3.tgz">
; 83.84KB
==================================================
85,851 bytes in 0.02 seconds (4191.94KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-ppcre/2019-05-21/cl-ppcre-20190521-git.tgz">
; 151.38KB
==================================================
155,009 bytes in 0.03 seconds (5219.86KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/uax-15/2020-03-25/uax-15-20200325-git.tgz">
; 731.75KB
==================================================
749,307 bytes in 0.05 seconds (13550.83KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/split-sequence/2019-05-21/split-sequence-v2.0.0.tgz">
; 10.09KB
==================================================
10,329 bytes in 0.00 seconds (10086.91KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/md5/2018-02-28/md5-20180228-git.tgz">
; 15.48KB
==================================================
15,847 bytes in 0.00 seconds (15475.59KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/ironclad/2020-07-15/ironclad-v0.50.tgz">
; 1421.33KB
==================================================
1,455,439 bytes in 0.08 seconds (17547.25KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-base64/2015-09-23/cl-base64-20150923-git.tgz">
; 8.39KB
==================================================
8,589 bytes in 0.00 seconds (8387.70KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/bordeaux-threads/2020-06-10/bordeaux-threads-v0.8.8.tgz">
; 23.15KB
==================================================
23,709 bytes in 0.00 seconds (23153.32KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/alexandria/2020-07-15/alexandria-20200715-git.tgz">
; 53.29KB
==================================================
54,572 bytes in 0.01 seconds (5329.30KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/postmodern/2020-07-15/postmodern-20200715-git.tgz">
; 498.72KB
==================================================
510,691 bytes in 0.05 seconds (11082.70KB/sec)
; Loading "postmodern"
.To load "flexi-streams":
  Load 1 ASDF system:
    asdf
  Install 2 Quicklisp releases:
    flexi-streams trivial-gray-streams
; Fetching #<URL "http://beta.quicklisp.org/archive/trivial-gray-streams/2018-10-18/trivial-gray-streams-20181018-git.tgz">
; 7.84KB
==================================================
8,024 bytes in 0.00 seconds (3917.97KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/flexi-streams/2020-07-15/flexi-streams-20200715-git.tgz">
; 431.83KB
==================================================
442,194 bytes in 1.59 seconds (271.93KB/sec)
; Loading "flexi-streams"
[package impl-specific-gray]......................
[package trivial-gray-streams]....................
[package flexi-streams]...........................
..................................................
..................................................
..................................................
..................................................
..................
To load "postmodern":
  Load 1 ASDF system:
    postmodern
; Loading "postmodern"
[package alexandria]..............................
..................................................
..................................................
[package alexandria-2]............................
[package md5].....................................
[package split-sequence]..........................
[package bordeaux-threads]........................
[package ironclad]................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
[package cl-base64]...............................
[package cl-ppcre]................................
..................................................
..................................................
[package uax-15]..................................
[package usocket].................................
[package cl-postgres].............................
[package cl-postgres-error].......................
[package cl-postgres-oid].........................
[package cl-postgres.features]....................
[package cl-postgres-trivial-utf-8]...............
[package cl-postgres-ieee-floats].................
..................................................
..................................................
[package s-sql]...................................
..................................................
[package global-vars].............................
[package cl-unicode]..............................
[package cl-unicode-names]........................
[package closer-mop]..............................
[package closer-common-lisp]......................
[package closer-common-lisp-user].................
..................................................
[package postmodern]..............................
.....................
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "qmynd":
  Load 8 ASDF systems:
    alexandria asdf bordeaux-threads flexi-streams ironclad
    trivial-gray-streams uiop usocket
  Install 9 Quicklisp releases:
    asdf-finalizers babel cffi chipz cl+ssl qmynd salza2
    trivial-features trivial-garbage
; Fetching #<URL "http://beta.quicklisp.org/archive/salza2/2013-07-20/salza2-2.0.9.tgz">
; 15.16KB
==================================================
15,525 bytes in 0.00 seconds (15161.13KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/asdf-finalizers/2017-04-03/asdf-finalizers-20170403-git.tgz">
; 6.51KB
==================================================
6,663 bytes in 0.00 seconds (3253.42KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/trivial-garbage/2020-03-25/trivial-garbage-20200325-git.tgz">
; 10.16KB
==================================================
10,404 bytes in 0.00 seconds (10160.16KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/cffi/2020-07-15/cffi_0.23.0.tgz">
; 255.55KB
==================================================
261,680 bytes in 0.05 seconds (5010.72KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/cl+ssl/2020-06-10/cl+ssl-20200610-git.tgz">
; 53.03KB
==================================================
54,298 bytes in 0.01 seconds (4418.78KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/chipz/2019-02-02/chipz-20190202-git.tgz">
; 36.24KB
==================================================
37,110 bytes in 0.00 seconds (7248.05KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/trivial-features/2020-07-15/trivial-features-20200715-git.tgz">
; 10.80KB
==================================================
11,061 bytes in 0.00 seconds (10801.76KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/babel/2020-07-15/babel-20200715-git.tgz">
; 266.48KB
==================================================
272,874 bytes in 0.04 seconds (7613.67KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/qmynd/2019-07-10/qmynd-20190710-git.tgz">
; 47.73KB
==================================================
48,872 bytes in 0.01 seconds (4338.78KB/sec)
; Loading "qmynd"
[package babel-encodings].........................
[package babel]...................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
..................................................
[package list-of].................................
[package cffi-sys]................................
[package cffi-callbacks]..........................
[package cffi]....................................
..................................................
..................................................
[package cffi-features]...........................
[package trivial-garbage].........................
[package cl+ssl]..................................
[package chipz]...................................
..................................................
[package salza2]..................................
[package qmynd]...................................
[package qmynd-impl]..............................
...................
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "cl-csv":
  Load 3 ASDF systems:
    alexandria asdf cl-unicode
  Install 4 Quicklisp releases:
    cl-csv cl-interpol iterate named-readtables
; Fetching #<URL "http://beta.quicklisp.org/archive/iterate/2020-06-10/iterate-20200610-git.tgz">
; 324.93KB
==================================================
332,733 bytes in 0.05 seconds (6498.69KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/named-readtables/2020-06-10/named-readtables-20200610-git.tgz">
; 26.38KB
==================================================
27,014 bytes in 0.00 seconds (13190.43KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-interpol/2020-07-15/cl-interpol-20200715-git.tgz">
; 42.67KB
==================================================
43,690 bytes in 0.02 seconds (2133.30KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-csv/2018-08-31/cl-csv-20180831-git.tgz">
; 25.35KB
==================================================
25,961 bytes in 0.00 seconds (12676.27KB/sec)
; Loading "cl-csv"
[package iterate].................................
..................................................
[package editor-hints.named-readtables]...........
..................................................
[package cl-interpol].............................
[package cl-csv].........................
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "cl-fad":
  Load 3 ASDF systems:
    alexandria asdf bordeaux-threads
  Install 1 Quicklisp release:
    cl-fad
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-fad/2020-06-10/cl-fad-20200610-git.tgz">
; 24.09KB
==================================================
24,666 bytes in 0.01 seconds (1852.91KB/sec)
; Loading "cl-fad"
[package cl-fad]..................................
[package path]....................................
[package cl-fad-ccl].............
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "lparallel":
  Load 2 ASDF systems:
    alexandria bordeaux-threads
  Install 1 Quicklisp release:
    lparallel
; Fetching #<URL "http://beta.quicklisp.org/archive/lparallel/2016-08-25/lparallel-20160825-git.tgz">
; 76.71KB
==================================================
78,551 bytes in 0.02 seconds (3835.50KB/sec)
; Loading "lparallel"
[package lparallel.util]..........................
[package lparallel.thread-util]...................
[package lparallel.raw-queue].....................
[package lparallel.cons-queue]....................
[package lparallel.vector-queue]..................
[package lparallel.queue].........................
[package lparallel.biased-queue]..................
[package lparallel.kernel]........................
[package lparallel.kernel-util]...................
[package lparallel.promise].......................
[package lparallel.ptree].........................
[package lparallel.slet]..........................
[package lparallel.defpun]........................
..................................................
[package lparallel.cognate].......................
..................................................
[package lparallel]
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "esrap":
  Load 2 ASDF systems:
    alexandria asdf
  Install 1 Quicklisp release:
    esrap
; Fetching #<URL "http://beta.quicklisp.org/archive/esrap/2020-03-25/esrap-20200325-git.tgz">
; 67.73KB
==================================================
69,353 bytes in 0.02 seconds (4232.97KB/sec)
; Loading "esrap"
[package esrap]...................................
....................................
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "drakma":
  Load 8 ASDF systems:
    asdf chipz cl+ssl cl-base64 cl-ppcre flexi-streams
    trivial-gray-streams usocket
  Install 3 Quicklisp releases:
    chunga drakma puri
; Fetching #<URL "http://beta.quicklisp.org/archive/puri/2018-02-28/puri-20180228-git.tgz">
; 29.30KB
==================================================
30,002 bytes in 0.00 seconds (14649.41KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/chunga/2020-04-27/chunga-20200427-git.tgz">
; 20.12KB
==================================================
20,599 bytes in 0.00 seconds (10058.11KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/drakma/2019-11-30/drakma-v2.0.7.tgz">
; 72.55KB
==================================================
74,290 bytes in 0.01 seconds (4836.59KB/sec)
; Loading "drakma"
[package puri]....................................
[package chunga]..................................
[package drakma]..............
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "local-time":
  Load 2 ASDF systems:
    asdf cl-fad
  Install 1 Quicklisp release:
    local-time
; Fetching #<URL "http://beta.quicklisp.org/archive/local-time/2020-06-10/local-time-20200610-git.tgz">
; 276.27KB
==================================================
282,905 bytes in 0.05 seconds (5638.25KB/sec)
; Loading "local-time"
[package local-time]..............................
..
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "command-line-arguments":
  Load 1 ASDF system:
    asdf
  Install 1 Quicklisp release:
    command-line-arguments
; Fetching #<URL "http://beta.quicklisp.org/archive/command-line-arguments/2020-03-25/command-line-arguments-20200325-git.tgz">
; 11.66KB
==================================================
11,935 bytes in 0.00 seconds (11655.27KB/sec)
; Loading "command-line-arguments"
[package command-line-arguments]..............
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "abnf":
  Load 3 ASDF systems:
    asdf cl-ppcre esrap
  Install 1 Quicklisp release:
    cl-abnf
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-abnf/2020-03-25/cl-abnf-20200325-git.tgz">
; 8.15KB
==================================================
8,343 bytes in 0.00 seconds (8147.46KB/sec)
; Loading "abnf"
[package abnf]....
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "db3":
  Load 1 ASDF system:
    asdf
  Install 1 Quicklisp release:
    cl-db3
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-db3/2020-02-18/cl-db3-20200218-git.tgz">
; 6.02KB
==================================================
6,162 bytes in 0.00 seconds (6017.58KB/sec)
; Loading "db3"
[package db3]....
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "ixf":
  Load 7 ASDF systems:
    alexandria asdf babel cl-ppcre local-time md5
    split-sequence
  Install 2 Quicklisp releases:
    cl-ixf ieee-floats
; Fetching #<URL "http://beta.quicklisp.org/archive/ieee-floats/2017-08-30/ieee-floats-20170830-git.tgz">
; 5.20KB
==================================================
5,325 bytes in 0.00 seconds (5200.20KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-ixf/2018-02-28/cl-ixf-20180228-git.tgz">
; 8.47KB
==================================================
8,677 bytes in 0.00 seconds (8473.63KB/sec)
; Loading "ixf"
[package ieee-floats].............................
[package ixf]........
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "py-configparser":
  Load 1 ASDF system:
    asdf
  Install 2 Quicklisp releases:
    parse-number py-configparser
; Fetching #<URL "http://beta.quicklisp.org/archive/parse-number/2018-02-28/parse-number-v1.7.tgz">
; 5.58KB
==================================================
5,715 bytes in 0.00 seconds (5581.05KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/py-configparser/2017-08-30/py-configparser-20170830-svn.tgz">
; 8.25KB
==================================================
8,452 bytes in 0.00 seconds (4126.95KB/sec)
; Loading "py-configparser"
[package org.mapcar.parse-number].................
[package py-configparser].....
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "sqlite":
  Load 3 ASDF systems:
    asdf cffi iterate
  Install 1 Quicklisp release:
    cl-sqlite
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-sqlite/2019-08-13/cl-sqlite-20190813-git.tgz">
; 14.29KB
==================================================
14,632 bytes in 0.00 seconds (14289.06KB/sec)
; Loading "sqlite"
[package sqlite-ffi]..............................
[package sqlite.cache]............................
[package sqlite]......................
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "trivial-backtrace":
  Load 1 ASDF system:
    asdf
  Install 1 Quicklisp release:
    trivial-backtrace
; Fetching #<URL "http://beta.quicklisp.org/archive/trivial-backtrace/2020-06-10/trivial-backtrace-20200610-git.tgz">
; 9.01KB
==================================================
9,230 bytes in 0.00 seconds (9013.67KB/sec)
; Loading "trivial-backtrace"
[package trivial-backtrace]..
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "cl-markdown":
  Load 2 ASDF systems:
    asdf cl-ppcre
  Install 7 Quicklisp releases:
    anaphora asdf-system-connections cl-containers
    cl-markdown dynamic-classes metabang-bind
    metatilities-base
; Fetching #<URL "http://beta.quicklisp.org/archive/metabang-bind/2020-02-18/metabang-bind-20200218-git.tgz">
; 22.01KB
==================================================
22,543 bytes in 0.00 seconds (22014.65KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/dynamic-classes/2013-01-28/dynamic-classes-20130128-git.tgz">
; 6.98KB
==================================================
7,148 bytes in 0.00 seconds (6980.47KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/metatilities-base/2019-12-27/metatilities-base-20191227-git.tgz">
; 68.99KB
==================================================
70,648 bytes in 0.04 seconds (1769.03KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/asdf-system-connections/2017-01-24/asdf-system-connections-20170124-git.tgz">
; 4.82KB
==================================================
4,933 bytes in 0.00 seconds (4817.38KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-containers/2020-04-27/cl-containers-20200427-git.tgz">
; 226.04KB
==================================================
231,463 bytes in 0.04 seconds (5513.12KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/anaphora/2019-10-07/anaphora-20191007-git.tgz">
; 5.97KB
==================================================
6,114 bytes in 0.00 seconds (5970.70KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-markdown/2019-12-27/cl-markdown-20191227-git.tgz">
; 71.70KB
==================================================
73,424 bytes in 0.01 seconds (5515.62KB/sec)
; Loading "cl-markdown"
..................................................
[package metabang.utilities]......................
..................................................
[package metabang.bind]...........................
[package metabang.bind.developer].................
..................................................
[package metabang-dynamic-classes]................
[package metabang.cl-containers]..................
..................................................
[package anaphora]................................
[package anaphora-basic]..........................
[package anaphora-symbol].........................
[package cl-markdown].............................
[package cl-markdown-user]........................
...........
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "mssql":
  Load 4 ASDF systems:
    asdf cffi iterate parse-number
  Install 2 Quicklisp releases:
    cl-mssql garbage-pools
; Fetching #<URL "http://beta.quicklisp.org/archive/garbage-pools/2013-07-20/garbage-pools-20130720-git.tgz">
; 43.17KB
==================================================
44,210 bytes in 0.01 seconds (3924.89KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-mssql/2019-08-13/cl-mssql-20190813-git.tgz">
; 15.37KB
==================================================
15,740 bytes in 0.00 seconds (7685.55KB/sec)
; Loading "mssql"
[package garbage-pools]...........................
[package mssql]...........
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "uuid":
  Load 2 ASDF systems:
    asdf ironclad
  Install 2 Quicklisp releases:
    trivial-utf-8 uuid
; Fetching #<URL "http://beta.quicklisp.org/archive/trivial-utf-8/2011-10-01/trivial-utf-8-20111001-darcs.tgz">
; 5.91KB
==================================================
6,055 bytes in 0.00 seconds (5913.09KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/uuid/2020-07-15/uuid-20200715-git.tgz">
; 21.69KB
==================================================
22,210 bytes in 0.00 seconds (21689.45KB/sec)
; Loading "uuid"
[package trivial-utf-8]...........................
[package uuid]....
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "quri":
  Load 4 ASDF systems:
    alexandria asdf babel split-sequence
  Install 2 Quicklisp releases:
    cl-utilities quri
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-utilities/2010-10-06/cl-utilities-1.2.4.tgz">
; 22.46KB
==================================================
22,998 bytes in 0.00 seconds (22458.98KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/quri/2020-06-10/quri-20200610-git.tgz">
; 68.10KB
==================================================
69,734 bytes in 0.03 seconds (2619.22KB/sec)
; Loading "quri"
[package cl-utilities]............................
[package quri.port]...............................
[package quri.uri]................................
[package quri.encode].............................
[package quri.error]..............................
[package quri.util]...............................
[package quri.decode].............................
[package quri.uri.ftp]............................
[package quri.uri.http]...........................
[package quri.uri.ldap]...........................
[package quri.uri.file]...........................
[package quri.etld]...............................
[package quri.domain].............................
[package quri.parser].............................
..................................................
[package quri]..
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "cl-mustache":
  Load 2 ASDF systems:
    asdf uiop
  Install 1 Quicklisp release:
    cl-mustache
; Fetching #<URL "http://beta.quicklisp.org/archive/cl-mustache/2020-03-25/cl-mustache-20200325-git.tgz">
; 15.94KB
==================================================
16,325 bytes in 0.00 seconds (15942.38KB/sec)
; Loading "cl-mustache"
[package mustache]........
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "yason":
  Load 3 ASDF systems:
    alexandria asdf trivial-gray-streams
  Install 1 Quicklisp release:
    yason
; Fetching #<URL "http://beta.quicklisp.org/archive/yason/2019-12-27/yason-v0.7.8.tgz">
; 27.26KB
==================================================
27,918 bytes in 0.00 seconds (13631.84KB/sec)
; Loading "yason"
[package yason]...................
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
To load "zs3":
  Load 8 ASDF systems:
    alexandria asdf babel cl-base64 drakma ironclad puri
    trivial-gray-streams
  Install 3 Quicklisp releases:
    closure-common cxml zs3
; Fetching #<URL "http://beta.quicklisp.org/archive/closure-common/2018-10-18/closure-common-20181018-git.tgz">
; 27.18KB
==================================================
27,833 bytes in 0.00 seconds (27180.66KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/cxml/2020-06-10/cxml-20200610-git.tgz">
; 152.12KB
==================================================
155,775 bytes in 0.03 seconds (5433.00KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/zs3/2019-10-07/zs3-1.3.3.tgz">
; 55.81KB
==================================================
57,149 bytes in 0.01 seconds (5073.60KB/sec)
; Loading "zs3"
;;; Checking for wide character support... yes, using code points.
;;; Checking for wide character support... yes, using code points.
;;; Building Closure with CHARACTER RUNES
[package runes]...................................
[package utf8-runes]..............................
[package runes-encoding]..........................
[package hax].....................................
[package cxml]....................................
[package sax].....................................
..................................................
..................................................
[package cxml-xmls]...............................
[package dom].....................................
[package rune-dom]................................
[package klacks]..................................
[package zs3]...................................
To load "pgloader":
  Load 1 ASDF system:
    pgloader
; Loading "pgloader"
[package simple-date].............................
[package simple-date-cl-postgres-glue]............
[package pgloader.params].........................
[package pgloader.transforms].....................
[package pgloader.logs]...........................
[package pgloader.quoting]........................
[package pgloader.catalog]........................
[package pgloader.state]..........................
[package pgloader.monitor]........................
[package pgloader.queries]........................
[package pgloader.citus]..........................
[package pgloader.utils]..........................
[package pgloader.archive]........................
[package pgloader.parse-date].....................
[package pgloader.connection].....................
[package pgloader.pgsql]..........................
[package pgloader.sources]........................
[package pgloader.pgcopy].........................
[package pgloader.load]...........................
[package pgloader.ini]............................
[package pgloader.sql]............................
[package pgloader.source.csv].....................
[package pgloader.source.fixed]...................
[package pgloader.source.copy]....................
[package pgloader.source.ixf].....................
[package pgloader.source.db3].....................
[package pgloader.source.mysql]...................
[package pgloader.source.pgsql]...................
[package pgloader.source.sqlite]..................
[package pgloader.source.mssql]...................
[package pgloader.source.mssql.index-filter]......
[package pgloader.syslog].........................
[package pgloader.parser].........................
[package pgloader]................................
[package pgloader.user-symbols]...................
..................................................
..................................................
..................................................
..................................................
...........
Segmentation fault
pi@raspberrypi:~/pgloader $
Ahsoka commented 4 years ago

My initial interest and need for pgloader has faded as I have used another solution in place of pgloader. However, if you guys/gals are still interested in solving the issue I would still be willing to keep trying to make it work. Just let me know my next instructions in debugging.

phoe commented 4 years ago

I sadly have no idea how to debug this one further. Perhaps Clozure/ccl will be able to provide some more support, but I can't promise that.

jesusAlvSoto commented 3 years ago

@Ahsoka, what solution are you using now?

Ahsoka commented 3 years ago

@jesusAlvSoto My use case was to transfer all my data from an SQLite database to a Postgres database on my Raspberry Pi. However, due to the mounting complexity and infeasibility of using pgloader on the Raspberry Pi I wrote a small script, specifically for my use case, to transfer the data which can be seen here.

marshallm94 commented 2 weeks ago

@dimitri do you know of an instance where this has worked on a Raspberry Pi?

I started down the same path as @Ahsoka, but once things looked like they were going to go a similar way, I stopped...