phantomics / april

The APL programming language (a subset thereof) compiling to Common Lisp.
Apache License 2.0
597 stars 31 forks source link

I use april-c to bind varible,however Cannot save core with multiple threads running. #266

Closed secwang closed 1 year ago

secwang commented 1 year ago

(sb-ext:save-lisp-and-die #p"my-app-binary" :toplevel #'b:run :executable t)

debugger invoked on a SB-IMPL::SAVE-WITH-MULTIPLE-THREADS-ERROR in thread

<THREAD "main thread" RUNNING {1001090003}>:

Cannot save core with multiple threads running.

Interactive thread (of current session):

<THREAD "main thread" RUNNING {1001090003}>

Other threads:

<THREAD "april-language-kernel" waiting on: # {1007A909B3}>,

#<THREAD "april-language-kernel" waiting on: # {1007A90663}>,
#<THREAD "april-language-kernel" waiting on: # {1007A916F3}>,
#<THREAD "april-language-kernel" waiting on: # {1007A91053}>,
#<THREAD "april-language-kernel" waiting on: # {1007A90D03}>,
#<THREAD "april-language-kernel" waiting on: # {1007A913A3}>,
#<THREAD "april-language-kernel" waiting on: # {1007A91A43}>,
#<THREAD "april-language-kernel" waiting on: # {1007A91D93}>

See also: The SBCL Manual, Node "Saving a Core Image"

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name): 0: [ABORT] Exit debugger, returning to top level.

(SB-IMPL::DEINIT)

phantomics commented 1 year ago

I tried this, is this similar to what you're doing?

* (asdf:load-system 'april)

* (defvar bla (april:april-c "{⍵×⍳10}" 15))

* (defun dorun () (loop :for i :while t :do (sleep 0.1)))

* (sb-ext:save-lisp-and-die #p"/tmp/lbinary" :toplevel #'pkg:dorun :executable t)

Then I got the sb-impl::save-error but no language telling me that it was because of the multiple threads.

secwang commented 1 year ago

I tried to use april with postmorden. Here is my code. The strange thing here is that if I load this code, it can be executed, so according to my experience, I think it can generally be build into exe.

(load "~/quicklisp/setup.lisp")

(ql:quickload '(postmodern april)  :silent t)

(defpackage :b (:use :cl :postmodern :april) (:export :run))
(in-package :b)

(defparameter cns '("postgres" "postgres" "password" "host"))

(defun list-to-2d-array (list)
  (make-array (list (length list)
                    (length (first list)))
              :initial-contents list))

(defun run ()
    (with-connection cns
      (let* ((a  (query "select * from table ;"))
         (av (list-to-2d-array a)))
    (format t  "~A~%" (april-c "{[a]  +/ a[;6] > 0 }" av))
    (format t "~A~%" (april-c "{[a] +/ a[;6] < 0 }" av))
    (format t "~:d~%" (round (april-c "{[a] +/ a[;7] }" av)) )
    )))
secwang commented 1 year ago

I regenerate error by follow code.

(asdf:load-system 'april)
(defvar bla (april:april-c "{⍵×⍳10}" 15))
(defun dorun () bla)  
(sb-ext:save-lisp-and-die "app" :toplevel #'dorun :executable t)

debugger invoked on a SB-IMPL::SAVE-WITH-MULTIPLE-THREADS-ERROR in thread

<THREAD "main thread" RUNNING {10044601E3}>:

Cannot save core with multiple threads running.

Interactive thread (of current session):

<THREAD "main thread" RUNNING {10044601E3}>

Other threads:

<THREAD "april-language-kernel" waiting on: # {1002DE89B3}>,

#<THREAD "april-language-kernel" waiting on: # {1002DE8663}>,
#<THREAD "april-language-kernel" waiting on: # {1002DE96F3}>,
#<THREAD "april-language-kernel" waiting on: # {1002DE9053}>,
#<THREAD "april-language-kernel" waiting on: # {1002DE8D03}>,
#<THREAD "april-language-kernel" waiting on: # {1002DE93A3}>,
#<THREAD "april-language-kernel" waiting on: # {1002DE9A43}>,
#<THREAD "april-language-kernel" waiting on: # {1002DE9D93}>

See also: The SBCL Manual, Node "Saving a Core Image"

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name): 0: [ABORT] Exit debugger, returning to top level.

SBCL 2.2.11, On OS X. Install april from quicklisp locally dir.

phantomics commented 1 year ago

I've been trying this myself but on Linux kernel 5.4.0-17.1 with SBCL 2.1.7 I just get:

Could not save core.
   [Condition of type SB-IMPL::SAVE-ERROR]

I can try updating the SBCL version and seeing if that works.

Also,. can you try running this as the last line in your test instead of the save-lisp-and-die and see what it returns?

(loop :for worker :across (lparallel.kernel::workers lparallel::*kernel*) :collect (lparallel.kernel::running-category worker))

This will tell you if any threads in the system are busy; if not it should just return a list of NILs.

secwang commented 1 year ago

run with repl

(asdf:load-system 'april)
(defvar bla (april:april-c "{⍵×⍳10}" 15))
(defun dorun () bla) 
(loop :for worker :across (lparallel.kernel::workers lparallel::*kernel*) :collect (lparallel.kernel::running-category worker))
* BLA
* DORUN
* (NIL NIL NIL NIL NIL NIL NIL NIL)

in addition, my mac sbcl version is 2.3.0

phantomics commented 1 year ago

Thanks, looks like nothing's out of the ordinary in your threading kernel. I'll check into possible issues with SBCL and lparallel; for some reason I get the same Could not save core. error even if I try to save a bare system state just after starting up without loading April or any other package.

secwang commented 1 year ago

I suggest we can try the sbcl2.30 test? In my limited experience of using sbcl, this kind of failure to save to executable is relatively rare.

phantomics commented 1 year ago

I'm building 2.3.0 now, I'll see how it handles saving binaries.

phantomics commented 1 year ago

2.3.0 is built and I'm seeing the same error you do. Building binaries without April loaded is working. I'll see what I can find from this point.

secwang commented 1 year ago

good job

On Mon, Jan 9, 2023 at 13:05 Andrew Sengul @.***> wrote:

2.3.0 is built and I'm seeing the same error you do. Building binaries without April loaded is working. I'll see what I can find from this point.

— Reply to this email directly, view it on GitHub https://github.com/phantomics/april/issues/266#issuecomment-1375111286, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABK2522BD2SM5USASVXRXWDWROMCZANCNFSM6AAAAAATK5JEVM . You are receiving this because you authored the thread.Message ID: @.***>

phantomics commented 1 year ago

Try putting this in a file and loading it:

(asdf:load-system 'april)
(defvar bla (april:april-c "{⍵×⍳10}" 15))
(defun dorun () bla)
(lparallel::end-kernel :wait t)
(sb-ext:save-lisp-and-die "/tmp/app" :toplevel #'dorun :executable t)

The (end-kernel) will shut down all lparallel workers. This won't work if you do it from a REPL because there will be another kernel with Swank-related tasks still running.

secwang commented 1 year ago

It works , great work!

phantomics commented 1 year ago

Great, I'm interested to hear more about your April use case if possible. Let me know if there are any further issues.

secwang commented 1 year ago

At present, it is not of practical use. I'm learning apl/april/j/k and rewriting the previous statistical scripts with them by the way. I want to find a balance between writing speed and running speed, so that it can be used for larger tasks. At present, let's take a look at the design of april and use the cl library to do things. According to the input, it is a very good design, and I like it very much. I have used cl before. On Mon, Jan 16, 2023 at 18:26 Andrew Sengul @.***> wrote:

Great, I'm interested to hear more about your April use case if possible. Let me know if there are any further issues.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you modified the open/close state.[image]Message ID: @.***>

phantomics commented 1 year ago

Interesting. If you encounter any situations where April seems particularly slow let me know, there are places where significant optimization is possible but it's a matter of finding these cases in order to make improvements.

secwang commented 1 year ago

Under my test, the same task, the compiled runtime is almost twice longer that of dyalogapl. If it is to explain execution, the quickload of cl will take more time. To be honest, I haven't found out why april is slow. The current conclusion is like this.

However, because I am familiar with cl, april is actually the most convenient apl to process input and output and refer to other array languge. The same task, in dyalogapl and k/q, needs to be very painfully built from the bottom and try various environmental problems. I think even if april is not the fastest apl, it is actually a space in engineering.

Another problem for me is that I still need time to learn apl. Personally, I think dyalog actually provides very good documentation and entry-level environment and development support because of their revenue models. This is a good place to learn apl. This may be a very difficult place for april as open source software. In fact, I don't think there are many people familiar with common lisp and array language in my background. April may need to provide more direct repl. Of course, this is a very good project. Thank you. Cheers.

On the other point, I think the reason why common lisp is not so popular is that people always have to learn emacs. Learning the learning curve of lisp and emacs at the same time is crazy. Of course, emacs is a good environment, but it is not short/simple enough. A short enough environment, similar to kdb+ repl, I think it makes sense to attract novices.

phantomics commented 1 year ago

April now has a REPL mode for Emacs, see: https://github.com/phantomics/april/tree/master/aprepl

Let me know if the installation instructions there make sense. Once it's installed, you can enter M-x aprepl to enter the REPL. This is still alpha software, hence it's not mentioned in the main README.

April's core lexical functions and operators are almost exactly the same as Dyalog's so you can use Dyalog's user manual as a reference for April. This section of the README summarizes the main differences between Dyalog and April. April notably lacks Dyalog's control statements like :If but it includes k-style if statements as a more flexible alternative to guards.

I agree with you about the learning curve of Emacs and I have some ideas about how to create a better CL end-user experience. That's a whole different discussion but these videos of mine (https://vimeo.com/237947324) (https://vimeo.com/269495385) may provide some food for thought.

secwang commented 1 year ago

I noticed aprepl, and I just don't want to open my emacs. You have done great work.

On Tue, Jan 17, 2023 at 11:12 AM Andrew Sengul @.***> wrote:

April now has a REPL mode for Emacs, see: https://github.com/phantomics/april/tree/master/aprepl

Let me know if the installation instructions there make sense. Once it's installed, you can enter M-x aprepl to enter the REPL. This is still alpha software, hence it's not mentioned in the main README.

April's core lexical functions and operators are almost exactly the same as Dyalog's so you can use Dyalog's user manual as a reference for April. This section of the README https://github.com/phantomics/april#unique-language-features-in-april summarizes the main differences between Dyalog and April. April notably lacks Dyalog's control statements like :If but it includes k-style if statements as a more flexible alternative to guards.

I agree with you about the learning curve of Emacs and I have some ideas about how to create a better CL end-user experience. That's a whole different discussion but these videos of mine (https://vimeo.com/237947324) (https://vimeo.com/269495385) may provide some food for thought.

— Reply to this email directly, view it on GitHub https://github.com/phantomics/april/issues/266#issuecomment-1384780307, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABK2526AUJYYU3EQKBLJZELWSYEYVANCNFSM6AAAAAATK5JEVM . You are receiving this because you modified the open/close state.Message ID: @.***>

justin2004 commented 1 year ago

@secwang I use April from a REPL in vim. Let me know if you'd like the details on how I do that.