Now QL supports distributions with slash in their names

[svetlyak40wt]

This makes it possible to use custom distributions, hosted at Ultralisp.org. These custom distributions have names of form username/dist-name and distinfo.txt file looks like this:

name: svetlyak40wt/my-forks
version: 20210112065000
distinfo-subscription-url: http://dist.ultralisp.org/svetlyak40wt/my-forks.txt
distinfo-template-url: http://dist.ultralisp.org/svetlyak40wt/my-forks/{{version}}/distinfo.txt
release-index-url: http://dist.ultralisp.org/svetlyak40wt/my-forks/20210112065000/releases.txt
system-index-url: http://dist.ultralisp.org/svetlyak40wt/my-forks/20210112065000/systems.txt

Such names also make sense as a URL's part. That is why I think it is a good idea to support them in QuickLisp client.

[svetlyak40wt]

@xach please, review this pull.

[quicklisp]

The first version was better - I do not use UIOP code in Quicklisp.

[svetlyak40wt]

Ok, @quicklisp I've removed the last commit from the pull.

[svetlyak40wt]

Great! Thank you!

[quicklisp]

This has been published and is available with (ql:update-client) or by installing Quicklisp fresh.

[quicklisp]

I have to revert this change, because it is too slow to use wild-inferiors in this situation.

[svetlyak40wt]

@quicklisp I didn't notice any slowdown with this patch. Could you please explain or give me a hint how to reproduce the problem?

I'll try to fix it and will make another pull request. Probably custom search function will be faster than (directory (qmerge "dists/**/distinfo.txt").

[svetlyak40wt]

On my system listing of a pathname with ** wildcard takes ~150ms:

UPLOAD> (time (directory (ql:qmerge "dists/**/distinfo.txt")))
Evaluation took:
  0.149 seconds of real time
  0.147717 seconds of total run time (0.062139 user, 0.085578 system)
  [ Run times consist of 0.026 seconds GC time, and 0.122 seconds non-GC time. ]
  99.33% CPU
  329,009,080 processor cycles
  8 page faults
  16,175,760 bytes consed

(#P"/Users/art/projects/lisp/cl-info/.qlot/dists/mgl-pax/distinfo.txt"
 #P"/Users/art/projects/lisp/cl-info/.qlot/dists/quicklisp/distinfo.txt"
 #P"/Users/art/projects/lisp/cl-info/.qlot/dists/ultralisp/distinfo.txt")

Whereas old version lists them in 1ms (but does not supports nested dirs:

UPLOAD> (time (directory (ql:qmerge "dists/*/distinfo.txt")))
Evaluation took:
  0.001 seconds of real time
  0.001201 seconds of total run time (0.000364 user, 0.000837 system)
  100.00% CPU
  2,647,030 processor cycles
  65,296 bytes consed

(#P"/Users/art/projects/lisp/cl-info/.qlot/dists/mgl-pax/distinfo.txt"
 #P"/Users/art/projects/lisp/cl-info/.qlot/dists/quicklisp/distinfo.txt"
 #P"/Users/art/projects/lisp/cl-info/.qlot/dists/ultralisp/distinfo.txt")

Are these 150ms so critical? Is it called in a loop (I didn't find such place).

[svetlyak40wt]

Oh, seems it depends on how many nested files in the dists dir. I've got 150ms on the dists with ~4000 files. That is bad :(

[svetlyak40wt]

Zach, I was able to build this prototype which is as fast as a previous solution but is able to work with nested subdirectories:

(defun dists-dir ()
  (ql:qmerge (make-pathname :directory (list :relative "dists"))))

(defun find-distinfo (paths)
  (loop for path in paths
        for name = (pathname-name path)
        for type = (pathname-type path)
          thereis (and name
                       type
                       (string-equal name "distinfo")
                       (string-equal type "txt")
                       path)))

(defun find-distinfo-files ()
  "This function searches all distinfo.txt files in the dist/ subdirectory.

   It goes deeper until finds distinfo.txt file and thus works much
   faster than (directory (ql:qmerge \"dists/*/distinfo.txt\")) which
   has to traverse whole file tree."
  (loop with results = nil
        with queue = (list (dists-dir))
        for path = (car queue) then (car queue)
        while path
        do (let* ((listing (ql-impl-util::directory-entries path))
                  (distinfo (find-distinfo listing)))
             (if distinfo
                 (push distinfo results)
                 (setf queue
                       (append queue
                               (remove-if-not #'ql-impl-util::directoryp
                                              listing))))
             (setf queue
                   (cdr queue)))
        finally (return results)))

Here is its performance test:

TEST-QL> (time (find-distinfo-files))
Evaluation took:
  0.001 seconds of real time
  0.001957 seconds of total run time (0.000777 user, 0.001180 system)
  200.00% CPU
  4,197,446 processor cycles
  98,064 bytes consed

(#P"/Users/art/projects/lisp/cl-info/.qlot/dists/svetlyak40wt/mgl-pax/distinfo.txt"
 #P"/Users/art/projects/lisp/cl-info/.qlot/dists/ultralisp/distinfo.txt"
 #P"/Users/art/projects/lisp/cl-info/.qlot/dists/quicklisp/distinfo.txt")

If this code is OK, I'll fix my branch and will make another pull-request. Ok?