Upload file size limit

[jgarte]

Hi, is there a limit on file size by default in ningle (or (c)lack?)?

Flask has a feature like this. I'm wondering if I can configure something similar in ningle or (c)lack.

I tried uploading a relatively large file (334.2MB) but I got this error:

#<PACKAGE "NINGLE-UPLOAD">
NINGLE-UPLOAD(5): (start :port 8899)

Hunchentoot server is started.
Listening on localhost:8899.
#S(CLACK.HANDLER::HANDLER
   :SERVER :HUNCHENTOOT
   :ACCEPTOR #<SB-THREAD:THREAD "clack-handler-hunchentoot" RUNNING
                {10047AC0F3}>)
NINGLE-UPLOAD(6): Heap exhausted during allocation: 293339136 bytes available, 334184704 requested.
Gen  Boxed    Raw   Code  Mixed  LgRaw LgCode  LgMix  Pin       Alloc     Waste        Trig   Dirty GCs Mem-age
 0       0      0      0    318      0      0      0    0     5052672    157440    10737418       -   0  0.0000
 1    1070    573      7    210  40869      0     95 40852   700446352   1182064   711183770   41480   2  0.0000
 2       0      0      0      0      0      0      0    0           0         0     2000000       0   0  0.0000
 3       0      0      0      0      0      0      0    0           0         0     2000000       0   0  0.0000
 4       0      0      0      0      0      0      0    0           0         0     2000000       0   0  0.0000
 5       0      0      0      0      0      0      0    0           0         0     2000000       0   0  0.0000
 6     789    325      3     41     53      0    123    0    21262448    593808     2000000     143   0  0.0000
           Total bytes allocated    =     726761472
           Dynamic-space-size bytes =    1073741824
GC control variables:
   *GC-INHIBIT* = false
   *GC-PENDING* = true
   *STOP-FOR-GC-PENDING* = false
[2022-02-16 00:14:31 [ERROR]] Error while processing connection: The condition The condition Heap exhausted (no more space for allocation).
293339136 bytes available, 334184704 requested.

PROCEED WITH CAUTION. occurred with errno: 0. occurred with errno: 0.
Heap exhausted during allocation: 293339136 bytes available, 334184720 requested.
Gen  Boxed    Raw   Code  Mixed  LgRaw LgCode  LgMix  Pin       Alloc     Waste        Trig   Dirty GCs Mem-age
 0       0      0      0    103      0      0      0    0     1596592     90960    10737418       -   0  0.0000
 1    1054    568      7    208  40869      0     95 40852   700139456   1112128   710876874   41509   3  0.0000
 2       0      0      0      0      0      0      0    0           0         0     2000000       0   0  0.0000
 3       0      0      0      0      0      0      0    0           0         0     2000000       0   0  0.0000
 4       0      0      0      0      0      0      0    0           0         0     2000000       0   0  0.0000
 5       0      0      0      0      0      0      0    0           0         0     2000000       0   0  0.0000
 6     789    325      3     41     53      0    123    0    21262448    593808     2000000     144   0  0.0000
           Total bytes allocated    =     722998496
           Dynamic-space-size bytes =    1073741824
GC control variables:
   *GC-INHIBIT* = false
   *GC-PENDING* = true
   *STOP-FOR-GC-PENDING* = false
[2022-02-16 00:14:32 [ERROR]] Error while processing connection: The condition The condition Heap exhausted (no more space for allocation).
293339136 bytes available, 334184720 requested.

PROCEED WITH CAUTION. occurred with errno: 0. occurred with errno: 0.

Here is the upload form code:

;; License: MIT

(uiop:define-package #:ningle-upload
  (:use #:cl)
  (:import-from #:cl-fad)
  (:import-from #:ningle)
  (:import-from #:spinneret)
  (:import-from #:log4cl))

(in-package #:ningle-upload)

(defvar *app* (make-instance 'ningle:app))

(defvar *server* nil)

(defparameter *downloads-dir* #P"/tmp/downloads/")

(setf (ningle:route *app* "/")
      (lambda (params)
        (declare (ignore params))
        (spinneret:with-html-string
          (:h1 "Ningle Uploader Example")

          (:h2 "Already Uploaded")
          (let ((files
                  (when (probe-file *downloads-dir*)
                    (cl-fad:list-directory *downloads-dir*))))
            (if files
                (:ul
                 (loop for file in files
                       do (:li (:p (file-namestring file)))))
                (:p "There is no files yet.")))

          (:h2 "Add More")
          (:form :method "POST"
                 :action "/"
                 :enctype "multipart/form-data"
                 (:input :type "file"
                         :multiple t
                         :name "upload")
                 (:input :type "submit")))))

(setf (ningle:route *app* "/" :method :POST)
      (lambda (params)
        ;; In case of multiple files, params will have a multiple "upload"
        ;; entries:
        ;;
        ;; (("upload" #<FLEXI-STREAMS::VECTOR-INPUT-STREAM {700821F9F3}>
        ;;            "file1.png" "image/png")
        ;;  ("upload" #<FLEXI-STREAMS::VECTOR-INPUT-STREAM {7008223413}>
        ;;            "file2.png" "image/png")
        ;;  ("upload" #<FLEXI-STREAMS::VECTOR-INPUT-STREAM {7008223F23}>
        ;;            "file3.png" "image/png"))
        ;; 
        ;; And we need to process them one by one:
        (loop for row in params
              for name = (first row)
              when (string= name "upload")
                ;; Here stream will have type FLEXI-STREAMS::VECTOR-INPUT-STREAM
                do (destructuring-bind (stream filename content-type)
                       (rest row)
                     (when stream
                       (log:info "Accepting upload." filename content-type)
                       (let ((full-path (merge-pathnames filename *downloads-dir*)))
                         (ensure-directories-exist full-path)
                         (uiop:slurp-input-stream full-path stream))))
                   ;; read-line
                   ;; read-stream function
                   ;; will give a single stream ended by newline

              finally
                 ;; Redirecting user back to the list of files:
                 (setf (lack.response:response-status ningle:*response*)
                       302
                       (lack.response:response-headers ningle:*response*)
                       (append (lack.response:response-headers ningle:*response*)
                               (list :location "/"))))))

(defun start (&key (interface "localhost")
                (port 8080))
  (setf *server*
        (clack:clackup *app*
                       :address interface
                       :port port)))

[fukamachi]

There's no limitation of ningle/lack/clack, and also no features to limit. It means that it's up to what webserver to use.

Perhaps Hunchentoot handles POST content naively, and it puts all data on memory.

[jgarte]

@fukamachi

Thanks for the reply.

Do you happen to know of a CL web server that doesn't handle POST content naively?

[fukamachi]

Woo can be an option.

[jgarte]

Woo can be an option.

Woohoo, that worked! Thanks.