=jingle= is [[https://github.com/fukamachi/ningle][ningle]], but with bells and whistles.
Requirements
Installation
=jingle= is not yet in Quicklisp, so in order to install it you will need to clone the repo and add it to your [[https://www.quicklisp.org/beta/faq.html][Quicklisp local-projects]].
cd ~/quicklisp/local-projects git clone https://github.com/dnaeon/cl-jingle.git
The =JINGLE.DEMO= system provides a ready-to-run example REST API, which comes with an [[https://swagger.io/specification/][OpenAPI 3.x spec]], [[https://swagger.io/tools/swagger-ui/][Swagger UI]], and a command-line interface app built with [[https://github.com/dnaeon/clingon][clingon]].
In order to build the demo application, simply execute this command.
make demo
This will build the =jingle-demo= app, which you can find in the =bin/= directory.
In order to start the HTTP server, execute the following command.
bin/jingle-demo serve
Once the HTTP server is up and running navigate to [[http://localhost:5000/api/docs/][http://localhost:5000/api/docs/]] which should take you to the Swagger UI.
[[./images/jingle-swagger-ui.png]]
Make sure to check the various sub-commands provided by =jingle-demo=, which allow you to interface with the REST APIs.
You can also run the demo app in Docker. First, build the image.
make demo-docker
Once the image is built you can start up the service by executing the following command.
docker run -p 5000:5000 cl-jingle:latest serve --address 0.0.0.0
You can also view a short demo of the command-line interface application, which interfaces with the REST API here.
[[./images/jingle-demo.gif]]
Start up your REPL and load the =JINGLE= system.
CL-USER> (ql:quickload :jingle) To load "jingle": Load 1 ASDF system: jingle ; Loading "jingle" .................................................. [package jingle.core]............................. [package jingle]
(:JINGLE)
First thing we need to do is to create a new =JINGLE:APP= instance.
CL-USER> (defparameter app (jingle:make-app)) APP
When creating a new instance of =JINGLE:APP= you can provide additional keyword args, which specify what HTTP server to use, address to bind to, the port to listen on, middlewares, etc..
A very simple HTTP handler, which returns =Hello, World!= looks like this.
(defun hello-handler (params) (declare (ignore params)) "Hello, World!")
The following is an example of an HTTP handler which echoes back the payload you send to it.
(defun echo-handler (params) "A simple handler which echoes back the payload you send to it" (declare (ignore params)) (jingle:set-response-header :content-type (jingle:request-content-type jingle:request)) (jingle:set-response-header :content-length (jingle:request-content-length jingle:request)) (maphash (lambda (k v) (jingle:set-response-header k v)) (jingle:request-headers jingle:request)) (jingle:request-content jingle:request))
Next thing we need to do is register our handlers.
CL-USER> (setf (jingle:route app "/hello") #'hello-handler) CL-USER> (setf (jingle:route app "/echo" :method :post) #'echo-handler)
And now we can start the app.
CL-USER> (jingle:start app)
Trying out our endpoints using =curl(1)= gives us this result.
$ curl -vvv --get http://localhost:5000/hello
GET /hello HTTP/1.1 Host: localhost:5000 User-Agent: curl/7.86.0 Accept: /
And this is our echo handler.
$ curl -v -s --data '{"foo": "bar", "baz": "42"}' -H "My-Header: SomeValue" -H "Content-Type: application/json" -X POST http://localhost:5000/echo
POST /echo HTTP/1.1 Host: localhost:5000 User-Agent: curl/7.86.0 Accept: / My-Header: SomeValue Content-Type: application/json Content-Length: 27
In order to stop the application, evaluate the following expression.
CL-USER> (jingle:stop app)
** Handlers
Handlers are regular [[https://github.com/fukamachi/ningle][ningle]] routes, which accept a single argument, representing the request parameters.
** Environment
=jingle= exports the special variable =JINGLE:ENV= which is dynamically bound to the request environment of [[https://github.com/fukamachi/lack][Lack]]. You can query the environment directly from =jingle= and don't have to worry about where the environment is coming from.
** Headers
=jingle= provides the =JINGLE:SET-RESPONSE-HEADER= function for setting up HTTP response headers.
A simple handler which sets the =Content-Type= header to =text/plain= looks like this.
(defun hello (params) (declare (ignore params)) (jingle:set-response-header :content-type "text/plain") "Hello, World!")
Other useful functions which operate on HTTP headers are =JINGLE:GET-REQUEST-HEADER= and =JINGLE:GET-RESPONSE-HEADER=, which retrieve the value of the HTTP header associated with the request and response respectively.
** Status Codes
The =JINGLE:SET-RESPONSE-STATUS= function sets the Status Code for the HTTP Response.
(defun foo-handler (params) (declare (ignore params)) (jingle:set-response-status :accepted) "Task accepted")
Arguments passed to =JINGLE:SET-RESPONSE-STATUS= may be a number (e.g. =400=), a keyword (e.g. =:bad-request=), or a string (e.g. =Bad Request=) of the status code. The following three expressions are equivalent, and they all set the HTTP Status Code to =400 (Bad Request)=.
(jingle:set-response-status 400) (jingle:set-response-status :bad-request) (jingle:set-response-status "Bad Request")
Another useful function operating on HTTP Status Codes is =JINGLE:EXPLAIN-STATUS-CODE=.
CL-USER> (jingle:explain-status-code 400) "Bad Request" CL-USER> (jingle:explain-status-code :bad-request) "Bad Request"
=JINGLE:STATUS-CODE-KIND= returns the kind of the HTTP Status Code as classified by [[https://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml][IANA]], e.g.
CL-USER> (jingle:status-code-kind 400) :CLIENT-ERROR CL-USER> (jingle:status-code-kind :unauthorized) :CLIENT-ERROR CL-USER> (jingle:status-code-kind :internal-server-error) :SERVER-ERROR CL-USER> (jingle:status-code-kind :moved-permanently) :REDIRECTION CL-USER> (jingle:status-code-kind 100) :INFORMATIONAL CL-USER> (jingle:status-code-kind "Accepted") :SUCCESS
Other HTTP status code predicates you may find useful are =JINGLE:INFORMATIONAL-CODE-P=, =JINGLE:SUCCESS-CODE-P=, =JINGLE:REDIRECTION-CODE-P=, =JINGLE:CLIENT-ERROR-CODE-P= and =JINGLE:SERVER-ERROR-CODE-P=.
** Static Resources
Static resources can be served by adding them using =JINGLE:STATIC-PATH= method, e.g.
(jingle:static-path app "/static/" "~/public_html/")
You can serve static resources from multiple directories as well. In order to do that simply install them, before you start up the app.
(jingle:static-path app "/static-1/" "/path/to/static-1/") (jingle:static-path app "/static-2/" "/path/to/static-2/") (jingle:static-path app "/static-3/" "/path/to/static-3/")
** Directory Browser
The =JINGLE:SERVE-DIRECTORY= method installs a middleware which allows you to browse the contents of a given path. For example the following code exposes the =~/Documents= and =~/Projects= directories.
(jingle:serve-directory app "/docs" "~/Documents") (jingle:serve-directory app "/projects" "~/Projects")
When accessing the directories from the browser make sure to add a slash at the end of the paths. For example the above directories will have to accessed at http://localhost:5000/docs/ and http://localhost:5000/projects/ respectively, if you are using the default HTTP port when starting up the app.
** Middlewares
You can use regular [[https://github.com/fukamachi/lack#middlewares][Lack middlewares]] with =jingle= as well. Simply install them using the =JINGLE:INSTALL-MIDDLEWARE= method.
The following simple middleware pushes a new property to the request environment, which can be queried by the HTTP handlers.
First, implement the middleware.
(defun my-middleware (app) "A custom middleware which pushes a new property to the request environment and exposes it to HTTP handlers." (lambda (env) (setf (getf env :my-middleware/message) "my middleware message") (funcall app env)))
Then we create a =JINGLE:APP= and install it.
CL-USER> (defparameter app (jingle:make-app)) CL-USER> (jingle:install-middleware app #'my-middleware)
An example handler which uses the message placed by our middleware may look like this.
(defun my-handler (params) (declare (ignore params)) (jingle:set-response-status :ok) (jingle:set-response-header :content-type "text/plain") (getf jingle:env :my-middleware/message))
Finally we have to register our handler and start the app.
CL-USER> (setf (jingle:route app "/my-middleware") #'my-handler) CL-USER> (jingle:start app)
Trying it out using =curl(1)= returns the following response.
$ curl -vvv --get http://localhost:5000/my-middleware
GET /my-middleware HTTP/1.1 Host: localhost:5000 User-Agent: curl/7.86.0 Accept: /
Here's an example which uses Lack's =accesslog= middleware and how to use it with =jingle=. First, load the respective system, which provides the middleware, and then simply install it into the =jingle= app.
CL-USER> (ql:quickload :lack-middleware-accesslog) CL-USER> (jingle:install-middleware app lack.middleware.accesslog:lack-middleware-accesslog)
Search for other middlewares you can already use in Quicklisp, e.g.
CL-USER> (ql:system-apropos "lack-middleware")
You can use middlewares to push metadata into the environment for HTTP handlers to use. For example, if your HTTP handlers need to read from and write to a database, you may want to create a middleware, which pushes a =CL-DBI= connection into the environment, so that HTTP handlers can use it, when needed.
In order to clear out all installed middlewares you can use the =JINGLE:CLEAR-MIDDLEWARES= method, e.g.
CL-USER> (jingle:clear-middlewares app)
** Redirects
Redirects in =jingle= are handled by the =JINGLE:REDIRECT= function.
An example HTTP handler which redirects to [[https://lispcookbook.github.io/cl-cookbook/][The Common Lisp Cookbook]] looks like this.
(defun to-the-cookbook (params) (declare (ignore params)) (jingle:redirect "https://lispcookbook.github.io/cl-cookbook/"))
Register the HTTP handler and start the app.
CL-USER> (setf (jingle:route app "/cookbook") #'to-the-cookbook) CL-USER> (jingle:start app)
Navigate to http://localhost:5000/cookbook and you will be automatically redirected.
There is also another way for defining redirects using =JINGLE:REDIRECT-ROUTE=. The following example shows how to install two redirect routes to your =jingle= app, without having to explicitely define the HTTP handlers in advance.
CL-USER> (jingle:redirect-route app "/sbcl" "https://sbcl.org/") CL-USER> (jingle:redirect-route app "/ecl" "https://ecl.common-lisp.dev/")
** Request Parameters
The =JINGLE:GET-REQUEST-PARAM= function may be used within HTTP handlers to get the value associated with a given parameter.
Suppose we have the following example HTTP handler, which returns information about supported products and is exposed via the =/api/v1/product/:name= endpoint.
(defparameter products '((:|id| 1 :|name| "foo") (:|id| 2 :|name| "bar") (:|id| 3 :|name| "baz") (:|id| 4 :|name| "qux") (:|id| 5 :|name| "foo v2") (:|id| 6 :|name| "bar v3") (:|id| 7 :|name| "baz v4") (:|id| 8 :|name| "qux v5")) "The list of our supported products")
(defun find-product-by-name (name) "Finds a product by name" (find name products :key (lambda (item) (getf item :|name|)) :test #'string=))
(defun product-handler (params) "Handles requests for /api/v1/product/:name endpoint" (jingle:set-response-status :ok) (jingle:set-response-header :content-type "application/json") (let* ((name (jingle:get-request-param params :name)) (product (find-product-by-name name))) (if product (jonathan:to-json product) (progn (jingle:set-response-status :not-found) (jonathan:to-json '(:|error| "Product not found"))))))
Register the HTTP handler and start the app.
CL-USER> (setf (jingle:route app "/api/v1/product/:name") #'product-handler) CL-USER> (jingle:start app)
Testing it out with different product names using =curl(1)=.
$ curl -s --get http://localhost:5000/api/v1/product/foo | jq '.' { "id": 1, "name": "foo" }
$ curl -s --get http://localhost:5000/api/v1/product/bar | jq '.' { "id": 2, "name": "bar" }
$ curl -s --get http://localhost:5000/api/v1/product/unknown | jq '.' { "error": "Product not found" }
Another example HTTP handler which returns a list of products in a paginated way, exposed via the =/api/v1/products= endpoint.
(defun take (items from to) "A helper function to return the ITEMS between FROM and TO range" (let* ((len (length items)) (to (if (>= to len) len to))) (if (>= from len) nil (subseq items from to))))
(defun products-handler (params)
"Handles requests for /api/v1/product and returns a page of products"
(jingle:set-response-status :ok)
(jingle:set-response-header :content-type "application/json")
;; Parse the FROM' andTO' query parameters. Use default values of
;; 0 and 5 for the params.
(let ((from (parse-integer (jingle:get-request-param params "from" "0") :junk-allowed t))
(to (parse-integer (jingle:get-request-param params "to" "5") :junk-allowed t)))
(cond
((or (null from) (null to)) (jingle:set-response-status :bad-request) nil) ;; NIL added here for the response body
((or (minusp from) (minusp to)) (jingle:set-response-status :bad-request) nil) ;; NIL added here for the response body
(t (jonathan:to-json (take products from to))))))
Register the new API endpoint.
CL-USER> (setf (jingle:route app "/api/v1/products") #'products-handler)
Testing it out using =curl(1)= with different values for =from= and =to= query params.
$ curl -s --get 'http://localhost:5000/api/v1/products?from=0&to=2' | jq '.' [ { "id": 1, "name": "foo" }, { "id": 2, "name": "bar" } ]
$ curl -s --get 'http://localhost:5000/api/v1/products?from=2&to=4' | jq '.' [ { "id": 3, "name": "baz" }, { "id": 4, "name": "qux" } ]
Another way to retrieve request parameter values is to use the =JINGLE:WITH-REQUEST-PARAMS= macro. The previous example handler can be rewritten this way.
(defun products-handler (params) (jingle:with-json-response (jingle:with-request-params ((from-param "from" "0") (to-param "to" "5")) params ;; Parse the query parameters and make sure we've got good values (let ((from (parse-integer from-param :junk-allowed t)) (to (parse-integer to-param :junk-allowed t))) (cond ((or (null from) (null to)) (jingle:set-response-status :bad-request) nil) ;; NIL added here for the response body ((or (minusp from) (minusp to)) (jingle:set-response-status :bad-request) nil) ;; NIL added here for the response body (t (take products from to)))))))
** Macros
The following helper macros are available in =jingle=.
The =JINGLE:WITH-JSON-RESPONSE= macro sets up various HTTP headers such as =Content-Type= to =application/json= for you and evaluates the body. The last evaluated expression from the body is encoded as a JSON object using =JONATHAN:TO-JSON=.
The following example uses =LOCAL-TIME= and =JONATHAN= systems, so make sure you have them loaded already.
(defclass ping-response () ((message :initarg :message :initform "pong" :reader ping-response-message :documentation "Message to send as part of the response") (timestamp :initarg :timestamp :initform (local-time:now) :reader ping-response-timestamp)) (:documentation "A response sent as part of a PING request"))
(defmethod jonathan:%to-json ((object ping-response)) (jonathan:with-object (jonathan:write-key-value "message" (ping-response-message object)) (jonathan:write-key-value "timestamp" (ping-response-timestamp object))))
(defun ping-handler (params) (declare (ignore params)) (jingle:with-json-response (make-instance 'ping-response)))
Register the HTTP handler and start the app.
CL-USER> (setf (jingle:route app "/api/v1/ping") #'ping-handler) CL-USER> (jingle:start app)
Trying it you should see results similar to the ones below.
$ curl -s --get http://localhost:5000/api/v1/ping | jq '.' { "message": "pong", "timestamp": 1670593969 }
$ curl -s --get http://localhost:5000/api/v1/ping | jq '.' { "message": "pong", "timestamp": 1670593974 }
$ curl -s --get http://localhost:5000/api/v1/ping | jq '.' { "message": "pong", "timestamp": 1670593976 }
The =JINGLE:WITH-REQUEST-PARAMS= macro provides an easy way to bind symbols to request params from within HTTP handlers.
(defun foo-handler (params) (jingle:with-request-params ((foo "foo") (bar "bar")) params ;; Use FOO and BAR params in order to ... ...))
The =JINGLE:WITH-HTML-RESPONSE= is similar to =JINGLE:WITH-JSON-RESPONSE=, but sets up the response with a =Content-Type: text/html; charset=utf-8= header.
** Error Handling
The =JINGLE:BASE-HTTP-ERROR= condition may be used as the base for user-defined conditions.
If a condition is signalled from within HTTP handlers and the condition is a sub-class of =JINGLE:BASE-HTTP-ERROR=, then the =JINGLE:HANDLE-ERROR= method will be invoked.
The purpose of =JINGLE:HANDLE-ERROR= is to handle the error and set up an appropriate HTTP response, which will be returned to the client.
The rest of this section describes how to create and use custom errors for a very simple REST API. The API we will develop provides the following endpoints.
GET /api/v1/product => Returns a list of products (supports from and to query params)
GET /api/v1/product/:name => Returns a product by name, if found
The error responses which we will return to clients would look like this.
{
"error": "
First we will define our =API-ERROR= condition, and then define the =JINGLE:HANDLE-ERROR= method on it, so that we return consistent error responses to our API clients.
(define-condition api-error (jingle:base-http-error) () (:documentation "Represents a condition which will be signalled on API errors"))
(defmethod jingle:handle-error ((error api-error)) "Handles the error and sets up the HTTP error response to be sent to clients" (with-accessors ((code jingle:http-error-code) (body jingle:http-error-body)) error (jingle:set-response-status code) (jingle:set-response-header :content-type "application/json") (jonathan:to-json (list :|error| body))))
Next, we will implement some helper functions that signal common client-error HTTP responses.
(defun throw-not-found-error (message) "Throws a 404 (Not Found) HTTP response" (error 'api-error :code :not-found :body message))
(defun throw-bad-request-error (message) "Throws a 400 (Bad Request) HTTP response" (error 'api-error :code :bad-request :body message))
Having our conditions and error-related functions we will also define another helper function, which will be responsible for parsing HTTP query parameters as integers, which we will use in our handlers.
(defun get-int-param (params name &optional default)
"Gets the NAME parameter from PARAMS and parses it as an integer.
In case of invalid input it will signal a 400 (Bad Request) error"
(let ((raw (jingle:get-request-param params name default)))
(typecase raw
(number raw)
(null (throw-bad-request-error (format nil "missing value for ~A param" name)))
(string (let ((parsed (parse-integer raw :junk-allowed t)))
(unless parsed
(throw-bad-request-error (format nil "invalid value for ~A param" name)))
parsed))
(t (throw-bad-request-error (format nil "unsupported value for ~A param" name))))))
We will be building on top of the /products/ API, which was shown in a previous section. The =PRODUCTS= var will be our "database" in this simple API.
(defparameter products '((:|id| 1 :|name| "foo") (:|id| 2 :|name| "bar") (:|id| 3 :|name| "baz") (:|id| 4 :|name| "qux") (:|id| 5 :|name| "foo v2") (:|id| 6 :|name| "bar v3") (:|id| 7 :|name| "baz v4") (:|id| 8 :|name| "qux v5")) "The list of our supported products")
(defun find-product-by-name (name) "Finds a product by name" (find name products :key (lambda (item) (getf item :|name|)) :test #'string=))
(defun take (items from to) "A helper function to return the ITEMS between FROM and TO range" (let* ((len (length items)) (to (if (>= to len) len to))) (if (>= from len) nil (subseq items from to))))
And these are the actual HTTP handlers, which will accept and handle client requests.
(defun get-product-handler (params) "Handles requests for the /api/v1/product/:name endpoint" (jingle:with-json-response (let* ((name (jingle:get-request-param params :name)) (product (find-product-by-name name))) (unless product (throw-not-found-error "product not found")) product)))
(defun get-products-page-handler (params)
"Handles requests for the /api/v1/product endpoint"
(jingle:with-json-response
(let ((from (get-int-param params "from" 0))
(to (get-int-param params "to" 2)))
(when (or (minusp from) (minusp to))
(throw-bad-request-error "from and to must be positive"))
(take products from to))))
Finally, we will create our =JINGLE:APP=, register our handlers and start serving HTTP requests.
CL-USER> (defparameter app (jingle:make-app)) CL-USER> (setf (jingle:route app "/api/v1/product") #'get-products-page-handler) CL-USER> (setf (jingle:route app "/api/v1/product/:name") #'get-product-handler) CL-USER> (jingle:start app)
Time to test things out.
from and to params$ curl -s --get 'http://localhost:5000/api/v1/product' | jq '.' [ { "id": 1, "name": "foo" }, { "id": 2, "name": "bar" } ]
$ curl -s --get 'http://localhost:5000/api/v1/product?from=2&to=4' | jq '.' [ { "id": 3, "name": "baz" }, { "id": 4, "name": "qux" } ]
$ curl -s --get 'http://localhost:5000/api/v1/product?from=bad-value' | jq '.'
{
"error": "invalid value for from param"
}
$ curl -s --get 'http://localhost:5000/api/v1/product?to=-42' | jq '.'
{
"error": "from and to must be positive"
}
$ curl -s --get 'http://localhost:5000/api/v1/product/foo' | jq '.' { "id": 1, "name": "foo" }
$ curl -s --get 'http://localhost:5000/api/v1/product/unknown' | jq '.' { "error": "product not found" }
Great, things work as expected and our API clients will receive consistent error responses with the proper HTTP status codes set.
** Reverse URLs
When you register routes in your app with names, you can then refer to these routes by their names. This is useful in situations where you need to get the URL for a particular route.
In order to get the URL for a route with a particular name use the =JINGLE:URL-FOR= generic function.
Consider the HTTP handlers we have shown in the previous section of this document.
(defun get-product-handler (params) "Handles requests for the /api/v1/product/:name endpoint" (jingle:with-json-response (let* ((name (jingle:get-request-param params :name)) (product (find-product-by-name name))) (unless product (throw-not-found-error "product not found")) product)))
(defun get-products-page-handler (params)
"Handles requests for the /api/v1/product endpoint"
(jingle:with-json-response
(let ((from (get-int-param params "from" 0))
(to (get-int-param params "to" 2)))
(when (or (minusp from) (minusp to))
(throw-bad-request-error "from and to must be positive"))
(take products from to))))
We can register these handlers and associate them with a name, which we can later refer to.
CL-USER> (defparameter app (jingle:make-app)) CL-USER> (setf (jingle:route app "/api/v1/product" :method :get :identifier "get-products-page")
CL-USER> (setf (jingle:route app "/api/v1/product/:name" :method :get :identifier "get-product-by-name")
Now, we can get the actual URLs for our HTTP handlers by using their names.
CL-USER> (jingle:url-for app "get-product-by-name" :name "foo")
CL-USER> (jingle:url-for app "get-products-page" :|from| 0 :|to| 100)
Resolving URLs using =JINGLE:URL-FOR= is also useful when you are creating test cases for your HTTP handlers. Within your test cases instead of manually constructing the URLs to the respective HTTP handlers you may refer to them by using their names.
Make sure to also check the =JINGLE.DEMO= system, which uses a /handler registry/, which is used for registering the HTTP handlers for a =JINGLE:APP=.
Once you resolve the URL for a particular handler you can construct the final URL, which will contain scheme, hostname, etc.
CL-USER> (jingle:url-for app "get-product-by-name" :name "foo")
CL-USER> (quri:merge-uris * (quri:make-uri :host "example.org" :scheme "https"))
** Testing HTTP Handlers
The =JINGLE:TEST-APP= is a test app meant to be used for test cases. The difference between =JINGLE:TEST-APP= and =JINGLE:APP= is that the test app always binds on =127.0.0.1= and listens on a random port within a given range.
Also, when using =JINGLE:URL-FOR= generic function with a =JINGLE:TEST-APP= the result is a full URL, which contains the scheme, the hostname and the port of the running test HTTP server.
Make sure to check the =JINGLE.DEMO.TEST= system for some examples, which provides the test suite for the demo application.
=jingle= is hosted on [[https://github.com/dnaeon/cl-jingle][Github]]. Please contribute by reporting issues, suggesting features or by sending patches using pull requests.
This project is Open Source and licensed under the [[http://opensource.org/licenses/BSD-2-Clause][BSD License]].
Authors
Marin Atanasov Nikolov dnaeon@gmail.com