How to send url encoded arrays and hash tables?

[daninus14]

Just for reference opened a ticket in quri#90 incorrectly. Moving to dexador now.

Basically dexador uses quri to encode post parameters which limits what we can pass as post parameters when making a request. A common example of a request is to encode a list of dictionary like structures, which is hard to do with quri which dexador uses.

I explain the problem below and also provide a scratch sample solution I can develop into a PR.

This is a copy paste from what I wrote over there:

This is a sample query which is difficult to do with quri:

curl https://someurl.com/api/something \
  -d "line_items[0][price]"=24 \
  -d "line_items[0][quantity]"=2 \
  -d mode=payment

For some data as follows (represented here with json for ease of understanding since there's no standard hash table representation in CL)

{
  line_items: [
    {
      price: "12",
      quantity: 2,
    },
    {
      price: "1",
      quantity: 10,
    },
    {
      price: "24",
      quantity: 4,
    },
    {
      price: "39",
      quantity: 1,
    },
  ];
}

To pass this to quri, even if we have an association list

'(("line_items" .
   ((("price" . "12") ("quantity" . 2))
    (("price" . "1") ("quantity" . 10))
    (("price" . "24") ("quantity" . 4))
    (("price" . "39") ("quantity" . 1)))))

Does not work.

quri does not deal with converting from lists to a notation bracketed with indices, and it does not deal with embedded data at all.

Currently the valid association list can only be done by previously "flattening" the given data structure with the appropriate labels corresponding to the data.

This would need to be cleaned, but this is my working code that is working. It's scratch code, I haven't fully cleaned it up, but once I do I can add it as a PR or you can use this if you want as a starting point:

(defgeneric encode-key-value (key value))

(defmethod encode-key-value ((key string) (value string))
  (cons key value))

(defmethod encode-key-value (key (value (eql nil)))
  (cons (format nil "~A" key) "false"))

(defmethod encode-key-value (key (value (eql t)))
  (cons (format nil "~A" key) "true"))

(defmethod encode-key-value (key (value string))
  (cons (format nil "~A" key) value))

(defmethod encode-key-value (key value)
  (cons (format nil "~A" key) (format nil "~A" value)))

(defmethod encode-key-value (key (value list))
  (let ((result '()))
    (if (assoc-utils:alistp value)
        (loop for (alist-key . alist-val) in value
              for encoded = (encode-key-value (format nil "~A[~A]" key alist-key)
                                              alist-val)
              do  (if (consp (cdr encoded))
                      (setf result (append result encoded))
                      (push encoded result)))
        (loop for element in value
              for i from 0
              for encoded = (encode-key-value (format nil "~A[~A]" key i) element)
              do  (if (consp (cdr encoded))
                      (setf result (append result encoded))
                      (push encoded result))))
    result))

(defmethod encode-key-value (key (value standard-object))
  (encode-key-value key (util-clos:object-to-hash-table value)))

(defmethod encode-key-value (key (value hash-table))
  (loop for ht-key being each hash-key of value
          using (hash-value ht-value)
        collect (encode-key-value (format nil "~A[~A]" key ht-key) ht-value)))

(defgeneric encode-form-content (content))

(defmethod encode-form-content ((content hash-table))
  (let ((result '()))
    (loop for key being each hash-key of content
            using (hash-value value)
          for encoded = (encode-key-value key value)
          do  (if (consp (cdr encoded))
                  (setf result (append result encoded))
                  (push encoded result)))
    result))

(defmethod encode-form-content ((content list))
  (unless (assoc-utils:alistp content)
    (error "Provided data ~A is not an association list AKA alist." content))
  (let ((result '()))
    (loop for (key . value) in content
          for encoded = (encode-key-value key value)
          do  (if (consp (cdr encoded))
                  (setf result (append result encoded))
                  (push encoded result)))
    result))

(encode-form-content
 (serapeum:dict "somedataneeded" "someothervalue"
                "data1" "somevalue"
                "line_items" (list (serapeum:dict "price" "12"
                                                  "quantity" 1)
                                   (serapeum:dict "price" "24"
                                                  "quantity" 1))))
(encode-form-content
 (list (cons "somedataneeded" "someothervalue")
       (cons "data1" "somevalue")
       (cons "line_items" (list (serapeum:dict "price" "12"
                                               "quantity" 1)
                                (serapeum:dict "price" "24"
                                               "quantity" 1)))))

And here are the results

CORE> (encode-form-content
 (serapeum:dict "somedata" "somevalue"
                "some_other_data" "some_other_value"
                "line_items" (list (serapeum:dict "price" "12"
                                                  "quantity" 1)
                                   (serapeum:dict "price" "24"
                                                  "quantity" 1))))
(("some_other_data" . "some_other_value") ("somedata" . "somevalue")
 ("line_items[0][price]" . "12")
 ("line_items[0][quantity]" . "1")
 ("line_items[1][price]" . "24")
 ("line_items[1][quantity]" . "1"))

CORE> (encode-form-content
 (list (cons "somedata" "somevalue")
       (cons "some_other_data" "some_other_value")
       (cons "line_items" (list (serapeum:dict "price" "12"
                                               "quantity" 1)
                                (serapeum:dict "price" "24"
                                               "quantity" 1)))))
(("some_other_data" . "some_other_value") ("somedata" . "somevalue")
 ("line_items[0][price]" . "12")
 ("line_items[0][quantity]" . "1")
 ("line_items[1][price]" . "24")
 ("line_items[1][quantity]" . "1"))

Originally posted by @daninus14 in https://github.com/fukamachi/quri/issues/90#issuecomment-2499271620

[vindarel]

Generally I agree that making Dexador work with more data structures by default would be a good thing, easier for users.

(for instance, passing hash-tables to set headers, instead of restricting to an alist. This didn't bother me enough yet to send a PR though)

[daninus14]

If you have to interact with an API which only accepts url-encoded data and doesn't accept JSON then you have no choice but to do something like this. I think it makes sense that there should be a library that takes care of it.