json to data-frame

[gassechen]

Hi. I'm making a package for data-frames. The goal is to convert an arbitrary json to a data-frame (like doing python3 with pandas). I have this code that works, but I don't know if the use of "eval" is completely correct.

(defun make-dataframe (data &optional (df-name "DF"))
  ;; Ensure session is not NIL
  ;; Verify and set the structure of the data
  (setf *data* (check-structure-type data))

  ;; Initialize the database
  (setf *db* nil)

  ;; Extract keys from data and process them
  (let* ((keys-list (get-keys-as-list *data*)))
    (dolist (key keys-list)
      (let* ((hash-table-list (get-hash-table-list key)))
        (process-hash-table-list hash-table-list)))

    ;; Define the dataframe using lisp-stat:defdf
    (eval  `(lisp-stat:defdf ,(intern (str:upcase df-name))
        ,(make-data-frame (extract-keys-columns *db*) *db*)))))

[gassechen]

https://github.com/gassechen/json-to-data-frame

[snunez1]

defdf is a macro, so you do have to use eval on it. It was modeled on defparameter, and I've been thinking for a while now that it should be converted to a function, as I've needed it in situations similar to yours. Nevertheless, it's a macro now and anything other than a top level definition means eval. You might consider eval-when to see if you can make it work elsewhere (here's a stackoverflow thread on uses of eval). Alternatively perhaps you can convert defdf into a function equivalent, and I'd welcome that as a pull request.

Your json-to-data-frame seems to be more generic than existing json-> data-frame converters; You might want to refer to the one used in plot for ideas on how to (and how not to) do similar conversions: encode.lisp. I wasn't able to figure out a generic json parser, and look forward to seeing this completed!

[gassechen]

This is what I'm doing with lisp-stat, data-frame and vega plot https://youtu.be/ZGgvDj00SZE?feature=shared

[snunez1]

This is what I'm doing with lisp-stat, data-frame and vega plot https://youtu.be/ZGgvDj00SZE?feature=shared

Very interesting! The web page graphics look very professional.

Do you have what you need as far as using the eval statement? It's correct as far as this usage goes. If there's nothing else I can help with I'd like to close this issue. You can always reopen another if you like.