conversion from markdown to org-mode can distort the response especially with `

[armindarvish]

Hi,

Following some discussion on Reddit, I am opening this issue to collect the examples of problems with org-mode conversion. Unfortunately, it is hard to recreate issues because chatGPT response is not fixed. But below is an example I found from previous chats. As you can see it converted ``` to === in the begining of the first code block and from there the code blocks are all screwed up.

I don't know the date and time of this specific chat so cannot tell you if this specific one is still a problem or not. Also, my guess is (and that is just a guess) that this is hapenning because of a stream response that can be tricky to deal with (Did I just recieve a single ` in some inline code or is it ``` that is coming in a stream for a code block). I haven't digged deeper in the code to see how you are dealing with stream conversions but so far it seems that it happens especially when the source does not include the language and with backquotes in elisp.

I'll try to see if I create some more examples, and may be others can add their examples if they come across any.

Here is the example:

Prompt: Can you tell me how to make an emacs-lisp code to use chatgpt 3.5 model API?

Response (converted to org-mode):

Certainly! Here are the steps to follow to make an emacs-lisp code to use chatgpt 3.5 model API:

1. First, you need to obtain an API key from OpenAI for chatgpt 3.5 model. This requires creating an account on their website and following their instructions for obtaining an API key.

2. Once you have the API key, you can use the built-in URL and HTTP libraries in Emacs to create and send POST requests to the OpenAI API. Here is an example code to get you started:

=== (require 'url) (require 'json)

(defun chatgpt (text) "Send a text message to the chatgpt 3.5 model API and return the response." (let* ((url-request-method "POST") (url-request-extra-headers =(("Authorization" . ,(concat "Bearer " "")) ("Content-Type" . "application/json"))) (url-request-data (json-encode =(("model" . "davinci") ("prompt" . ,text))))) (with-current-buffer (url-retrieve-synchronously "https://api.openai.com/v1/engines/davinci-codex/completions") (goto-char (point-min)) (re-search-forward "\n\n") (json-read-object))))

+end_src

3. This code defines a function =chatgpt= that takes a =text= argument, sends a JSON request to the OpenAI API, and returns the response as a JSON object. To test the function, you can call it with a sample text message like this:

+end_src

(chatgpt "Hello, how are you?")

+end_src

4. This should return a JSON object containing the response from the chatgpt 3.5 model API. You can then analyze and use the response in your Emacs Lisp code as needed.

I hope this helps! Let me know if you have any other questions.

[armindarvish]

Ok, I just tried to force it and got some new examples. I am including the whole conversation even though some of it is not directly relevant becuase this can give you ideas on how to recreate it. This is the whole content of my gptel buffer. You can see a wrong = in the very first response and then if you scroll down all the way you see that I finally managed to force it and there are = instead of ` all over it:

*** Can you tell me what ` is used for in emacs-lisp and provide extensive examples of elisp-code on how to use it?

The backtick symbol (=) is used in Emacs Lisp to quote expressions selectively, which means that some parts of the expression will not be evaluated and will remain in their original form. The primary use of backquote is to write templates for complex data structures or for code.

Here's an example of how to use backquote in Emacs Lisp:

+begin_src elisp

;; Simple expression without backquote (setq my-list (list 1 2 3)) ; => (1 2 3)

;; Expression with backquote (setq my-list `'(1 ,(+ 1 1) 3)) ; => (1 2 3)

+end_src

In the example above, we create a list =my-list= using both the =list= function and the backquote syntax. The first line sets =my-list= to the list =(1 2 3)=, whereas the second line sets it to =(1 2 3)= again, but with the second element evaluated as the result of the addition operation =(+ 1 1)=.

Here's another example that shows how to use backquote with =,@=, which is used to splice a list or vector into its parent expression:

+begin_src elisp

;; Splicing a list using ,@ (setq my-list-1 `(1 2 ,@'(3 4))) ; => (1 2 3 4)

;; Splicing a vector using ,@ (setq my-list-2 `(1 2 ,@[3 4])) ; => (1 2 3 4)

+end_src

In the example above, we define two new lists, =my-list-1= and =my-list-2=. In both cases, we splice the list =(3 4)= or the vector =[3 4]= into the parent expression using the =,@= syntax.

Overall, backquote is a powerful feature in Emacs Lisp that allows you to write code and data structures in a concise and flexible way.

*** Can you show me more complicated examples of how to use ` in elisp?

Sure! Here are some more complex examples of how to use backquote in Emacs Lisp:

1. Generating HTML code using backquote

+begin_src elisp

(defun create-html-page (title body) `(,title (:body ,@body)))

;; Usage: (create-html-page "My Page" '((:h1 "Welcome") (:p "This is a test.")))

+end_src

In this example, we define a function called =create-html-page= that takes a =title= and a =body= as arguments. Using backquote, we generate an HTML page by splicing the =title= and =body= arguments into an HTML template. We use =,@= to splice the =body= list into the =(:body ...)= element.

2. Generating pattern matchers using backquote

+begin_src elisp

(defun create-matcher (pattern &optional transform) (let ((matcher-code (build-matcher-code pattern))) (lambda (input) (if (and (stringp input) (string-match-p ,matcher-code input)) ,(if transform (funcall ,transform ,(substring input (match-end 0))) t) nil))))

;; Usage: (funcall (create-matcher "^\(\w+\)@\(\w+\)\..+$" (lambda (email) (list (match-string 2 email) (match-string 1 email)))) "john@example.com")

+end_src

In this example, we define a function called =create-matcher= that takes a regular expression =pattern= and an optional =transform= function as arguments. We use backquote to generate code that builds a matcher function that can be used to match input strings against the given pattern.

3. Using =,@= to concatenate multiple sequences

+begin_src elisp

(let ((lst1 (list 1 2 3)) (lst2 (list 4 5 6))) `(,@lst1 ,@lst2))

+end_src

In this example, we create two lists =lst1= and =lst2=. Using backquote and =,@=, we concatenate the two lists into a new list =(1 2 3 4 5 6)=.

I hope these additional examples help demonstrate the flexibility and power of backquote in Emacs Lisp!

*** Can you show me example that also use let to set variables?

Certainly! Here's an example that uses backquote together with =let= to set local variables:

+begin_src elisp

(let ((x 10) (y 20)) (let ((result =(+ ,x ,y))) (message "The result is %s" result)))

+end_src

In this example, we start by defining two local variables, =x= and =y=. We then define a second local variable, =result=, using backquote to construct an expression that adds =x= and =y=. Finally, we call =message= to display the result.

When this code is executed, Emacs will display the message "The result is 30". This shows that backquote can be used together with =let= to create powerful and concise expressions that integrate local variables and other Lisp constructs.

*** That last example did not have any . I want you to show examples that use.

I apologize for the confusion earlier. Here are some examples that use the backquote symbol = in combination with the let statement:

1. Using backquote with let to create a custom function

+begin_src elisp

(let ((a 10) (b 20)) (defun my-function (x) =(+ ,x (+ ,a ,b))))

;; Usage: (my-function 5) ; returns 35

+end_src

In this example, we use backquote in combination with let to define a custom function =my-function=. The function takes a single argument =x=, and calculates the sum of =x=, =a= and =b=. When the backquoted expression is evaluated, the values of =a= and =b= are substituted into the expression, and the resulting list is returned.

2. Using backquote with let to create a complex data structure

+begin_src elisp

(let ((x 10) (y 20)) (let ((my-data =((result ,(+ x y)) (components (,x ,y)))) ;; Do something with the resulting data structure... ))

+end_src

In this example, we use backquote in combination with let to create a complex data structure called =my-data=. The structure contains two elements: a value called =result=, which is the sum of =x= and =y=, and a list called =components=, which contains the values of =x= and =y=.

Once the data structure is created, we could pass it to a function, use it to generate output, or manipulate it in some other way.

I hope these examples help illustrate the use of the backquote symbol = in Emacs Lisp!

---

[karthink]

@armindarvish thank you. It looks like there are two bugs:

Replacing isolated backquotes with =.

This is quite hard to fix -- regular markdown-mode in Emacs has the same parsing issue. I'll have to think about what would be a good heuristic.

Replacing triple backquote with === instead of #+begin_src.

Also, my guess is (and that is just a guess) that this is hapenning because of a stream response that can be tricky to deal with (Did I just recieve a single ` in some inline code or is it ``` that is coming in a stream for a code block).

Handling streams is indeed tricky -- the makeshift markdown parser gptel uses can handle incomplete or invalid states, and I eliminated a bunch of edge cases, but it looks like I missed this one. I'm having trouble reproducing this though.

[armindarvish]

I think for streams, a compromise solution would be to do the conversion after inserting markdown file. Just save the begin and end points and run your regex replacement afterwards for that region. If you use grouping, for example to capture text between two single `` without newlines, you can fix the first problem; and if you look for groups including newlines, etc. between ``` blocks, then you should not have the #+begin_src problem.

I did something similar for a package I am working on here: https://github.com/armindarvish/consult-gh/blob/main/consult-gh.el

[karthink]

After many attempts, I'm also unable to reproduce the backtick -> equals replacement bug inside these code blocks from above:

Using backquote with let to create a custom function
#+begin_src elisp
(let ((a 10)
(b 20))
(defun my-function (x)
=(+ ,x (+ ,a ,b))))

;; Usage:
(my-function 5) ; returns 35
#+end_src

#+begin_src elisp
(let ((x 10)
(y 20))
(let ((my-data =((result ,(+ x y))
(components (,x ,y))))
;; Do something with the resulting data structure...
))
#+end_src

a compromise solution would be to do the conversion after inserting markdown file.

This was what gptel did originally, and it was quite messy visually because of Org's fontification. I'm inclined to just fix the incremental parser, since mosts of its edge cases have been ironed out by now.

if you look for groups including newlines, etc. between ``` blocks, then you should not have the #+begin_src problem.

The triple backtick -> #+begin_src conversion is simple and should not be failing at all. I would like to reproduce this bug before making any big changes.

[justussc]

one easy prompt to get a messed up md->org conversion is write a c++ program that prints the string ````` (5 backticks). just use a single printf statement. even using the programming prompt this gave

#+begin_src cpp
#include <cstdio>

int main() {
    printf("#+end_src

i was wondering if one approach could be to have an okish streaming conversion and then a final postprocess which just replaces all the written text again. i hacked this together here: https://github.com/justussc/gptel/tree/feature/postprocess_accumulated_streamed_responses

the idea being that later one could make this configurable and even call out into pandoc for the conversion.

[armindarvish]

In my experience calling pandoc would be messy (because it tends to create some other temporary files) and slow, but it might be an interesting optional setting whether to use pandoc or not.

[justussc]

i did use it for a bit with the hack i have in the above branch like this

(defun js/markdown-to-org (markdown-string)
  (with-temp-buffer
    (insert markdown-string)
    (shell-command-on-region (point-min) (point-max) "pandoc -f markdown -t org" t t)
    (buffer-string)))
;; ....
  (advice-add #'gptel--convert-markdown->org :override #'js/markdown-to-org)

and it handles the tricky cases (pandoc version 3.1.6). but it can change the text quite a bit at the end. so visually it can be distracting.

[karthink]

A couple of notes:

The markdown->org converter should now handle everything inside source blocks correctly, avoiding situations like this:

write a c++ program that prints the string ````` (5 backticks). just use a single printf statement.:

#+begin_src cpp
#include <cstdio>

int main() {
    printf("#+end_src

Backtick handling outside source blocks is still messy, it will mess up things like printf("`````\n").

Many other edge cases in the converter have been fixed. I plan to make it easier to disable the built-in conversion and plug-in your external converters (or whatever other filters you want) for streaming responses. In the non-streaming case you can already use gptel-response-filter-functions.

[karthink]

Backtick handling outside source blocks is still messy, it will mess up things like printf("`````\n").

This should also be fixed now.

If anyone in this thread is still using gptel, I would appreciate any efforts to get the LLM to trip up the new markdown->org converter (streaming responses only). If there are issues that will give me new cases to test against, so that's good too.

It should handle most edge cases correctly now, and if you've been using Pandoc or equivalent, I urge you to try gptel's defaults.

Again, please note: this is for streaming responses only. Markdown -> Org conversion of the one-shot responses should be fixed soon.

[karthink]

@gregoryg I know you use Pandoc to convert gptel's markdown responses to Org. Could you try using the built-in option instead and check if its performance is satisfactory now? You could try both streaming and bunched responses.

The converter is much improved now compared to last year. I'd be curious to see where it fails.

[gregoryg]

@gregoryg I know you use Pandoc to convert gptel's markdown responses to Org. Could you try using the built-in option instead and check if its performance is satisfactory now? You could try both streaming and bunched responses.

The converter is much improved now compared to last year. I'd be curious to see where it fails.

@karthink thank you for pinging me on this

I had already switched back to your built-in conversion function.

It turns out that in a long chat involving code generation, the org-mode source blocks become part of the conversational context, therefore the LLM will start directly producing org-mode style blocks in spite of the system prompt telling it to use markdown. Pandoc makes a complete hash of mixed org and markdown syntax. Your in-built methods work perfectly!

I have not tried streaming with it, since I could not use streaming with my pandoc function. I'll test that as well