to-file: stream decoding error on windows

[kilianmh]

On Windows, portacle (sbcl) 64-bit: The CP1252 stream decoding error because character with code XXXX cannot be encoded. Appears when using to-file (sbcl) for certain files.

This should reproduce the error on portacle windows 64bit

(ql:quickload '(dexador str))
(str:to-file (dex:get "https://api.apis.guru/v2/list.json"))

Here is a thread that deals with the problem:

The solution in this case is to use :external-format :utf-8 but I'm not sure whether this would be safe for other implementations / operating-systems / architectures?

[vindarel]

It would be just a parameter to add to to-file? (along a &rest to pass to with-open-fileto mitigate further issues).

(to-file s :external-format :utf-8)
(to-file s :utf-8 t) ; ?
(to-file s :utf8 t) ; saves one keystroke
(to-utf8-file s) ; ?

In my .sbclrc I have

(setf sb-impl::*default-external-format* :utf-8)

so we probably should respect the user's setting and not enforce it.

[kilianmh]

It would be just a parameter to add to to-file? (along a &rest to pass to with-open-file to mitigate further issues).

Yeah

One way whould be with feature expression:

  (defun to-file (pathname s &key (if-exists :supersede) 
                                          (if-does-not-exist :create )
                                          (:external-format #-sbcl :default #+sbcl :utf-8))                                                  

Another solution would be using a handler-case similiar to this:

  (defun to-file (pathname s &key (if-exists :supersede)
                               (if-does-not-exist :create)
                               (external-format :default))
    (handler-case
        (with-open-file (f pathname :direction :output :if-exists if-exists
                                    :if-does-not-exist if-does-not-exist
                                    :external-format external-format)
          (write-sequence s f))
      (sb-int:stream-decoding-error ()
        (with-open-file (f pathname :direction :output :if-exists if-exists
                                    :if-does-not-exist if-does-not-exist
                                    :external-format :utf-8)
          (write-sequence s f)))))