Dumping xml fails in python3 (utf-8)

ajays20078 commented 7 years ago

There is a issue with dumping xml using file handler in python3.

Contents of 1.xml

<catalog>
   <book id="bk101">
      <author>Gambardella, Matthew</author>
      <title>XML Developer's Guide</title>
      <genre>Computer</genre>
      <price>44.95</price>
      <publish_date>2000-10-01</publish_date>
      <description>An in-depth look at creating applications 
      with XML.</description>
   </book>
   <book id="bk102">
      <author>Ralls, Kim</author>
      <title>Midnight Rain</title>
      <genre>Fantasy</genre>
      <price>5.95</price>
      <publish_date>2000-12-16</publish_date>
      <description>A former architect battles corporate zombies, 
      an evil sorceress, and her own childhood to become queen 
      of the world.</description>
   </book>
</catalog>

>>> import anyconfig
>>> a = anyconfig.load("1.xml")
>>> fh = open("2.xml" ,"w")
>>> anyconfig.dump(a, fh)

  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.4/dist-packages/anyconfig/api.py", line 384, in dump
    dumper.dump(data, path_or_stream, **options)
  File "/usr/local/lib/python3.4/dist-packages/anyconfig/backend/base.py", line 294, in dump
    self.dump_to_stream(cnf, path_or_stream, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/anyconfig/backend/xml.py", line 205, in dump_to_stream
    etree_write(tree, stream)
  File "/usr/local/lib/python3.4/dist-packages/anyconfig/backend/xml.py", line 141, in etree_write
    tree.write(stream, encoding='UTF-8', xml_declaration=True)
  File "/usr/lib/python3.4/xml/etree/ElementTree.py", line 778, in write
    short_empty_elements=short_empty_elements)
  File "/usr/lib/python3.4/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/lib/python3.4/xml/etree/ElementTree.py", line 837, in _get_writer
    yield file.write
  File "/usr/lib/python3.4/contextlib.py", line 336, in __exit__
    raise exc_details[1]
  File "/usr/lib/python3.4/contextlib.py", line 321, in __exit__
    if cb(*exc_details):
  File "/usr/lib/python3.4/contextlib.py", line 267, in _exit_wrapper
    callback(*args, **kwds)
TypeError: must be str, not bytes

ajays20078 commented 7 years ago

Although opening file in wb mode fixes the above issue, it will cause issue with other config libraries like json, as you need to explicitly set encoding while dumping a file opened in wb mode.

Contents of 1.json

{
"one" : "1"
}

>>> import anyconfig
>>> a = anyconfig.load("1.json")
>>> fh = open("2.json" ,"wb")
>>> anyconfig.dump(a, fh)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.4/dist-packages/anyconfig/api.py", line 384, in dump
    dumper.dump(data, path_or_stream, **options)
  File "/usr/local/lib/python3.4/dist-packages/anyconfig/backend/base.py", line 294, in dump
    self.dump_to_stream(cnf, path_or_stream, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/anyconfig/backend/base.py", line 77, in wrapper
    return func(*args[1:], **kwargs)
  File "/usr/lib/python3.4/json/__init__.py", line 179, in dump
    fp.write(chunk)
TypeError: 'str' does not support the buffer interface

So opening the file in wb mode is not an option.

ssato commented 7 years ago

(Excuse me for my late reply because I was on Christmas holidays.)

Sorry but I could not get the point of this issue. AFAIK, each backend requires different open mode to read/write files to process them correctly as you have seen, for example:

'r' and 'w' by default in base parser class: https://github.com/ssato/python-anyconfig/blob/master/anyconfig/backend/base.py#L102
'rb' and 'wb' in XML (xml.etree.ElementTree or lxml) backend: https://github.com/ssato/python-anyconfig/blob/master/anyconfig/backend/xml.py#L150

Do you mean that anyconfig should hide that difference and automatically re-open (I don't know even if it's possible) with the correct open mode for each backend?

ajays20078 commented 7 years ago

Yeah, anyconfig should ideally hide the difference, or else the caller needs to be aware of file formats which would defeat the purpose of anyconfig(if it caller has to be format aware then the caller can directly invoke the parsers for that format rather than invoking anyconfig).

May be one way of doing it is, having a function for file mode, and the caller can use the function while opening the file and the function assigns the mode based on format.

ssato commented 7 years ago

Thanks a lot for the details!

I guess that the former should be difficult, so add a new api to help opening the file in 80b428a and related commits in the open-api (experimental) branch. The code snippets to use this new 'open' API are found in the test case added in 021de69.

Could you please take a look at these to confirm that my understanding of the issue are correct ?

ajays20078 commented 7 years ago

It seems to fail for other config types like bson, have added a comment regarding the same in test cases.

ssato commented 7 years ago

FYI. I merged the open-api branch.

ajays20078 commented 7 years ago

Test cases seem to fail with other config types, as mentioned in the comment above.

ssato commented 7 years ago

Could you please let me know which config types you're talking about?

ajays20078 commented 7 years ago

This seems to fail for yml

ssato commented 7 years ago

I've just added a yaml test case and it looks OK. Am I missing something?

ssato commented 7 years ago

Could you please let me know if this issue still remains? If not, I want to close this issue.

ssato / python-anyconfig

Dumping xml fails in python3 (utf-8) #58