WIP: various improvements

[edsu]

Thanks for all this work! Just out of curiosity does python setup.py test work for you under Python 2 and 3? I'm seeing errors in my own environment but thought you would be better able to address them if you are seeing them as well. If you aren't seeing errors then I can dig into them and see what it is about environment that is causing the problem.

[jlebonzec]

@edsu I'm sorry, this Pull Request was aimed to go to my own fork. I planned to organize it a bit better (and write a description!) so review is easy, and I would potentially add a few features (though features would go in another PR).

As is, the tests are passing in python3. I still have to test it in python2.

It may be better to just close this PR until I finalize it.

Thanks for the review and good spirit though, much appreciated! :)

[edsu]

Ok, no worries. I can close this PR for now. I did like the look of the changes you made. I was getting errors under python 2 and 3 when running the test suite. But perhaps you have some things that haven't been committed/pushed yet. I'll keep an eye out for your changes!

[jlebonzec]

I have only been testing with Python 3.6. I guess the requirements are that it should work with Python2.7 and Python3.0. With these conditions, we should ensure backwards-compatibility.

I created the clone of this PR to my fork, feel free to follow it: https://github.com/jlebonzec/pymarc/pull/1

[edsu]

I'm not sure if this is useful or not, but:

(pymarc)bash-3.2$ python -V
Python 3.6.5
(pymarc)bash-3.2$ python setup.py test
running test
Searching for six>=1.9.0
Best match: six 1.10.0
Processing six-1.10.0-py3.6.egg

Using /Users/ed/Projects/pymarc/.eggs/six-1.10.0-py3.6.egg
running egg_info
writing pymarc.egg-info/PKG-INFO
writing dependency_links to pymarc.egg-info/dependency_links.txt
writing requirements to pymarc.egg-info/requires.txt
writing top-level names to pymarc.egg-info/top_level.txt
reading manifest file 'pymarc.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pymarc.egg-info/SOURCES.txt'
running build_ext
test_encode_decode (test.encode.Encode) ... ok
test_encode_decode_alphatag (test.encode.Encode) ... ok
test_encode_decode (test.encode_test.Encode) ... ok
test_encode_decode_alphatag (test.encode_test.Encode) ... ok
test_add_subfield (test.field.FieldTest) ... ok
test_alphatag (test.field.FieldTest) ... ok
test_controlfield_string (test.field.FieldTest) ... ok
test_delete_subfield (test.field.FieldTest) ... ok
test_delete_subfield_only_by_code (test.field.FieldTest) ... ok
test_encode (test.field.FieldTest) ... ok
test_format_field (test.field.FieldTest) ... ok
test_indicators (test.field.FieldTest) ... ok
test_is_subject_field (test.field.FieldTest) ... ok
test_iter_over_controlfield (test.field.FieldTest) ... ok
test_iterator (test.field.FieldTest) ... ok
test_membership (test.field.FieldTest) ... ok
test_non_integer_tag (test.field.FieldTest) ... ok
test_setitem (test.field.FieldTest) ... ok
test_setitem_no_key (test.field.FieldTest) ... ok
test_setitem_repeated_key (test.field.FieldTest) ... ok
test_string (test.field.FieldTest) ... ok
test_subfield_short (test.field.FieldTest) ... ok
test_subfields (test.field.FieldTest) ... ok
test_subfields_created (test.field.FieldTest) ... ok
test_subfields_multi (test.field.FieldTest) ... ok
test_tag_normalize (test.field.FieldTest) ... ok
test_value (test.field.FieldTest) ... ok
test_add_subfield (test.field_test.FieldTest) ... ok
test_alphatag (test.field_test.FieldTest) ... ok
test_controlfield_string (test.field_test.FieldTest) ... ok
test_delete_subfield (test.field_test.FieldTest) ... ok
test_delete_subfield_only_by_code (test.field_test.FieldTest) ... ok
test_encode (test.field_test.FieldTest) ... ok
test_format_field (test.field_test.FieldTest) ... ok
test_indicators (test.field_test.FieldTest) ... ok
test_is_subject_field (test.field_test.FieldTest) ... ok
test_iter_over_controlfield (test.field_test.FieldTest) ... ok
test_iterator (test.field_test.FieldTest) ... ok
test_membership (test.field_test.FieldTest) ... ok
test_non_integer_tag (test.field_test.FieldTest) ... ok
test_setitem (test.field_test.FieldTest) ... ok
test_setitem_no_key (test.field_test.FieldTest) ... ok
test_setitem_repeated_key (test.field_test.FieldTest) ... ok
test_string (test.field_test.FieldTest) ... ok
test_subfield_short (test.field_test.FieldTest) ... ok
test_subfields (test.field_test.FieldTest) ... ok
test_subfields_created (test.field_test.FieldTest) ... ok
test_subfields_multi (test.field_test.FieldTest) ... ok
test_tag_normalize (test.field_test.FieldTest) ... ok
test_value (test.field_test.FieldTest) ... ok
testOneRecord (test.json_test.JsonReaderTest)
Tests case when in source json there is only one record not ... ok
testRoundtrip (test.json_test.JsonReaderTest)
Tests that result of loading records from the test file ... ok
test_as_dict_single (test.json_test.JsonTest) ... ok
test_as_json_multiple (test.json_test.JsonTest) ... ok
test_as_json_simple (test.json_test.JsonTest) ... ok
test_as_json_types (test.json_test.JsonTest) ... ok
test_alif (test.marc8.MARC8Test) ... ok
test_eszett_euro (test.marc8.MARC8Test) ... ok
test_marc8_reader (test.marc8.MARC8Test) ... ok
test_marc8_reader_to_unicode (test.marc8.MARC8Test) ... ok
test_marc8_reader_to_unicode_bad_eacc_sequence (test.marc8.MARC8Test) ... ok
test_marc8_reader_to_unicode_bad_escape (test.marc8.MARC8Test) ... ok
test_marc8_to_unicode (test.marc8.MARC8Test) ... ok
test_reading_utf8_with_flag (test.marc8.MARC8Test) ... ok
test_reading_utf8_without_flag (test.marc8.MARC8Test) ... ok
test_record_create_force_utf8 (test.marc8.MARC8Test) ... ok
test_subscript_2 (test.marc8.MARC8Test) ... ok
test_writing_unicode (test.marc8.MARC8Test) ... ok
test_alif (test.marc8_test.MARC8Test) ... ok
test_eszett_euro (test.marc8_test.MARC8Test) ... ok
test_marc8_reader (test.marc8_test.MARC8Test) ... ok
test_marc8_reader_to_unicode (test.marc8_test.MARC8Test) ... ok
test_marc8_reader_to_unicode_bad_eacc_sequence (test.marc8_test.MARC8Test) ... ok
test_marc8_reader_to_unicode_bad_escape (test.marc8_test.MARC8Test) ... ok
test_marc8_to_unicode (test.marc8_test.MARC8Test) ... ok
test_reading_utf8_with_flag (test.marc8_test.MARC8Test) ... ok
test_reading_utf8_without_flag (test.marc8_test.MARC8Test) ... ok
test_record_create_force_utf8 (test.marc8_test.MARC8Test) ... ok
test_subscript_2 (test.marc8_test.MARC8Test) ... ok
test_writing_unicode (test.marc8_test.MARC8Test) ... ok
test_add_grouped_fields (test.ordered_fields.OrderedFieldsTest) ... ok
test_add_ordered_fields (test.ordered_fields.OrderedFieldsTest) ... ok
test_add_grouped_fields (test.ordered_fields_test.OrderedFieldsTest) ... ok
test_add_ordered_fields (test.ordered_fields_test.OrderedFieldsTest) ... ok
test_bad_indicator (test.reader.MARCReaderFileTest) ... WARNING:root:missing indicators: b'\x1fa9780253325525 (alk. paper)'
ok
test_iterator (test.reader.MARCReaderFileTest) ... ok
test_map_records (test.reader.MARCReaderFileTest) ... ok
test_multi_map_records (test.reader.MARCReaderFileTest) ... ok
test_regression_45 (test.reader.MARCReaderFileTest) ... WARNING:root:more than 2 indicators found: b'  \\\x1faRussian Federation\x1fbKostroma Oblast\x1fdKostroma'
ok
test_string (test.reader.MARCReaderFileTest) ... ok
test_bad_indicator (test.reader.MARCReaderStringTest) ... WARNING:root:missing indicators: b'\x1fa9780253325525 (alk. paper)'
ERROR
test_iterator (test.reader.MARCReaderStringTest) ... ERROR
test_map_records (test.reader.MARCReaderStringTest) ... ERROR
test_multi_map_records (test.reader.MARCReaderStringTest) ... ERROR
test_regression_45 (test.reader.MARCReaderStringTest) ... WARNING:root:more than 2 indicators found: b'  \\\x1faRussian Federation\x1fbKostroma Oblast\x1fdKostroma'
ERROR
test_string (test.reader.MARCReaderStringTest) ... ERROR
test_bad_indicator (test.reader_test.MARCReaderFileTest) ... WARNING:root:missing indicators: b'\x1fa9780253325525 (alk. paper)'
ok
test_iterator (test.reader_test.MARCReaderFileTest) ... ok
test_map_records (test.reader_test.MARCReaderFileTest) ... ok
test_multi_map_records (test.reader_test.MARCReaderFileTest) ... ok
test_regression_45 (test.reader_test.MARCReaderFileTest) ... WARNING:root:more than 2 indicators found: b'  \\\x1faRussian Federation\x1fbKostroma Oblast\x1fdKostroma'
ok
test_string (test.reader_test.MARCReaderFileTest) ... ok
test_bad_indicator (test.reader_test.MARCReaderStringTest) ... WARNING:root:missing indicators: b'\x1fa9780253325525 (alk. paper)'
ok
test_iterator (test.reader_test.MARCReaderStringTest) ... ok
test_map_records (test.reader_test.MARCReaderStringTest) ... ok
test_multi_map_records (test.reader_test.MARCReaderStringTest) ... ok
test_regression_45 (test.reader_test.MARCReaderStringTest) ... WARNING:root:more than 2 indicators found: b'  \\\x1faRussian Federation\x1fbKostroma Oblast\x1fdKostroma'
ok
test_string (test.reader_test.MARCReaderStringTest) ... ok
test_add_field (test.record.RecordTest) ... ok
test_added_entries (test.record.RecordTest) ... ok
test_alphatag (test.record.RecordTest) ... ok
test_as_marc_consistency (test.record.RecordTest) ... ok
test_as_marc_with_explicit_leader (test.record.RecordTest) ... ok
test_author (test.record.RecordTest) ... ok
test_bad_base_address (test.record.RecordTest) ... ok
test_bad_leader (test.record.RecordTest) ... ok
test_copy (test.record.RecordTest) ... ok
test_field_not_found (test.record.RecordTest) ... ok
test_find (test.record.RecordTest) ... ok
test_init_with_leader (test.record.RecordTest) ... ok
test_init_with_leader_and_force_utf8 (test.record.RecordTest) ... ok
test_init_with_no_leader (test.record.RecordTest) ... ok
test_init_with_no_leader_but_with_force_utf8 (test.record.RecordTest) ... ok
test_isbn (test.record.RecordTest) ... ok
test_location (test.record.RecordTest) ... ok
test_membership (test.record.RecordTest) ... ok
test_multi_find (test.record.RecordTest) ... ok
test_multiple_isbn (test.record.RecordTest) ... ok
test_notes (test.record.RecordTest) ... ok
test_physicaldescription (test.record.RecordTest) ... ok
test_publisher (test.record.RecordTest) ... ok
test_pubyear (test.record.RecordTest) ... ok
test_quick_access (test.record.RecordTest) ... ok
test_remove_field (test.record.RecordTest) ... ok
test_remove_fields (test.record.RecordTest) ... ok
test_subjects (test.record.RecordTest) ... ok
test_title (test.record.RecordTest) ... ok
test_uniformtitle (test.record.RecordTest) ... ok
test_add_field (test.record_test.RecordTest) ... ok
test_added_entries (test.record_test.RecordTest) ... ok
test_alphatag (test.record_test.RecordTest) ... ok
test_as_marc_consistency (test.record_test.RecordTest) ... ok
test_as_marc_with_explicit_leader (test.record_test.RecordTest) ... ok
test_author (test.record_test.RecordTest) ... ok
test_bad_base_address (test.record_test.RecordTest) ... ok
test_bad_leader (test.record_test.RecordTest) ... ok
test_copy (test.record_test.RecordTest) ... ok
test_field_not_found (test.record_test.RecordTest) ... ok
test_find (test.record_test.RecordTest) ... ok
test_init_with_leader (test.record_test.RecordTest) ... ok
test_init_with_leader_and_force_utf8 (test.record_test.RecordTest) ... ok
test_init_with_no_leader (test.record_test.RecordTest) ... ok
test_init_with_no_leader_but_with_force_utf8 (test.record_test.RecordTest) ... ok
test_isbn (test.record_test.RecordTest) ... ok
test_location (test.record_test.RecordTest) ... ok
test_membership (test.record_test.RecordTest) ... ok
test_multi_find (test.record_test.RecordTest) ... ok
test_multiple_isbn (test.record_test.RecordTest) ... ok
test_notes (test.record_test.RecordTest) ... ok
test_physicaldescription (test.record_test.RecordTest) ... ok
test_publisher (test.record_test.RecordTest) ... ok
test_pubyear (test.record_test.RecordTest) ... ok
test_quick_access (test.record_test.RecordTest) ... ok
test_remove_field (test.record_test.RecordTest) ... ok
test_remove_fields (test.record_test.RecordTest) ... ok
test_subjects (test.record_test.RecordTest) ... ok
test_title (test.record_test.RecordTest) ... ok
test_uniformtitle (test.record_test.RecordTest) ... ok
test_combining_diacritic (test.utf8_test.MARCUnicodeTest)
issue 74: raises UnicodeEncodeError on Python 2 ... ok
test_copy_utf8 (test.utf8_test.MARCUnicodeTest) ... ok
test_read_utf8 (test.utf8_test.MARCUnicodeTest) ... ok
test_close_false (test.writer.JSONWriterTest) ... ok
test_close_true (test.writer.JSONWriterTest) ... ok
test_writing_0_records (test.writer.JSONWriterTest) ... /Users/ed/Projects/pymarc/test/writer.py:64: DeprecationWarning: Please use assertEqual instead.
  self.assertEquals(actual, expected)
ok
test_writing_1_record (test.writer.JSONWriterTest) ... ok
test_writing_3_records (test.writer.JSONWriterTest) ... ok
test_writing_empty_record (test.writer.JSONWriterTest) ... ok
test_close_false (test.writer.MARCWriterTest) ... ok
test_close_true (test.writer.MARCWriterTest) ... ok
test_write (test.writer.MARCWriterTest) ... ok
test_close_false (test.writer.TextWriterTest) ... ok
test_close_true (test.writer.TextWriterTest) ... ok
test_writing_0_records (test.writer.TextWriterTest) ... ok
test_writing_1_record (test.writer.TextWriterTest) ... ok
test_writing_3_records (test.writer.TextWriterTest) ... ok
test_writing_empty_record (test.writer.TextWriterTest) ... ok
test_close_false (test.writer.XMLWriterTest) ... ok
test_close_true (test.writer.XMLWriterTest) ... ok
test_writing_0_records (test.writer.XMLWriterTest) ... ok
test_writing_1_record (test.writer.XMLWriterTest) ... ok
test_writing_3_records (test.writer.XMLWriterTest) ... ok
test_writing_empty_record (test.writer.XMLWriterTest) ... ok
test_close_false (test.writer_test.JSONWriterTest) ... ok
test_close_true (test.writer_test.JSONWriterTest) ... ok
test_writing_0_records (test.writer_test.JSONWriterTest) ... ok
test_writing_1_record (test.writer_test.JSONWriterTest) ... ok
test_writing_3_records (test.writer_test.JSONWriterTest) ... ok
test_writing_empty_record (test.writer_test.JSONWriterTest) ... ok
test_close_false (test.writer_test.MARCWriterTest) ... ok
test_close_true (test.writer_test.MARCWriterTest) ... ok
test_write (test.writer_test.MARCWriterTest) ... ok
test_close_false (test.writer_test.TextWriterTest) ... ok
test_close_true (test.writer_test.TextWriterTest) ... ok
test_writing_0_records (test.writer_test.TextWriterTest) ... ok
test_writing_1_record (test.writer_test.TextWriterTest) ... ok
test_writing_3_records (test.writer_test.TextWriterTest) ... ok
test_writing_empty_record (test.writer_test.TextWriterTest) ... ok
test_close_false (test.writer_test.XMLWriterTest) ... ok
test_close_true (test.writer_test.XMLWriterTest) ... ok
test_writing_0_records (test.writer_test.XMLWriterTest) ... ok
test_writing_1_record (test.writer_test.XMLWriterTest) ... ok
test_writing_3_records (test.writer_test.XMLWriterTest) ... ok
test_writing_empty_record (test.writer_test.XMLWriterTest) ... ok
test_map_xml_with_no_xml (test.xml_test.XmlTest) ... ok
test_map_xml_with_one_xml (test.xml_test.XmlTest) ... ok
test_multi_map_with_two_xml (test.xml_test.XmlTest) ... ok
test_parse_xml_to_array (test.xml_test.XmlTest) ... ok
test_parse_xml_to_array_with_bad_tag (test.xml_test.XmlTest) ... ok
test_parse_xml_to_array_with_strict (test.xml_test.XmlTest) ... ok
test_record_to_xml (test.xml_test.XmlTest) ... ok
test_record_to_xml_with_namespace (test.xml_test.XmlTest) ... ok
test_record_to_xml_without_namespace (test.xml_test.XmlTest) ... ok

======================================================================
ERROR: test_bad_indicator (test.reader.MARCReaderStringTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ed/Projects/pymarc/test/reader.py", line 21, in tearDown
    self.fh.close()
AttributeError: 'MARCReaderStringTest' object has no attribute 'fh'

======================================================================
ERROR: test_iterator (test.reader.MARCReaderStringTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ed/Projects/pymarc/test/reader.py", line 21, in tearDown
    self.fh.close()
AttributeError: 'MARCReaderStringTest' object has no attribute 'fh'

======================================================================
ERROR: test_map_records (test.reader.MARCReaderStringTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ed/Projects/pymarc/test/reader.py", line 21, in tearDown
    self.fh.close()
AttributeError: 'MARCReaderStringTest' object has no attribute 'fh'

======================================================================
ERROR: test_multi_map_records (test.reader.MARCReaderStringTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ed/Projects/pymarc/test/reader.py", line 21, in tearDown
    self.fh.close()
AttributeError: 'MARCReaderStringTest' object has no attribute 'fh'

======================================================================
ERROR: test_regression_45 (test.reader.MARCReaderStringTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ed/Projects/pymarc/test/reader.py", line 21, in tearDown
    self.fh.close()
AttributeError: 'MARCReaderStringTest' object has no attribute 'fh'

======================================================================
ERROR: test_string (test.reader.MARCReaderStringTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ed/Projects/pymarc/test/reader.py", line 21, in tearDown
    self.fh.close()
AttributeError: 'MARCReaderStringTest' object has no attribute 'fh'

----------------------------------------------------------------------
Ran 222 tests in 0.263s

FAILED (errors=6)
Test failed: <unittest.runner.TextTestResult run=222 errors=6 failures=0>
error: Test failed: <unittest.runner.TextTestResult run=222 errors=6 failures=0>