search

devbisme / KiField

Edit/insert/delete part fields in KiCad schematics or libraries using a spreadsheet.
MIT License
70 stars 27 forks source link

issue with non ascii characters #22

Closed Gasman2014 closed 6 years ago

Gasman2014 commented 7 years ago

This unfortunately fails to cope with an Ω symbol and fails as follows.

Lib file is

EESchema-LIBRARY Version 2.3
DEF RC0805JR-070RL R 0 0 N Y 1 F N
F0 "R" 80 0 50 V V C CNN ""
F1 "RC0805JR-070RL" 0 0 50 H I L TNN ""
F2 "${KICUSTMOD}/IPC7351-Nominal.pretty/RESC2012X60.kicad_mod" 0 0 50 H I L TNN "footprint"
F3 "resistor_smd/yageo/eedf25a24562e2e6f379f3772bde2443.pdf" 0 0 50 H I L TNN "datasheet"
F4 "1485082349267" 0 0 50 H I L TNN "key"
F5 "RC0805JR-070RL" 0 0 50 H I L TNN "man"
F6 "Yageo" 0 0 50 H I L TNN "manufacturer"
F7 "0.00 Ω" 0 0 50 V V C CNN "DisplayValue"
DRAW
S -40 -100 40 100 0 1 10 N
X ~ 1 0 150 50 D 50 50 1 1 P
X ~ 2 0 -150 50 U 50 50 1 1 P
ENDDRAW
ENDDEF etc

Error is 


(kifield) [johnpateman@Johns-MBP] KiField $ kifield --extract /Users/johnpateman/Dropbox/Components/library/r0805.lib --insert /Users/johnpateman/Dropbox/Components/RESISTORS.csv -d 7
Extracting fields +[], -[] from files ['/Users/johnpateman/Dropbox/Components/library/r0805.lib'].
Extracting fields from /Users/johnpateman/Dropbox/Components/library/r0805.lib.
Extracting fields [], -[] from part library /Users/johnpateman/Dropbox/Components/library/r0805.lib.
Extracted library part: RC0805JR-071RL prefix R.
Extracted library part: RC0805JR-071RL value RC0805JR-071RL.
Extracted library part: RC0805JR-071RL footprint ${KICUSTMOD}/IPC7351-Nominal.pretty/RESC2012X60.kicad_mod.
Extracted library part: RC0805JR-071RL datasheet resistor_smd/yageo/eedf25a24562e2e6f379f3772bde2443.pdf.
Extracted library part: RC0805JR-071RL key 1485082496735.
Extracted library part: RC0805JR-071RL mpn RC0805JR-071RL.
Extracted library part: RC0805JR-071RL manufacturer Yageo.
Traceback (most recent call last):
  File "/Users/johnpateman/miniconda3/envs/kifield/bin/kifield", line 11, in <module>
    load_entry_point('kifield==0.1.4', 'console_scripts', 'kifield')()
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.4-py2.7.egg/kifield/__main__.py", line 147, in main
    exc_field_names=exc_fields)
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.4-py2.7.egg/kifield/kifield.py", line 998, in kifield
    part_fields_dict = extract_part_fields(extract_filenames, inc_field_names, exc_field_names)
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.4-py2.7.egg/kifield/kifield.py", line 574, in extract_part_fields
    f_part_fields_dict = extraction_functions[f_extension](f, inc_field_names, exc_field_names)
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.4-py2.7.egg/kifield/kifield.py", line 463, in extract_part_fields_from_lib
    component_name, name, value))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 5: ordinal not in range(128
```)
kasbah commented 7 years ago

Thanks for reporting. Could you update KiField to 0.1.5 please. I believe some unicode fixes were added.

Gasman2014 commented 7 years ago

Ok - have updated and that seems to be sorted in 0.1.5 - xlsx export now works with unicode characters. Not sure that .csv supports it but not critical. Thank you. Works well.

The only issue that had me foxed is that the '--overwrite' option is not very obvious - I could not understand why my changes were not being updated in the library at first.

kasbah commented 7 years ago

Could you describe in #23 how you would like it to work please?

Gasman2014 commented 7 years ago

I am really not bothered by it as it is just as easy to export as xlsx but exporting as a cvs is not unicode compatible whereas exporting as an xlsx is.

Original content of .lib >

...RC0805JR-07100KL 100 kΩ ... etc

output of script >

Refs    DisplayValue    datasheet   footprint   key manufacturer    mpn prefix  value
RC0805JR-07100KL    100 kΩ /Users/johnpateman/Dropbox/Components/Datasheets/resistor_smd/yageo/eedf25a24562e2e6f379f3772bde2443.pdf    Resistors_SMD:R_0805    1.48513E+12 Yageo   RC0805JR-07100KL    R
kasbah commented 7 years ago 


Nevermind, I was still running 1.1.4 outside of the virtualenv. 

This is the one I extracted by running 1.1.5 in a virtualenv: [test.csv.txt](https://github.com/xesscorp/KiField/files/724424/test.csv.txt) (renamed to .txt so github will accept it). Seems fine no? What platform are you on? What does running `file test.csv.txt` say, what about on the one you converted? What about if you remove it and run a fresh conversion with 1.1.5?
Gasman2014 commented 7 years ago

Your test.csv looks fine.

Am this on OS X (macOS) running Kifield in a 'conda environment with Python 2.7.

(kifield) [johnpateman@johns-mbp] ~ $ kifield --version
KiField 0.1.5
(kifield) [johnpateman@johns-mbp] ~ $ kifield --extract /Users/johnpateman/Dropbox/Components/library/r0805.lib --insert /Users/johnpateman/Dropbox/Components/RESISTORS.csv -d7
Extracting fields +[], -[] from files ['/Users/johnpateman/Dropbox/Components/library/r0805.lib'].
Extracting fields from /Users/johnpateman/Dropbox/Components/library/r0805.lib.
Extracting fields [], -[] from part library /Users/johnpateman/Dropbox/Components/library/r0805.lib.
Extracted library part: RC0805JR-071RL prefix R.
Extracted library part: RC0805JR-071RL value RC0805JR-071RL.
Extracted library part: RC0805JR-071RL footprint ${KICUSTMOD}/IPC7351-Nominal.pretty/RESC2012X60.kicad_mod.
Extracted library part: RC0805JR-071RL datasheet ${KISYSPDF}/resistor_smd/yageo/eedf25a24562e2e6f379f3772bde2443.pdf.
Extracted library part: RC0805JR-071RL key 1485082496735.
Extracted library part: RC0805JR-071RL mpn RC0805JR-071RL.
Extracted library part: RC0805JR-071RL manufacturer Yageo.
Extracted library part: RC0805JR-071RL DisplayValue 1.00 Ω.
Extracted library part: RC0805JR-071R5L prefix R.
Extracted library part: RC0805JR-071R5L value RC0805JR-071R5L.
Extracted library part: RC0805JR-071R5L footprint Resistors_SMD:R_0805.
Extracted library part: RC0805JR-071R5L datasheet /Users/johnpateman/Dropbox/Components/Datasheets/resistor_smd/yageo/eedf25a24562e2e6f379f3772bde2443.pdf.
Extracted library part: RC0805JR-071R5L key 1485125764119.
Extracted library part: RC0805JR-071R5L mpn RC0805JR-071R5L.
Extracted library part: RC0805JR-071R5L manufacturer Yageo.
Extracted library part: RC0805JR-071R5L DisplayValue 1.50 Ω.

...

Inserting extracted fields into CSV file /Users/johnpateman/Dropbox/Components/RESISTORS.csv.
Converting CSV file /Users/johnpateman/Dropbox/Components/RESISTORS.csv into an XLSX workbook.
Traceback (most recent call last):
  File "/Users/johnpateman/miniconda3/envs/kifield/bin/kifield", line 11, in <module>
    load_entry_point('kifield==0.1.5', 'console_scripts', 'kifield')()
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.5-py2.7.egg/kifield/__main__.py", line 154, in main
    recurse=args.recurse)
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.5-py2.7.egg/kifield/kifield.py", line 1030, in kifield
    insert_part_fields(part_fields_dict, insert_filenames, recurse)
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.5-py2.7.egg/kifield/kifield.py", line 1014, in insert_part_fields
    insertion_functions[f_extension](part_fields_dict, f, recurse)
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.5-py2.7.egg/kifield/kifield.py", line 746, in insert_part_fields_into_csv
    wb, dialect = csvfile_to_wb(filename)
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.5-py2.7.egg/kifield/kifield.py", line 129, in csvfile_to_wb
    dialect = csv.Sniffer().sniff(csv_file.read())
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 94: ordinal not in range(128)

and i end up with Ω for every Ω symbol. Not a showstopper and, as I said, xlsx works fine.

Gasman2014 commented 7 years ago

And trying to convert the test file to a .lib fails in the same manner;

(kifield) [johnpateman@johns-mbp] ~ $ kifield --extract /Users/johnpateman/Downloads/test.csv --insert /Users/johnpateman/Desktop/Test.lib -d7
Extracting fields +[], -[] from files ['/Users/johnpateman/Downloads/test.csv'].
Extracting fields from /Users/johnpateman/Downloads/test.csv.
Extracting fields [], -[] from CSV file /Users/johnpateman/Downloads/test.csv.
Converting CSV file /Users/johnpateman/Downloads/test.csv into an XLSX workbook.
Traceback (most recent call last):
  File "/Users/johnpateman/miniconda3/envs/kifield/bin/kifield", line 11, in <module>
    load_entry_point('kifield==0.1.5', 'console_scripts', 'kifield')()
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.5-py2.7.egg/kifield/__main__.py", line 154, in main
    recurse=args.recurse)
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.5-py2.7.egg/kifield/kifield.py", line 1027, in kifield
    part_fields_dict = extract_part_fields(extract_filenames, inc_field_names, exc_field_names, recurse)
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.5-py2.7.egg/kifield/kifield.py", line 594, in extract_part_fields
    f_part_fields_dict = extraction_functions[f_extension](f, inc_field_names, exc_field_names, recurse)
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.5-py2.7.egg/kifield/kifield.py", line 330, in extract_part_fields_from_csv
    wb,_ = csvfile_to_wb(filename)
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.5-py2.7.egg/kifield/kifield.py", line 129, in csvfile_to_wb
    dialect = csv.Sniffer().sniff(csv_file.read())
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 92: ordinal not in range(128)
kasbah commented 7 years ago

Hmm, can't reproduce this locally but I spotted something that might be the problem.

Is it possible for you to install this branch: https://github.com/kasbah/KiField/tree/unicode-sniff ?

git clone --branch unicode-sniff https://github.com/kasbah/KiField
cd KiField && pip install . --user --upgrade

Also, what does file test.csv say?

Gasman2014 commented 7 years ago

On 23 Jan 2017, at 18:19, Kaspar Emanuel notifications@github.com wrote:

cd KiField && pip install . --user --upgrade

(kifield) [johnpateman@Johns-MBP] ~ $ git clone --branch unicode-sniff https://github.com/kasbah/KiField Cloning into 'KiField'... remote: Counting objects: 373, done. remote: Compressing objects: 100% (121/121), done. remote: Total 373 (delta 78), reused 0 (delta 0), pack-reused 251 Receiving objects: 100% (373/373), 1023.12 KiB | 683.00 KiB/s, done. Resolving deltas: 100% (200/200), done. (kifield) [johnpateman@Johns-MBP] ~ $ cd KiField && pip install . --user --upgrade Processing /Users/johnpateman/KiField Requirement already up-to-date: future>=0.15.0 in /Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/future-0.16.0-py2.7.egg (from kifield==0.1.5) Collecting openpyxl>=2.3.2 (from kifield==0.1.5) Downloading openpyxl-2.4.1.tar.gz (154kB) 100% |################################| 163kB 1.7MB/s Requirement already up-to-date: jdcal in /Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/jdcal-1.3-py2.7.egg (from openpyxl>=2.3.2->kifield==0.1.5) Requirement already up-to-date: et_xmlfile in /Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/et_xmlfile-1.0.1-py2.7.egg (from openpyxl>=2.3.2->kifield==0.1.5) Building wheels for collected packages: openpyxl Running setup.py bdist_wheel for openpyxl ... done Stored in directory: /Users/johnpateman/Library/Caches/pip/wheels/ce/3f/b6/5b343f34236a4023eb2d7d2e46da5ca74e8b9cbe8afbce811a Successfully built openpyxl Installing collected packages: openpyxl, kifield Running setup.py install for kifield ... done Successfully installed kifield-0.1.5 openpyxl-2.4.1 (kifield) [johnpateman@Johns-MBP] KiField $ kifield --extract /Users/johnpateman/Downloads/test.csv --insert /Users/johnpateman/Desktop/Test.lib -d7 Extracting fields +[], -[] from files ['/Users/johnpateman/Downloads/test.csv']. Extracting fields from /Users/johnpateman/Downloads/test.csv. Extracting fields [], -[] from CSV file /Users/johnpateman/Downloads/test.csv. Converting CSV file /Users/johnpateman/Downloads/test.csv into an XLSX workbook. Traceback (most recent call last): File "/Users/johnpateman/miniconda3/envs/kifield/bin/kifield", line 11, in load_entry_point('kifield==0.1.5', 'console_scripts', 'kifield')() File "/Users/johnpateman/.local/lib/python2.7/site-packages/kifield/main.py", line 154, in main recurse=args.recurse) File "/Users/johnpateman/.local/lib/python2.7/site-packages/kifield/kifield.py", line 1025, in kifield part_fields_dict = extract_part_fields(extract_filenames, inc_field_names, exc_field_names, recurse) File "/Users/johnpateman/.local/lib/python2.7/site-packages/kifield/kifield.py", line 592, in extract_part_fields f_part_fields_dict = extraction_functions[f_extension](f, inc_field_names, exc_field_names, recurse) File "/Users/johnpateman/.local/lib/python2.7/site-packages/kifield/kifield.py", line 328, in extract_part_fields_fromcsv wb, = csvfile_to_wb(filename) File "/Users/johnpateman/.local/lib/python2.7/site-packages/kifield/kifield.py", line 129, in csvfile_to_wb dialect = csv.Sniffer().sniff(csv_file.read()) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 92: ordinal not in range(128)

kasbah commented 7 years ago

Thanks, and the command file test.csv says UTF-8?

I pushed another commit to the branch:

git pull && pip install . --upgrade
Gasman2014 commented 7 years ago

Still a bit of a problem.

Definitely UTF-8

(kifield) [johnpateman@Johns-MBP] KiField $ git pull && pip install . --upgrade remote: Counting objects: 4, done. remote: Compressing objects: 100% (4/4), done. remote: Total 4 (delta 0), reused 0 (delta 0), pack-reused 0 Unpacking objects: 100% (4/4), done. From https://github.com/kasbah/KiField 2bde94a..5349557 unicode-sniff -> origin/unicode-sniff Updating 2bde94a..5349557 Fast-forward kifield/kifield.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) Processing /Users/johnpateman/KiField Requirement already up-to-date: future>=0.15.0 in /Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/future-0.16.0-py2.7.egg (from kifield==0.1.5) Requirement already up-to-date: openpyxl>=2.3.2 in /Users/johnpateman/.local/lib/python2.7/site-packages (from kifield==0.1.5) Requirement already up-to-date: et-xmlfile in /Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/et_xmlfile-1.0.1-py2.7.egg (from openpyxl>=2.3.2->kifield==0.1.5) Requirement already up-to-date: jdcal in /Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/jdcal-1.3-py2.7.egg (from openpyxl>=2.3.2->kifield==0.1.5) Installing collected packages: kifield Found existing installation: kifield 0.1.5 Uninstalling kifield-0.1.5: Successfully uninstalled kifield-0.1.5 Running setup.py install for kifield ... done Successfully installed kifield-0.1.5 (kifield) [johnpateman@Johns-MBP] KiField $ kifield --extract /Users/johnpateman/Downloads/test.csv --insert /Users/johnpateman/Desktop/Test.lib -d7 Extracting fields +[], -[] from files ['/Users/johnpateman/Downloads/test.csv']. Extracting fields from /Users/johnpateman/Downloads/test.csv. Extracting fields [], -[] from CSV file /Users/johnpateman/Downloads/test.csv. Converting CSV file /Users/johnpateman/Downloads/test.csv into an XLSX workbook. Traceback (most recent call last): File "/Users/johnpateman/miniconda3/envs/kifield/bin/kifield", line 11, in load_entry_point('kifield==0.1.5', 'console_scripts', 'kifield')() File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/main.py", line 154, in main recurse=args.recurse) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 1025, in kifield part_fields_dict = extract_part_fields(extract_filenames, inc_field_names, exc_field_names, recurse) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 592, in extract_part_fields f_part_fields_dict = extraction_functions[f_extension](f, inc_field_names, exc_field_names, recurse) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 328, in extract_part_fields_fromcsv wb, = csvfile_to_wb(filename) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 129, in csvfile_to_wb dialect = csv.Sniffer().sniff(csv_file.read()) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 92: ordinal not in range(128) (kifield) [johnpateman@Johns-MBP] KiField $

On 23 Jan 2017, at 19:39, Kaspar Emanuel notifications@github.com wrote:

Thanks, and the command file test.csv says UTF-8?

I pushed another commit to the branch:

git pull && pip install . --upgrade — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/xesscorp/KiField/issues/22#issuecomment-274594453, or mute the thread https://github.com/notifications/unsubscribe-auth/AIAyua3cOmZEVzFFCdv0pRN1Pky6c62fks5rVQIEgaJpZM4LqXtf.

kasbah commented 7 years ago

Ok, thanks, I think someone with access to an OSX machine will have to debug this.

Have you tried with python 3?

kasbah commented 7 years ago

Maybe it's a bug in the csv module. What version of python are you running exactly?

Gasman2014 commented 7 years ago

Python 2.7.12 but it would be easy enough fro me to roll another virtenv with Python 3 - will try later tonight.

John

On 23 Jan 2017, at 19:59, Kaspar Emanuel notifications@github.com wrote:

Maybe it's a bug in the csv module. What version of python are you running exactly?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/xesscorp/KiField/issues/22#issuecomment-274599740, or mute the thread https://github.com/notifications/unsubscribe-auth/AIAyuRm3lkpsfpPNiM7nDzsOz4FsV8nnks5rVQa5gaJpZM4LqXtf.

Gasman2014 commented 7 years ago

Ok, have tried this with Python 3.4.5 and still no luck - sorry!

(test) [johnpateman@johns-mbp] ~ $ python -V
Python 3.4.5 :: Continuum Analytics, Inc.
(test) [johnpateman@johns-mbp] ~ $ kifield --version
KiField 0.1.5
(test) [johnpateman@johns-mbp] ~ $ kifield --extract Downloads/test.csv --insert test.xlsx --debug 5
Extracting fields +[], -[] from files ['Downloads/test.csv'].
Extracting fields from Downloads/test.csv.
Extracting fields [], -[] from CSV file Downloads/test.csv.
Converting CSV file Downloads/test.csv into an XLSX workbook.
Traceback (most recent call last):
  File "/Users/johnpateman/miniconda3/envs/test/bin/kifield", line 11, in <module>
    sys.exit(main())
  File "/Users/johnpateman/miniconda3/envs/test/lib/python3.4/site-packages/kifield/__main__.py", line 154, in main
    recurse=args.recurse)
  File "/Users/johnpateman/miniconda3/envs/test/lib/python3.4/site-packages/kifield/kifield.py", line 1027, in kifield
    part_fields_dict = extract_part_fields(extract_filenames, inc_field_names, exc_field_names, recurse)
  File "/Users/johnpateman/miniconda3/envs/test/lib/python3.4/site-packages/kifield/kifield.py", line 594, in extract_part_fields
    f_part_fields_dict = extraction_functions[f_extension](f, inc_field_names, exc_field_names, recurse)
  File "/Users/johnpateman/miniconda3/envs/test/lib/python3.4/site-packages/kifield/kifield.py", line 330, in extract_part_fields_from_csv
    wb,_ = csvfile_to_wb(filename)
  File "/Users/johnpateman/miniconda3/envs/test/lib/python3.4/site-packages/kifield/kifield.py", line 129, in csvfile_to_wb
    dialect = csv.Sniffer().sniff(csv_file.read())
  File "/Users/johnpateman/miniconda3/envs/test/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 92: ordinal not in range(128)
kasbah commented 7 years ago

Ok, one more try. I pushed to my unicode-sniff branch. Give it a go when you get a chance.

Gasman2014 commented 7 years ago

OK, popped this one onto the Py2.7 install and some progress - a different error!!

Converting an XLSX workbook and saving as CSV file r0805libtest.csv. Traceback (most recent call last): File "/Users/johnpateman/miniconda3/envs/kifield/bin/kifield", line 11, in load_entry_point('kifield==0.1.5', 'console_scripts', 'kifield')() File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/main.py", line 154, in main recurse=args.recurse) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 1028, in kifield insert_part_fields(part_fields_dict, insert_filenames, recurse) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 1012, in insert_part_fields insertion_functions[f_extension](part_fields_dict, f, recurse) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 753, in insert_part_fields_into_csv wb_to_csvfile(wb, filename, dialect) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 157, in wb_to_csvfile writer.writerow([cell.value for cell in row]) TypeError: write() argument 1 must be unicode, not str

On 24 Jan 2017, at 16:28, Kaspar Emanuel notifications@github.com wrote:

Ok, one more try. I pushed to my unicode-sniff branch. Give it a go when you get a chance.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/xesscorp/KiField/issues/22#issuecomment-274855468, or mute the thread https://github.com/notifications/unsubscribe-auth/AIAyuZPBVDObutV45bIrdfvQVf5JGX4dks5rViajgaJpZM4LqXtf.

Gasman2014 commented 7 years ago

but when I tried the same function with Py3.4.5, in my ’test’ env I am back to the original problem?

(test) [johnpateman@Johns-MBP] KiField $ git pull && pip install . --upgrade Already up-to-date. Processing /Users/johnpateman/KiField Requirement already up-to-date: future>=0.15.0 in /Users/johnpateman/miniconda3/envs/test/lib/python3.4/site-packages (from kifield==0.1.5) Requirement already up-to-date: openpyxl>=2.3.2 in /Users/johnpateman/miniconda3/envs/test/lib/python3.4/site-packages (from kifield==0.1.5) Requirement already up-to-date: jdcal in /Users/johnpateman/miniconda3/envs/test/lib/python3.4/site-packages (from openpyxl>=2.3.2->kifield==0.1.5) Requirement already up-to-date: et-xmlfile in /Users/johnpateman/miniconda3/envs/test/lib/python3.4/site-packages (from openpyxl>=2.3.2->kifield==0.1.5) Installing collected packages: kifield Found existing installation: kifield 0.1.5 Uninstalling kifield-0.1.5: Successfully uninstalled kifield-0.1.5 Running setup.py install for kifield ... done Successfully installed kifield-0.1.5 (test) [johnpateman@Johns-MBP] KiField $ kifield --extract /Users/johnpateman/Dropbox/Components/library/r0805.dcm --insert r0805dcmtest2.csv --debug 5 Extracting fields +[], -[] from files ['/Users/johnpateman/Dropbox/Components/library/r0805.dcm']. Extracting fields from /Users/johnpateman/Dropbox/Components/library/r0805.dcm. Extracting fields [], -[] from part description file /Users/johnpateman/Dropbox/Components/library/r0805.dcm. Traceback (most recent call last): File "/Users/johnpateman/miniconda3/envs/test/bin/kifield", line 11, in load_entry_point('kifield==0.1.5', 'console_scripts', 'kifield')() File "/Users/johnpateman/miniconda3/envs/test/lib/python3.4/site-packages/kifield/main.py", line 154, in main recurse=args.recurse) File "/Users/johnpateman/miniconda3/envs/test/lib/python3.4/site-packages/kifield/kifield.py", line 1025, in kifield part_fields_dict = extract_part_fields(extract_filenames, inc_field_names, exc_field_names, recurse) File "/Users/johnpateman/miniconda3/envs/test/lib/python3.4/site-packages/kifield/kifield.py", line 592, in extract_part_fields f_part_fields_dict = extraction_functions[f_extension](f, inc_field_names, exc_field_names, recurse) File "/Users/johnpateman/miniconda3/envs/test/lib/python3.4/site-packages/kifield/kifield.py", line 505, in extract_part_fields_from_dcm dcm = Dcm(filename) File "/Users/johnpateman/miniconda3/envs/test/lib/python3.4/site-packages/kifield/dcm.py", line 89, in init self.header = file.readline() File "/Users/johnpateman/miniconda3/envs/test/lib/python3.4/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 68: ordinal not in range(128)

On 24 Jan 2017, at 19:50, John Pateman johnpateman@me.com wrote:

OK, popped this one onto the Py2.7 install and some progress - a different error!!

Converting an XLSX workbook and saving as CSV file r0805libtest.csv. Traceback (most recent call last): File "/Users/johnpateman/miniconda3/envs/kifield/bin/kifield", line 11, in load_entry_point('kifield==0.1.5', 'console_scripts', 'kifield')() File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/main.py", line 154, in main recurse=args.recurse) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 1028, in kifield insert_part_fields(part_fields_dict, insert_filenames, recurse) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 1012, in insert_part_fields insertion_functions[f_extension](part_fields_dict, f, recurse) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 753, in insert_part_fields_into_csv wb_to_csvfile(wb, filename, dialect) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 157, in wb_to_csvfile writer.writerow([cell.value for cell in row]) TypeError: write() argument 1 must be unicode, not str

On 24 Jan 2017, at 16:28, Kaspar Emanuel <notifications@github.com mailto:notifications@github.com> wrote:

Ok, one more try. I pushed to my unicode-sniff branch. Give it a go when you get a chance.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/xesscorp/KiField/issues/22#issuecomment-274855468, or mute the thread https://github.com/notifications/unsubscribe-auth/AIAyuZPBVDObutV45bIrdfvQVf5JGX4dks5rViajgaJpZM4LqXtf.

kasbah commented 7 years ago

This is really a pain in the a**. Why on earth doesn't the csv module in python2 and 3 support unicode? I pushed a unicode-sniff-2 branch with a different approach: I switched out csv with the unicodecsv module.

Gasman2014 commented 7 years ago

Have recloned from githubbut still seem to be in the same situation. It is frustrating (but then the whole Python 2 / 3 thing is frustrating!). Personally, it is not an issue as xlsx is an acceptable format for what I need to achieve, however, for the purposes of testing, If you would like me to try anything else I would be happy to oblige. I think I have tried the latest version - anything else I should try?

John

On 24 Jan 2017, at 20:30, Kaspar Emanuel notifications@github.com wrote:

This is really a pain in the a**. Why on earth doesn't the csv module in python2 and 3 support unicode? I pushed a unicode-sniff-2 branch with a different approach: I switched out the csv with the unicodecsv https://github.com/jdunck/python-unicodecsv module.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/xesscorp/KiField/issues/22#issuecomment-274929199, or mute the thread https://github.com/notifications/unsubscribe-auth/AIAyuUxUuoz5sPX4BK_wQCs9bzWl3D_Iks5rVl99gaJpZM4LqXtf.

kasbah commented 7 years ago

You were cloning/pulling the unicode-sniff branch, yeah?

Gasman2014 commented 7 years ago

Sorry - maybe not, it would seem :( All I see on Github is ‘master' and searching for unicode-sniff in branches shows nothing. (Sorry, not very good on git - usually use svn for my projects).

John

On 27 Jan 2017, at 20:24, Kaspar Emanuel notifications@github.com wrote:

You were cloning/pulling the unicode-sniff branch, yeah? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/xesscorp/KiField/issues/22#issuecomment-275765449, or mute the thread https://github.com/notifications/unsubscribe-auth/AIAyuRB3uKafJzaWZafqEiLRsueqr2Q_ks5rWlJ-gaJpZM4LqXtf.

kasbah commented 7 years ago

It's on my fork. Try: git clone --branch unicode-sniff https://github.com/kasbah/KiField

Gasman2014 commented 7 years ago

Sorry, still no dice :(.

John

(kifield) [johnpateman@Johns-MacBook-Pro] ~ $ git clone --branch unicode-sniff https://github.com/kasbah/KiField Cloning into 'KiField'... remote: Counting objects: 390, done. remote: Compressing objects: 100% (138/138), done. remote: Total 390 (delta 91), reused 0 (delta 0), pack-reused 251 Receiving objects: 100% (390/390), 1.00 MiB | 591.00 KiB/s, done. Resolving deltas: 100% (213/213), done. (kifield) [johnpateman@Johns-MacBook-Pro] KiField $ kifield --extract /Users/path/library/r1206.lib --insert test.csv

Refs DisplayValue datasheet footprint key manufacturer mpn prefix value RC1206JR-07100KL 100kΩ /Users/path/Datasheets/resistor_smd/yageo/318515b8790f80c69d65f2bbdbc27acc.pdf Resistors_SMD:R_1206 1.48517E+12 Yageo RC1206JR-07100KL R RC1206JR-07100KL RC1206JR-07100RL 100Ω /Users/path/Datasheets/resistor_smd/yageo/318515b8790f80c69d65f2bbdbc27acc.pdf Resistors_SMD:R_1206 1.48517E+12 Yageo RC1206JR-07100RL R RC1206JR-07100RL RC1206JR-0710KL 10.0kΩ /Users/path/Datasheets/resistor_smd/yageo/5aeec7a931f2be6dd05c309ddba1c0e0.pdf Resistors_SMD:R_1206 1.48517E+12 Yageo RC1206JR-0710KL R RC1206JR-0710KL RC1206JR-0710RL 10.0Ω /Users/path/Datasheets/resistor_smd/yageo/318515b8790f80c69d65f2bbdbc27acc.pdf Resistors_SMD:R_1206 1.48517E+12 Yageo RC1206JR-0710RL R RC1206JR-0710RL

On 27 Jan 2017, at 20:36, Kaspar Emanuel notifications@github.com wrote:

git clone --branch unicode-sniff https://github.com/kasbah/KiField https://github.com/kasbah/KiField

kasbah commented 7 years ago

Thanks for checking it out but I think you didn't run the install! You will need to pip install --upgrade . in the new KiField dir first.

Gasman2014 commented 7 years ago

(kifield) [johnpateman@Johns-MacBook-Pro] KiField $ pip install —upgrade

You must give at least one requirement to install (see "pip help install”)

(kifield) [johnpateman@Johns-MacBook-Pro] KiField $ pip install kifield —upgrade

Requirement already up-to-date: kifield in /Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages Requirement already up-to-date: future>=0.15.0 in /Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/future-0.16.0-py2.7.egg (from kifield) Requirement already up-to-date: openpyxl>=2.3.2 in /Users/johnpateman/.local/lib/python2.7/site-packages (from kifield) Requirement already up-to-date: et-xmlfile in /Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/et_xmlfile-1.0.1-py2.7.egg (from openpyxl>=2.3.2->kifield) Requirement already up-to-date: jdcal in /Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/jdcal-1.3-py2.7.egg (from openpyxl>=2.3.2->kifield) (kifield) [johnpateman@Johns-MacBook-Pro] KiField $ kifield --extract /Users/johnpateman/Dropbox/Components/library/r1206.lib --insert test.csv Traceback (most recent call last): File "/Users/johnpateman/miniconda3/envs/kifield/bin/kifield", line 11, in load_entry_point('kifield==0.1.5', 'console_scripts', 'kifield')() File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/main.py", line 154, in main recurse=args.recurse) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 1030, in kifield insert_part_fields(part_fields_dict, insert_filenames, recurse) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 1014, in insert_part_fields insertion_functions[f_extension](part_fields_dict, f, recurse) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 746, in insert_part_fields_into_csv wb, dialect = csvfile_to_wb(filename) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 129, in csvfile_to_wb dialect = csv.Sniffer().sniff(csv_file.read()) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 93: ordinal not in range(128)

?? What am I missing here??

On 27 Jan 2017, at 21:45, Kaspar Emanuel notifications@github.com wrote:

Thanks for checking it out, but I think you didn't run the install! You will need to pip install --upgrade . in the new KiField dir first.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/xesscorp/KiField/issues/22#issuecomment-275783190, or mute the thread https://github.com/notifications/unsubscribe-auth/AIAyuSzzhguHcbtO-T6II7Mmx8xFgz4Fks5rWmVcgaJpZM4LqXtf.

kasbah commented 7 years ago

It's 

pip install . --upgrade

Notice the . ! :smile:

Gasman2014 commented 7 years ago

:( From scratch (just to be sure)

(kifield) [johnpateman@Johns-MacBook-Pro] ~ $ git clone --branch unicode-sniff https://github.com/kasbah/KiField Cloning into 'KiField'... remote: Counting objects: 390, done. remote: Compressing objects: 100% (138/138), done. remote: Total 390 (delta 91), reused 0 (delta 0), pack-reused 251 Receiving objects: 100% (390/390), 1.00 MiB | 968.00 KiB/s, done. Resolving deltas: 100% (213/213), done. (kifield) [johnpateman@Johns-MacBook-Pro] ~ $ cd KiField/ (kifield) [johnpateman@Johns-MacBook-Pro] KiField $ pip install . --upgrade Processing /Users/johnpateman/KiField Requirement already up-to-date: future>=0.15.0 in /Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/future-0.16.0-py2.7.egg (from kifield==0.1.5) Requirement already up-to-date: openpyxl>=2.3.2 in /Users/johnpateman/.local/lib/python2.7/site-packages (from kifield==0.1.5) Requirement already up-to-date: et-xmlfile in /Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/et_xmlfile-1.0.1-py2.7.egg (from openpyxl>=2.3.2->kifield==0.1.5) Requirement already up-to-date: jdcal in /Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/jdcal-1.3-py2.7.egg (from openpyxl>=2.3.2->kifield==0.1.5) Installing collected packages: kifield Found existing installation: kifield 0.1.5 Uninstalling kifield-0.1.5: Successfully uninstalled kifield-0.1.5 Running setup.py install for kifield ... done Successfully installed kifield-0.1.5 (kifield) [johnpateman@Johns-MacBook-Pro] KiField $ kifield --extract /Users/johnpateman/Dropbox/Components/library/r1206.lib --insert test.csv Traceback (most recent call last): File "/Users/johnpateman/miniconda3/envs/kifield/bin/kifield", line 11, in load_entry_point('kifield==0.1.5', 'console_scripts', 'kifield')() File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/main.py", line 154, in main recurse=args.recurse) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 1028, in kifield insert_part_fields(part_fields_dict, insert_filenames, recurse) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 1012, in insert_part_fields insertion_functions[f_extension](part_fields_dict, f, recurse) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 753, in insert_part_fields_into_csv wb_to_csvfile(wb, filename, dialect) File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 157, in wb_to_csvfile writer.writerow([cell.value for cell in row]) TypeError: write() argument 1 must be unicode, not str

On 27 Jan 2017, at 21:51, Kaspar Emanuel notifications@github.com wrote:

It's

pip install . --upgrade Notice the . ! 😄

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/xesscorp/KiField/issues/22#issuecomment-275784824, or mute the thread https://github.com/notifications/unsubscribe-auth/AIAyuVfi6ljDP_T8KJvo0Kg4WV4gjymHks5rWmbtgaJpZM4LqXtf.

kasbah commented 7 years ago

That doesn't seem much better unfortunately. I am going to put this issue on hold for now until I can replicate it myself to debug. Thanks for your help!

Gasman2014 commented 7 years ago

Not sure I have been of much help - blindly leading a bit of a numpty through this! :)

Keep me posted - as I say, I am actually very pleased with the xlsx export which is really helpful - and Excel does a nice .csv export ….! (I know that isn’t exactly the point, though).

I am trying to build some ‘atomic' parts libraries and using KiField is really useful to ensure field consistency. And, of course, when KiCad eventually supports a database rather than flat library files, I will be ahead of the curve :) (Not holding my breath)

BW

John

On 27 Jan 2017, at 21:58, Kaspar Emanuel notifications@github.com wrote:

That doesn't seem much better unfortunately. I am going to put this issue on hold for now until I can replicate it myself to debug. Thanks for your help!

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/xesscorp/KiField/issues/22#issuecomment-275786392, or mute the thread https://github.com/notifications/unsubscribe-auth/AIAyuRsODcCLKUzxhnXCToCM0MOP7Hhtks5rWmiIgaJpZM4LqXtf.

kasbah commented 7 years ago

Ah cool, did you see my initiative on using standard field names? I will prepare a document on the results but it looks like the fields will be:

| Name            | Value            |
|-----------------+------------------|
| Manuf<Integer>  | <Manufacturer>   |
| MPN<Integer>    | <MPN>            |
| Vendor<Integer> | <Vendor>         |
| SKU<Integer>    | <SKU>            |
| UPN<Integer>    | User Part Number |

Where the <Integer> bit is ommitted on the first instance.

Gasman2014 commented 7 years ago

Yes, your suggestions look good to me.

I know there is a bit of a discussion about this, but overall it seems a bit of a philosophical debate - do you decide on your components whist you design or do a design and then choose your components? At some point before you layout your PCB, each item on your schematic has to be turned into a real part with a defined footprint - and at that point you have already committed to a physical, orderable part (OK so you may’ve a choice of ordering it from DIgiKey Newark or Farnell but it is an actual part that you have committed to). And as soon as you have committed to a part you can produce a BOM. At least KiCad allows component choice to be made at any point in that process - unlike some other EDA tools. I think the idea of being able to change a footprint from a 1206 to an 0805 or whatever is a really useful aspect of KiCad and however it is achieved, it is essential that alternative workflows can still co-exist.

Personally I like the idea that a BOM can be produced from the schematic as soon as it is laid out - and your fields will help facilitate this. It is important that the idea is not forced on people who choose not to use them. Clearly there needs to be some sort of standardisation of nomenclature and I think your suggestion is very sensible and covers all options.

I still think that the actual answer to this problem is to have some sort of database driven library repo. This would solve all these issues for everyone! There are a number of things out there which are quite promising but none of them are quite linked up yet.

Mike Crash - has a nice front end to a MySQL database [KicadPartManager] that goes part way there but needs a bit of work to be polished http://mikecrash.com/index.php?name=Content&pa=showpage&pid=10 http://mikecrash.com/index.php?name=Content&pa=showpage&pid=10 . It can parse a schematic and lookup from an internal database of parts (some issues in my tests).

Piopus has a very useful Octopart lookup tool and KiCad library generator at https://github.com/pioupus/kicadLibCreator https://github.com/pioupus/kicadLibCreator . Makes generating an internally consistent library a snip.

Qeda is a very useful tool for generating IPC7351 consistent footprints and systematic lib files. https://github.com/qeda/qeda https://github.com/qeda/qeda

There is a nice parts database system called PartKeepr https://www.partkeepr.com https://www.partkeepr.com/ which also runs on a MySQL/PostgresSQL backend and which, in the latest version, is also using Octopart to lookup components.

And I have also been playing with EleLa http://www.mikrocontroller.net/topic/318610?page=single http://www.mikrocontroller.net/topic/318610?page=single (hope your German is good!) which is also an interesting solution - but windows based so I am running under emulation. In the longer term not a good fit with KiCad but an interesting and quite polished piece of software. It also supports a MySQL database interface but I have only been able to get it working with SQLite.

BW

John

On 27 Jan 2017, at 22:24, Kaspar Emanuel notifications@github.com wrote:

Ah cool, did you see my initiative on using standard field names https://forum.kicad.info/t/standard-symbol-field-names-initiative/4870? I will prepare a document on the results but it looks like the fields will be:

| Name | Value | |-----------------+------------------| | Manuf | | | MPN | | | Vendor | | | SKU | | | UPN | User Part Number | Where the bit is ommitted on the first instance.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/xesscorp/KiField/issues/22#issuecomment-275791585, or mute the thread https://github.com/notifications/unsubscribe-auth/AIAyuRxdCF6R1Rp3i0D0QbPfMf1-dfT2ks5rWm6cgaJpZM4LqXtf.