search

hildogjr / KiCost

Build cost spreadsheet for a KiCad project.
MIT License
493 stars 97 forks source link

Octopart.com TypeError: 'NoneType' object is not iterable #524

Closed craigarno closed 1 year ago

craigarno commented 1 year ago

Issue / Problem report

I'm trying to get "Octopart", "LCSC", and "RS Components" to populate. Every part of my Config.yaml is enabled, except Octopart, and appears to be working. Problems occur when I enable Octopart with either of the client values below.

   Octopart:
    # Octopart API Key
    key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    # API level: 3 or 4
    # level: 4
    # The extended API is for the Pro plan
    # extended: false
    # Only enabled if the key is defined
    enable: true
    # Directory for the APIs caches
    cache_path: ~/.cache/kicost/Octopart

I have my free Octopart Key installed, replaced below with xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. When I enable Octopart, this is what happens. I tried both "Client" values (ID and Secret) and the same thing happens with either: image

  1. C:\Users\craig\Desktop\555\KiCAD>kicost --version KiCost v1.1.11

  2. DEBUG:Distributors solved {'tme'}, remaining {'arrow', 'rs', 'lcsc'} (kicost.distributors - log.py:115) DEBUG:Considering: Octopart ['arrow', 'digikey', 'farnell', 'lcsc', 'mouser', 'newark', 'rs', 'tme'] (kicost.distributors - log.py:115)

    • Octopart [api] (https://octopart.com/) DEBUG:# Getting part data from Octopart... (kicost.distributors - log.py:115) DEBUG:Queries 3 (kicost.distributors - log.py:115) DEBUG:Cached entries 0 (kicost.distributors - log.py:115) DEBUG:URL http://octopart.com/api/v4/rest/parts/match query: (kicost.distributors - log.py:115) DEBUG:queries=[{"reference": "1", "mpn": "2N3906"}, {"reference": "2", "mpn": "10K"}, {"reference": "5", "mpn": "CA0555EX"}]&apikey=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx (kicost.distributors - log.py:115) DEBUG:{ "errors": [ { "message": "No app for token xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx not found" } ], "data": null } (kicost.distributors - log.py:115) DEBUG:Status Code: <200> (kicost.distributors - log.py:115) Progress: 0%| | 0/3 [00:00<?, ?part/s]Traceback (most recent call last): File "C:\Program Files\KiCad\6.0\bin\Lib\runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Program Files\KiCad\6.0\bin\Lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Program Files\KiCad\6.0\bin\Scripts\kicost.exe__main.py", line 7, in File "C:\Program Files\KiCad\6.0\bin\Lib\site-packages\kicost__main__.py", line 487, in main main_real() File "C:\Program Files\KiCad\6.0\bin\Lib\site-packages\kicost\main.py", line 477, in main_real kicost(in_file=args.input, eda_name=args.eda, File "C:\Program Files\KiCad\6.0\bin\Lib\site-packages\kicost\kicost.py", line 251, in kicost query_part_info(parts, dist_list, currency) File "C:\Program Files\KiCad\6.0\bin\Lib\site-packages\kicost\kicost.py", line 78, in query_part_info get_dist_parts_info(parts, dist_list, currency) File "C:\Program Files\KiCad\6.0\bin\Lib\site-packages\kicost\distributors\init__.py", line 50, in get_dist_parts_info distributor_class.get_dist_parts_info(parts, dist_list, currency) File "C:\Program Files\KiCad\6.0\bin\Lib\site-packages\kicost\distributors\distributor.py", line 144, in get_dist_parts_info solved = api.query_part_info(parts, list(remaining), currency) File "C:\Program Files\KiCad\6.0\bin\Lib\site-packages\kicost\distributors\api_octopart.py", line 448, in query_part_info api_octopart.get_part_info(unsolved[slc]) File "C:\Program Files\KiCad\6.0\bin\Lib\site-packages\kicost\distributors\api_octopart.py", line 211, in get_part_info for r in results: TypeError: 'NoneType' object is not iterable Progress: 0%| | 0/3 [00:00<?, ?part/s]

C:\Users\craig\Desktop\555\KiCAD>

  1. Write the command used to call KiCost (or the graphical interface configuration); kicost -w -i 555.xml --debug 8

  2. One BoM to reproduce the error (with the EDA version). 555.zip

set-soft commented 1 year ago

Hi @craigarno ! The implemented API is Octopart, looks like they moved to Nexar, which used a very different access mechanism. I don't think a Nexar ID can be used to access the Octopart API 3 or 4.

craigarno commented 1 year ago

I see this information on OctoPart:

APIv3 user? As of June 30, 2022 APIv3 access has been disabled. If you are looking for your APIv3 key you no longer have access to it. In order to access supply chain data again, you must sign up for the Nexar API.

Looking for the Octopart API? The Octopart API is now part of the Nexar API. The Nexar API, built on GraphQL, will give you access to all of the data available from the Octopart API in addition to design and manufacturing data. Whether you are a new or existing API user, upgrading to the Nexar API will future-proof your business by providing you with data from all across the industry.

If the OctoPart API is part of the Nexus API, how can KiCost access the Nexus implementation of Octopart? Longer term, Nexus seems to have documentation for Python access to their database. https://docs.nexar.com/ https://github.com/NexarDeveloper/nexar-examples-py

KiCost is giving suppliers direct access to the market they need and want, without advertising cost/effort. I should think these suppliers would be delighted to support KiCost efforts.

Until we can sort this out, I have to leave Octopart disabled in my config so the rest of it can run to completion. While frustrating, KiCost is still a powerful tool even without Octopart.

set-soft commented 1 year ago

They don't mention API v4. So I guess it still work, but they won't give you a new key for it.

I don't have time to implement a new API right now. About "LCSC" and "RS Components": Last time I checked they didn't provide a usable API. BTW Arrow has an API, but quite useless, one of its terms of use is You will not maintain any cached data retrieved using the Arrow APIs ... ridiculous, I can't even run regression tests with this.

craigarno commented 1 year ago

All vendors with the possible exception of LCSC appear filled in without OctoPart credentials. I suspect this is from KitSpace. KitSpace appears to pay for API v4 OctoPart access and asks for donations to continue this service. I'm considering how much I might donate to KitSpace (I am a person of modest means), just like I've donated to KiCAD and LibreOffice projects. I didn't see any place to donate to KiCost. KitSpace donations should take pressure off the immediate need to develop a Nexus API. My expectations suggest maybe a year (thinking KitSpace donation amount) before a more robust Nexus API renders KitSpace subscriptions unnecessary.

I've been using KiCost for ~1 week and am impressed with the responsive and well thought out spreadsheet it produces for Excel after part scraping. KiCost doesn't appear to produce an OpenDocument / ISO-IEC 26300 Spreadsheet yet, which is useful for longer term storage of project information/documentation. In particular, I discovered by accident that the Cat# column takes me exactly where I need to go at the part vendor's site. Good documentation for each column can be found by hovering over the column label. All of this helps me clean up my BOM while dealing with serious supply chain availability & cost issues. I also make an effort to use vendors who are based in the country where I live and design with the assumption that this will help when trying to resolve issues as they need to be addressed without having to resort to heavy ITAR type restrictions.

KiCost allows me to be nimble and responsive to supply chain issues affecting my design. As an engineer who doesn't have a lot of time to work on supply issues while I work on design issues, this is important, maybe even critical to survival in today's Covid fraught landscape.

A big Thank You to all the people who donated efforts to bring KiCost into existence and are continuing to support keeping KiCost alive.

set-soft commented 1 year ago

Thanks @craigarno a note: now KiCost not only supports alternatives to KitSpace, but also caches the results from it. So I hope KitSpace will become less busy as people starts using the newer KiCost versions.

set-soft commented 1 year ago

Now the Nexar API is implemented. Not yet released, you need the git code. I'm looking for beta testers ;-)

craigarno commented 1 year ago

I added Nexar to my config. I can't tell if it did anything and it did finish quickly:

For anybody else stumbling onto this:

KiCost config documentation is here

Installing the development version of KiCost, open a KiCAD cmd window (in Windows) and execute

pip install git+https://github.com/hildogjr/KiCost.git

Verify the development version is installed:

>kicost --version
KiCost v1.1.13

Go here to create a Nexar API Account to get your credentials for your config.yaml file.

After running KiCost v1.1.13 there appear to be 3 new vendors [Arrow, LCSC, RS Components] in the spreadsheet from v1.1.12. No indication of errors. The whole collection of vendors I see from v1.1.13 are: Arrow, Digi-Key, Farnell, LCSC, Mouser, Newark, RS Components, TME

BTW, thanks @set-soft, it's nice to have 3 more purchase sources showing up!

This is all I can report for now.

set-soft commented 1 year ago

Hi @craigarno ! Thanks for testing the new Nexar API. The API is working fast and we send up to 20 searches at once. If we get 0 hits for a component with a specific manufacturer we are retrying it without manufacturer (in case the name doesn't match), I guess this is because you got 2 searches. The Nexar API is equivalent to the Octopart Pro API, you also get specs for the components, so you'll see descriptions and more data when looking at the comments of the cells with the part number for a distributor.

craigarno commented 1 year ago

@set-soft when I run it this morning on a different machine I get:

C:\Users\craig\555\KiCAD>kicost -w -i "555.xml" --eda kicad WARNING:(WC018) Unknown API Nexar (kicost - log.py:107)

and [Arrow, LCSC, RS Components] are missing from 555.xlsx file. Timestamp shows this file was updated/modified.

Here is the pip install output from an administrator KiCad 6 CMD window:

C:\Users\craig\Documents\KiCad\6.0>pip install git+https://github.com/hildogjr/KiCost.git
Collecting git+https://github.com/hildogjr/KiCost.git
  Cloning https://github.com/hildogjr/KiCost.git to c:\users\craig\appdata\local\temp\pip-req-build-z12wcmav
  Running command git clone --filter=blob:none --quiet https://github.com/hildogjr/KiCost.git 'C:\Users\craig\AppData\Local\Temp\pip-req-build-z12wcmav'
  Resolved https://github.com/hildogjr/KiCost.git to commit ff536b7d1f469a09d6ae59ca4bdb46d1a0419cfb
  Preparing metadata (setup.py) ... done
Requirement already satisfied: beautifulsoup4>=4.3.2 in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from kicost==1.1.13) (4.11.1)
Requirement already satisfied: lxml>=3.7.2 in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from kicost==1.1.13) (4.9.1)
Requirement already satisfied: XlsxWriter>=0.7.3 in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from kicost==1.1.13) (3.0.3)
Requirement already satisfied: tqdm>=4.30.0 in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from kicost==1.1.13) (4.64.0)
Requirement already satisfied: requests>=2.18.4 in c:\program files\kicad\6.0\bin\lib\site-packages (from kicost==1.1.13) (2.28.1)
Requirement already satisfied: validators>=0.14.2 in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from kicost==1.1.13) (0.20.0)
Requirement already satisfied: wxPython>=3.0.2 in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from kicost==1.1.13) (4.1.1)
Requirement already satisfied: colorama in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from kicost==1.1.13) (0.4.5)
Requirement already satisfied: pyyaml in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from kicost==1.1.13) (6.0)
Requirement already satisfied: kicost-digikey-api-v3 in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from kicost==1.1.13) (0.1.1)
Requirement already satisfied: soupsieve>1.2 in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from beautifulsoup4>=4.3.2->kicost==1.1.13) (2.3.2.post1)
Requirement already satisfied: certifi>=2017.4.17 in c:\program files\kicad\6.0\bin\lib\site-packages (from requests>=2.18.4->kicost==1.1.13) (2022.9.24)
Requirement already satisfied: charset-normalizer<3,>=2 in c:\program files\kicad\6.0\bin\lib\site-packages (from requests>=2.18.4->kicost==1.1.13) (2.1.1)
Requirement already satisfied: idna<4,>=2.5 in c:\program files\kicad\6.0\bin\lib\site-packages (from requests>=2.18.4->kicost==1.1.13) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\program files\kicad\6.0\bin\lib\site-packages (from requests>=2.18.4->kicost==1.1.13) (1.26.12)
Requirement already satisfied: decorator>=3.4.0 in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from validators>=0.14.2->kicost==1.1.13) (5.1.1)
Requirement already satisfied: pillow in c:\program files\kicad\6.0\bin\lib\site-packages (from wxPython>=3.0.2->kicost==1.1.13) (9.2.0)
Requirement already satisfied: numpy in c:\program files\kicad\6.0\bin\lib\site-packages (from wxPython>=3.0.2->kicost==1.1.13) (1.23.3)
Requirement already satisfied: six in c:\program files\kicad\6.0\bin\lib\site-packages (from wxPython>=3.0.2->kicost==1.1.13) (1.16.0)
Requirement already satisfied: tldextract in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from kicost-digikey-api-v3->kicost==1.1.13) (3.3.1)
Requirement already satisfied: python-dateutil in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from kicost-digikey-api-v3->kicost==1.1.13) (2.8.2)
Requirement already satisfied: pyopenssl in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from kicost-digikey-api-v3->kicost==1.1.13) (22.0.0)
Requirement already satisfied: setuptools in c:\program files\kicad\6.0\bin\lib\site-packages (from kicost-digikey-api-v3->kicost==1.1.13) (58.1.0)
Requirement already satisfied: inflection>=0.3.1 in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from kicost-digikey-api-v3->kicost==1.1.13) (0.5.1)
Requirement already satisfied: cryptography>=35.0 in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from pyopenssl->kicost-digikey-api-v3->kicost==1.1.13) (37.0.4)
Requirement already satisfied: filelock>=3.0.8 in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from tldextract->kicost-digikey-api-v3->kicost==1.1.13) (3.7.1)
Requirement already satisfied: requests-file>=1.4 in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from tldextract->kicost-digikey-api-v3->kicost==1.1.13) (1.5.1)
Requirement already satisfied: cffi>=1.12 in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from cryptography>=35.0->pyopenssl->kicost-digikey-api-v3->kicost==1.1.13) (1.15.1)
Requirement already satisfied: pycparser in c:\users\craig\documents\kicad\6.0\3rdparty\python39\site-packages (from cffi>=1.12->cryptography>=35.0->pyopenssl->kicost-digikey-api-v3->kicost==1.1.13) (2.21)

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: python.exe -m pip install --upgrade pip
set-soft commented 1 year ago

Hi @craigarno ! Did you verify the version? Are you sure you are running the copy installed using the above pip command? (and not an older copy)

craigarno commented 1 year ago

On the problematic system, I'll uninstall and reinstall later. Yes, I did a pull, pip install git+https://github.com/hildogjr/KiCost.git for install and “version” reports 1.1.13. Unless 1.1.13 could be pointing to an older repo version, which is why my next step is to uninstall, reinstall.

craigarno commented 1 year ago

I performed an uninstall/install of KiCost on the problematic system, and now KiCost works on that system. Both prior to uninstall/install and after kicost --version said 1.1.13.

Being the not so trusting sort, I tried installing KiCost on a 3rd development system, also Windows 11 (the problematic system is Windows 10 which is now working).

I get this image

I did finally make it install, but had to 

python -m pip install --upgrade pip
pip install tqdm
pip install bs4
pip install xlsxwriter
pip install validators

Before I could finally get:

>kicost --version
KiCost v1.1.13

image

KiCost now works on the 3rd system, which didn't have KiCost previously installed.

set-soft commented 1 year ago

I performed an uninstall/install of KiCost on the problematic system, and now KiCost works on that system. Both prior to uninstall/install and after kicost --version said 1.1.13.

I suspect you had a mix of files from different versions

Being the not so trusting sort, I tried installing KiCost on a 3rd development system, also Windows 11 (the problematic system is Windows 10 which is now working).

Thanks for reporting, I patched the code so the setup.py can run even when TQDM isn't installed.