search

hildogjr / KiCost

Build cost spreadsheet for a KiCad project.
MIT License
519 stars 98 forks source link

Switch to octopart query #433

Closed sebo83910 closed 3 years ago

sebo83910 commented 3 years ago

Hello,

I tried to switch to an Octopart query within the tool. So I did the following changes in the kicost.py file:

When I run the tool, I get: File "/Users/sebo/Projects/python/clipbom/env/lib/python3.9/site-packages/kicost/distributors/api_octopart.py", line 195, in query_part_info dist_xlate = distributors_modules_dict['api_octopart']['dist_translation'] KeyError: 'api_octopart' Progress: 0%|

What am I missing? thanks.

Sebastien

hildogjr commented 3 years ago

Appear that, for some reason (may be a bug or missed change), init_dist_dict() of api_octopart.py was called to create such definition.

sebo83910 commented 3 years ago

In the init.py in distributors, there are references to api_partinfo_kitspace. Aside from changing the import call to api_octopart, I guess there may be a bug at line 37:

distributors_modules_dict['api_partinfo_kitspace'] = {'handle': api_octopart}
should be:
distributors_modules_dict['api_octopart'] = {'handle': api_octopart}
hildogjr commented 3 years ago

I think you catch the issue. Tell me if still missing something.

Several portion of KiCost code could be re-write to better integration with different APIs/scrapes/... or the other functions. I started this some time ago (before it wasn't possible even run with such modifications). But still missing some work.

sebo83910 commented 3 years ago

So now I get a 401 http response. After some digging, I found out the api has changed to v4. --> https://octopart.com/api/v4/getting-started I guess the api_octopart must be adapted to these changes.

hildogjr commented 3 years ago

You are right. It will need some changes at function o line 84 and possible its call on 113, 200.

sebo83910 commented 3 years ago

Hi, just a little follow-up here. I managed to get an APIkeyv3 from Octopart. So the code execution went further. But it still fails :

http://octopart.com:80 "GET /api/v3/parts/match?queries=%5B%7B%22reference%22:%200,%20%22mpn%22:%20%22LTM8048EY%23PBF%22%7D,%20%7B%22reference%22:%201,%20%22mpn%22:%20%22LTM8054IY%23PBF%22%7D%5D&apikey=&include%5B%5D=specs&include%5B%5D=datasheets HTTP/1.1" 200 4905 Progress: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 3.03part/s]Creating the 'Classeur2.xlsx' spreadsheet... Progress: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 3.00part/s] Writing the global part information... Sorting the distributors... Writing the distributor part information... [] Traceback (most recent call last): File "/Users/sebo/Projects/python/clipbom/env/bin/kicost", line 8, in <module> sys.exit(main()) File "/Users/sebo/Projects/python/clipbom/env/lib/python3.9/site-packages/kicost/__main__.py", line 306, in main kicost(in_file=args.input, eda_name=args.eda, File "/Users/sebo/Projects/python/clipbom/env/lib/python3.9/site-packages/kicost/kicost.py", line 300, in kicost dico[part.fields.get('manf#')][dist_s]['url'] = part.url[dist_s] KeyError: 'digikey'

I guess this one will be a little diffcult to debug. It looks like "parts" is empty when used in spreadsheet.py. So I guess, somehow, the api_octopart methods are not correctly parsing the json information gathered after the url request.

In the method "query_part_info", the following line returns an empty dict: distributors_octopart = [d for d in distributors if distributors[d]['type']=='web' and distributors_modules_dict['api_octopart'].get('dist_translation')]

hildogjr commented 3 years ago

It could be some "translation" issue. It use the definition of line 68 (Octopart, if I remember, respond "Digi-Key" but KiCost uses "digikey" or vice-versa...).

set-soft commented 3 years ago

Hi @sebo83910 !

Did you check that the API was enabled? ('enabled' value == True in the dict). Anyways, I'm doing some clean-up to the scrape API, currently in this branch: https://github.com/xesscorp/KiCost/tree/distributors_cleanup (hopefully in the trunk soon).

You shouldn't need to modify the code, just define the KICOST_OCTOPART_KEY_V3 environment variable with the corresponding key. I don't have a key (last time I checked they offered one very limited for free, but asked for a credit card number). I checked that the distributors_octopart get correct initialization. But if anybody wants to help I can try to solve the problems.

hildogjr commented 3 years ago

Nice, @set-soft !

@sebo83910 , I didn't use Windows for some time but this tutorial may be handy https://www.digitalcitizen.life/simple-questions-what-are-environment-variables/

On Linux is you have to add KICOST_OCTOPART_KEY_V3=%my_key% on your .bashrc file at your $HOME.

sebo83910 commented 3 years ago

Hi @set-soft ,

No I didn't see this. I'll check. However, I see octopart is moving on to a new version of it's API. Version 4 is now organized around GraphQL. I believe V3 will soon be obsolete and unavailable.

@hildogjr Wouldn't be easier to add a input switch on the CLI instead of using an environment variable?

S/

Hi @sebo83910 !

Did you check that the API was enabled? ('enabled' value == True in the dict). Anyways, I'm doing some clean-up to the scrape API, currently in this branch: https://github.com/xesscorp/KiCost/tree/distributors_cleanup (hopefully in the trunk soon).

You shouldn't need to modify the code, just define the KICOST_OCTOPART_KEY_V3 environment variable with the corresponding key. I don't have a key (last time I checked they offered one very limited for free, but asked for a credit card number). I checked that the distributors_octopart get correct initialization. But if anybody wants to help I can try to solve the problems.

set-soft commented 3 years ago

@hildogjr Wouldn't be easier to add a input switch on the CLI instead of using an environment variable?

IMHO we can't add a command line option until we have a working API. Otherwise people will start to complain about problems when using the switch.

I think the environment variable is good for testing, and could also avoid the need of always passing the option in the future.

BTW KitSpace API works with GraphQL.

In the past @kasbah kindly provided a proxy to test Octopart API. Perhaps it can be done again for v4 ... lets see if @kasbah replies to it ;-)

kasbah commented 3 years ago

I've not tried the v4 Octopart API yet and don't think we are subscribed to it. There is a free tier on it though: https://octopart.com/api/v4/register

set-soft commented 3 years ago

Hi @sebo83910 !

Look at issue #492 we managed to get it working using an API v4 key. I'll add some configuration options. If you can test it please tell me.

set-soft commented 3 years ago

Octopart API should be working now. Can be enabled with arguments and with environment variables. And we have a test case.