Specify spreadsheet currency

[Laogeodritt]

From the looks of the documentation and --help output, it doesn't seem possible to target region-specific supplier websites—this makes kicost more or less only useful to those operating in the US out-of-the-box, as the USD costs don't necessarily reflect the cost from the same supplier elsewhere and in other currencies.

Would it be possible to add as an enhancement the ability to specify a country/region, e.g. using digikey.ca instead of digikey.com, either globally or on a per-supplier basis?

I'm not sure how consistently these region-specific sites are defined across suppliers—it seems they are generally listed exhaustively (DigiKey Mouser), so it might be possible to allow the user to specify just their TLD and keep a mapping of TLD to URL in each supplier's module. Alternatively, just let users specify the specific site variant they want for each supplier, and alter the TLD or subdomain as appropriate.

(For now I'm explicitly altering the source of a local copy - not the cleanest workaround!)

[sticilface]

I would also really like this. Im UK based.

[hildogjr]

I got some interesting results changing the string 'en-US' in req.add_header('Accept-Language', 'en-US') of the kicost.py file to force countries with Euro currency. I think is the way to work this issue. Setting this, the most of the web site redirect to the local server.

[hildogjr]

This is a BIG implementation and effort of the development team. The currency of all distributor follow the ISO 4217 https://currencysystem.com/codes/, but each web site change the configuration in an independent way (eg: digikey just be the link, mouser appear to be something with a cookie, it is executed a ChangeLocalization('se','EUR') function ). I am tagging this issue and open as discussion to start a study about the web site distributors to glimpse to include some code in KiCost webscrape routines. Maybe use some parameter in the KiCost terminal call as --currency, using the ISO4217 names and standard USD could some this for users. But we need some study (manual test in each web site do see the links used).

[kevindawson]

would it not make more sense to use the API from Digi-Key

scraping is just old hat

providing a structure that enables other users to assist in localization just makes sense

[hildogjr]

I don't know that is the requirement for use the API. @kevindawson, please attack here the link (if possible) if the requirements. If just need a Digi-Key free account, may KiCost could try first of all, use the API. Other things to think is how to solve this for others distributors. I now that some you have to have a not free account.

[kevindawson]

take a look at this

https://octopart.com/api/home

[xesscorp]

Using the Octopart API comes up a lot. Here are some problems with it:

1. You need to obtain and enter a user ID to use it.
2. It only allows you a set number of API queries per month before charges kick in.
3. Altium owns it now so who knows what will happen with it.

Mouser and Digikey also have their own APIs. Digikey required a user ID to access it; Mouser didn't but restricted the pricing data to purchase quantities of 1 and 10. And there were other restrictions the last time I checked. Then there are all the other distributors that don't have a public API at all.

Web-scraping was the only approach I found in 2015 that avoided all these problems and made KiCost easy to install and use. Before going down the API route, someone will have to go through all the available APIs and determine what their current restrictions are and how they impact on getting costs and other data from the distributors. Then you can make an informed decision about which way to go.

[kevindawson]

1, Octopart are using API's to collect data 1.1 using each supplies API is a better solution 2, KiCAD is growing 3, Digikey has released a Library for KiCAD :)) 4.1 try talking to the API devs, really 4.2 request a key for your software that suites both parties 4.2.1 you want high hits low quantity costing 4.2.2 they can then track your growth 5 having a user conf file with country and currency is quit normal

[xesscorp]

1, Octopart are using API's to collect data

I'm sure they are. I'm sure they also have financial arrangements with the distributors whose APIs they use. KiCost is a FOSS tool: we have no way to pay for API access to distributor data.

1.1 using each supplies API is a better solution

Saying APIs are "better" requires you to answer the question "Better, how?" Do they provide better data? Probably not since the data scraped from the webpage should be identical to what comes through the API. Is it better for the KiCost user? Probably not since they might have to go register with each distributor to get an access key that has to be installed and then they're tracked by it. Is is better for the developers? Maybe, except we have to rebuild all the interfaces to the distributors.

2, KiCAD is growing

I don't see how this pertains to the use of an API versus web scraping.

3, Digikey has released a Library for KiCAD :))

That seems to be an effort by a few engineers within Digikey. It doesn't directly impinge on Digikey's revenue-generating activities in the way that getting access to their part data would.

4.1 try talking to the API devs, really

I contacted Digikey in July/2013 about getting access to their API. After several queries, they declined my request in Dec/2013 and told me to "utilize the Digi-Key website for continued access to our component information, availability, and pricing." And I have followed their suggestion since.

I have checked the published APIs for Digikey and Mouser since then (but not recently). They had restrictions on what and how much data they provide, required secret codes, end-user registration, etc. But perhaps their API docs are inaccurate and you do have to talk with the API developers directly. If so, please contact them and relay their information back to us. That would be super helpful! Then we can make an informed decision.

4.2 request a key for your software that suites both parties

It really needs to suit three parties: 1) the distributor, 2) the KiCost developers, and 3) the end user. I can only speak for the developers and the end users, but here's what we would need in order to move to using an API:

a. No limits on the number of API accesses per unit time period. b. All the data reported on their web page must be accessible through the API. c. No secret keys. (We're FOSS and written in Python; I don't know where we would hide them.) d. No end-user registration.

Make sure you mention these requirements when you talk to the API developers.

4.2.1 you want high hits low quantity costing

I'm not even sure what you mean by this. What we want is unrestricted access to the part information provided on their public web pages. That's what we have now using web scraping.

4.2.2 they can then track your growth

But we don't want them to track our growth. Why do they even need to do that? These are companies that sell actual products and make money from that. KiCost would provide a means to expose their parts to a group of potential purchasers. That seems like a straight-up deal that doesn't require any tracking.

5 having a user conf file with country and currency is quit normal

That isn't needed when using a browser to access their web site, so why should KiCost need one? User configuration files are so old hat.

[hildogjr]

Interesting argument and acknowledgement about KiCost history. Yes, fit with user - distributors - KiCad is the core of KiCost in my vision too.

[kevindawson]

1.1 using each supplies API is a better solution

Saying APIs are "better" requires you to answer the question "Better, how?" Do they provide better data? Probably not since the data scraped from the webpage should be identical to what comes through the API. Is it better for the KiCost user? Probably not since they might have to go register with each distributor to get an access key that has to be installed and then they're tracked by it. Is is better for the developers? Maybe, except we have to rebuild all the interfaces to the distributors.

if you live outside the us/$ zone your software produces rubbish at best, suggest you add a warning to this affect! END

[hildogjr]

Some scrape module are using import Currency_Converter and others not. Standardize this and provide a --currency option to the user may be great. But first, understand how this library works. Do it take the conversion value from same web service? If not, may be better remove it for KiCost reliability.

[hildogjr]

Digikey, as example, have a specific web where could be get the url for each country-currency version of the site. The distributors submodule could have a get_url_site(locale='US',currency='USD'). Each other distributor have this system in the page used to redirect to the country specific link. KiCost could scrape one time these pages, before the main scrape, to get the link to be used on get_part_html_tree(), prioritize the locale (that could be get even by some default library of Python or forced by --locale of --currency in the call). To an user (double) check could write the variable structs distributor_dict[*][currency], distributor_dict[*][url] and distributor_dict[*][locale], that will create a comment in the distributor correspondent label in the spreadsheet, with distributor_dict[*][url] been used on get_part_html_tree().

Use the distributor_dict[*][currency] to ajust the number formatting in the spreadsheet https://github.com/xesscorp/KiCost/pull/214 and give a user BIGGER warning if all the currency are not the same.

[hildogjr]

Started the development. May be necessary added https://pypi.python.org/pypi/pycountry/ for iso4217 and iso3166

To KiCost dependences (also remove CurrencyConverter).

[hildogjr]

The code now is able to set the distributor locale but not the currency, that appear be configured on a cookie, not in the link of Digikey.

[hildogjr]

Working for serial mode kicost -qw --include digikey -i %file% --locale IT. The problem in the multiple process mode is related with the code in line 208 of kicost.py (not wait the multprocess to finish).

[hildogjr]

This website get some useful informations https://ipstack.com/ But I don't want to depend of other server to this functionality, but may be necessary a update on pycountry (https://bitbucket.org/flyingcircus/pycountry/issues/13405/create-relationship-between-iso3166).

[hildogjr]

It works fine since #262, but have a issue related if language-currency-format. Using kicost -wi %file% --locale IT the Euro comma is interpreted not as a decimal separator.

[bobc]

I think the same issues that affect Octopart also affect vendor APIs: they don't want server overload; they don't want competitors scraping all their data; they will monitor web access (API or regular) for data collection; you are equally at whim of their company policy and could start charging, or restrict access.

I understand that Octopart make money from vendors when you follow an Octopart link to a vendor. I guess that is still true under Altium ownership, which also seems to be benign so far.

In general, in my BOM tool I cache Octopart data for a configurable time like 48 hours which could apply equally well to vendor web sites.

[hildogjr]

275 give us one step forward to Mouser country/currency configuration.