search

sparkmicro / Ki-nTree

Fast part creation for KiCad and InvenTree
GNU General Public License v3.0
171 stars 31 forks source link

Scraping prices of components via TME API doesn't work #245

Open NightSkySK opened 3 days ago

NightSkySK commented 3 days ago

I've tried serval components creation via TME API and none of them contain supplier prices once uploaded to Inventree.

as example YAML cached result:

Category: MLCC SMD capacitors
CategoryId: 113537
Certificates:
- ROH
CustomerSymbol: ''
Description: "Capacitor: ceramic; 100nF; 50V; X7R; \xB110%; 0805"
Guarantee: null
MinAmount: 100
Multiples: 100
OfferId: null
OriginalSymbol: 0805B104K500CT
Packing:
- Amount: 4000
  Id: RL1
  IdUnit: ST
Photo: http://ce8dc832c.cloudimg.io/v7/_cdn_/DA/F2/B0/00/1/733101_1.jpg?width=640&height=480&wat=1&wat_url=_tme-wrk_%2Ftme_new_render3d.png&wat_scale=100p&ci_sign=90695e0671e587e169217987a4726ffe310da05a
Producer: WALSIN
ProductInformationPage: http://www.tme.eu/en/details/0805b104k500ct/mlcc-smd-capacitors/walsin/
ProductStatusList: []
SuppliedAmount: 0
Symbol: 0805B104K500CT
Thumbnail: //ce8dc832c.cloudimg.io/v7/_cdn_/DA/F2/B0/00/1/733101_1.jpg?width=100&height=75&q=75&ci_sign=3d2894ffeaff097857f2163591e8f4d9bae011bd
Unit: pcs
Weight: 0.034
WeightUnit: g
category: MLCC SMD capacitors
parameters:
  Capacitance: "0.1\xB5F"
  Case - inch: 0805
  Case - mm: '2012'
  Dielectric: X7R
  Manufacturer: WALSIN
  Operating temperature: "-55...125\xB0C"
  Operating voltage: 50V
  Tolerance: "\xB110%"
  Type of capacitor: ceramic
search_timestamp: 1708505132
subcategory: null

All other API which I've tested like LCSC, Mouser, and Digi-Key have prices in cached yaml:

category: Capacitors
currency: USD
detailed_description: "10 \xB5F \xB120% 6.3V Ceramic Capacitor X5R 0603 (1608 Metric)"
digi_key_part_number: 1276-1119-1-ND
manufacturer: Samsung Electro-Mechanics
manufacturer_part_number: CL10A106MQ8NNNC
parameters:
  Applications: General Purpose
  Capacitance: "10 \xB5F"
  ECCN: EAR99
  Failure Rate: '-'
  Features: '-'
  Height - Seated (Max): '-'
  Lead Spacing: '-'
  Lead Style: '-'
  Mounting Type: Surface Mount, MLCC
  Operating Temperature: "-55\xB0C ~ 85\xB0C"
  Package / Case: 0603 (1608 Metric)
  Packaging: Cut Tape (CT)
  Ratings: '-'
  Size / Dimension: 0.063" L x 0.031" W (1.60mm x 0.80mm)
  Temperature Coefficient: X5R
  Thickness (Max): 0.035" (0.90mm)
  Tolerance: "\xB120%"
  Voltage - Rated: 6.3V
pricing:
  1: 0.11
  10: 0.079
  50: 0.0452
  100: 0.035
  500: 0.02638
  1000: 0.02072
primary_datasheet: https://mm.digikey.com/Volume0/opasdata/d220001/medias/docus/609/CL10A106MQ8NNNC_Spec.pdf
primary_photo: https://mm.digikey.com/Volume0/opasdata/d220001/medias/images/2537/Ceramic-Capacitor-CL-Series.jpg
product_description: CAP CER 10UF 6.3V X5R 0603
product_url: https://www.digikey.com/en/products/detail/samsung-electro-mechanics/CL10A106MQ8NNNC/3886777
search_timestamp: 1708598201
subcategory: Ceramic Capacitors
eeintech commented 3 days ago

Hello @NightSkySK

I am not very familiar with TME's API, is it supposed to return prices by default, or does it need some kind of parameter in the call to add them to the response? I would appreciate if you'd find more info regarding this and we could try to add it to a future release 😃

NightSkySK commented 2 days ago

I investigated a bit kintree\search\tme_api.py so fetching product info from TME API is running in three or four steps:

First two are working without any problem, only /Products/GetPrices causing error [INFO] Warning: API data download failed (HTTP Error)

From my debug investigation. error is result of kintree\common\tools.py line 99 url_data = urllib.request.urlopen(url)

            url_data = urllib.request.urlopen(url)
            data = url_data.read()
            data_json = json.loads(data.decode('utf-8'))
            return data_json
    except (socket.timeout, requests.exceptions.ConnectTimeout, requests.exceptions.ReadTimeout):
        cprint(f'[INFO]\tWarning: {filetype} download socket timed out ({timeout}s)', silent=silent)
    except (urllib.error.HTTPError, requests.exceptions.ConnectionError):
        cprint(f'[INFO]\tWarning: {filetype} download failed (HTTP Error)', silent=silent)

and rise requests.exceptions.ConnectionError

I've checked the request body against TME API Manual and all seems to be correct. I've tried, also to replace /Products/GetPrices with Products/GetPricesAndStocks endpoint which are very similar but also the same error appears in the same moment.

I hope @T0jan will find some time to have a look on this case as I couldn't find root cause of this error.