INTEL parser

[da-ekchajzer]

Intel data could be exported from : https://ark.intel.com/content/www/us/en/ark.html

[csauge]

Seems that we need to scrap the intel web site and this seems not easy. Alternatively there are these possible sources:

• https://github.com/toUpperCase78/intel-processors => direct access to csv for some processors families but is it accurate ?
• https://www.techpowerup.com/cpu-specs/?mfgr=Intel&sort=name => very detailed info (AMD also) but need also complex scraping
• or directly wikipedia: https://en.wikipedia.org/wiki/List_of_Intel_processors

[da-ekchajzer]

Nice datasets !

• https://github.com/toUpperCase78/intel-processors has almost all data. It seems that the CPU family (architecture) is missing. We might be able to merge our dataset with this dataset based on the CPU name to complete our dataset (see : https://github.com/Boavizta/boaviztapi/blob/v0.3/boaviztapi/data/crowdsourcing/cpu_index.csv)

• https://www.techpowerup.com/cpu-specs/?mfgr=Intel&sort=name ⇒ it seems that some people have developed parsers for the website (see https://github.com/BaraaZ95/Techpowerup-scraper for example). Also in this data set the family is not given. I guess there is a link between the Codename value and the family, but cannot find it.

[csauge]

I have tried to scrap the techpowerup site and it seems to work. We need to parse all 3000 (AMD and Intel) processor pages to get such detailed information but this is easy. I think we can have a script of 50 lines of codes to extract a csv with all the CPUs. This is possible also to scrap for GPUs and SSDs. Nevertheless, I do not know if the list of CPU is exhaustive.

Here is an example for one CPU: https://www.techpowerup.com/cpu-specs/ryzen-5-3600.c2132

Header: ['Socket', 'Foundry', 'Process Size', 'Transistors', 'Die Size', 'I/O Process Size', 'I/O Die Size', 'Package', 'tCaseMax', 'Market', 'Production Status', 'Release Date', 'Launch Price', 'Part#', 'Bundled Cooler', 'Frequency', 'Turbo Clock', 'Base Clock', 'Multiplier', 'Multiplier Unlocked', 'TDP', 'FP32', 'Codename', 'Generation', 'Memory Support', 'ECC Memory', 'PCI-Express', 'Chipsets', '# of Cores', '# of Threads', 'SMP # CPUs', 'Integrated Graphics', 'Cache L1', 'Cache L2', 'Cache L3', 'Features']

Value: ['AMD Socket AM4', 'TSMC', '7 nm', '3,800 million', '74 mm²', '12 nm', '124 mm²', 'µOPGA-1331', '95°C', 'Desktop', 'Active', 'Jul 7th, 2019', '$199', '100-000000031', 'Wraith Stealth', '3.6 GHz', 'up to 4.2 GHz', '100 MHz', '36.0x', 'Yes', '65 W', '1,209.6 GFLOPS', 'Matisse', 'Ryzen 5 (Zen 2 (Matisse))', 'DDR4-3200 MHz Dual-channel', 'No', 'Gen 4, 16 Lanes(CPU only)', 'AMD 300 Series, AMD 400 Series, AMD 500 Series', '6', '12', '1', 'N/A', '64K (per core)', '512K (per core)', '32MB (shared)', 'MMX SSE SSE2 SSE3 SSSE3 SSE4A SSE4.1 SSE4.2 AES AVX AVX2 BMI1 BMI2 SHA F16C FMA3 AMD64 EVP AMD-V SMAP SMEP SMT Precision Boost 2']

[da-ekchajzer]

I don't know if it is exhaustive, but it is definitely more complete than our current dataset. If we could scrap all those processors characteristics, it would be wonderful.

The only data manipulation that should be done is the extraction of the CPU family - here zen 2 from Ryzen 5 (Zen 2 (Matisse). The list of family can be found here https://github.com/Boavizta/boaviztapi/blob/v0.3/boaviztapi/data/crowdsourcing/cpu_manufacture.csv.

What are you using for scrapping the pages ? Maybe we could do it inside a notebook in the repo to help reproduce the scrapping.

I also see that they have GPU (https://www.techpowerup.com/gpu-specs/) and SSD (https://www.techpowerup.com/ssd-specs/) specs. It will be very helpful in the future.

We should be aware that tech power might block our scrapping (see https://www.reddit.com/r/datasets/comments/y6isgi/scraping_gpus_from_techpowerupcom_using_python/).

[csauge]

I am currently using BeautifulSoup to scrap with rotating IP, changing http headers and set a random interval to send requests. It works ok but i have the captcha every 50 requests... I think i need a scrapping expert to do this faster... I can provide a csv after that.

[da-ekchajzer]

@AirLoren any thoughts from our scrapping expert ?

[csauge]

Fix is done with https://github.com/Boavizta/boaviztapi/pull/170 and https://github.com/Boavizta/boaviztapi-utils/commit/ac0ca06fa22d9ccee37fd2a29050821e8a1068c9 Ticket to close