Support memory_type property on memory

[damageboy]

First of all, I'd like to thank you for all the hard work you must have put into this.

From the little that I see there is no code available for the backend, so I can't submit a PR or work on this myself...

I'd love to have the ability to get memory_type property on the Memory object that could differentiate the memory type a-la the current pcpartpicker options:

Let me know if there is any way I could help, even if it means working on the backend...

[JonathanVusich]

No problem! It's been fun. The backend unfortunately is quite tricky to deal with since there is no official API for the website and can break very often whenever there is a main feature update on the website. So now I maintain a dataset that is updated fairly regularly than can then be pulled from using this Python package. Some of the documentation for the code and for the README is now out of date, unfortunately. I will try to get to it in the next few weeks, but I am rather busy right now. If you would like to do that, I would be very grateful.

I will have a look at the backend and see if I can incorporate that. It would be handy to include.

[JonathanVusich]

It turns out that at this time it is not really practical to add functionality to retrieve this information. How important is this to you?

[damageboy]

It's hard to put a figure on it....

I'm trying to make this API work for me in a horrible horrible spreadsheet I'm building for purchasing a fairly large cluster of computers...

I know I can filter for ECC memory through that site's search functionality, and I really wouldn't mind putting the effort (or at least try) with the backend...

If you're willing to make me a collaborator, or make it opensource I'd love to give it a try...

[JonathanVusich]

I went ahead and opened up the pcpartpicker-scraper repo. Feel free to modify it and make changes. I will need to set up CI to ensure that nothing is broken though before I will accept any PRs.

[damageboy]

Thanks! I'm running now my soon-to-be-a-PR version in the background scraping, with a specific memory+ECC support baked in.

It's ugly and probably needs some more love, but the basic idea of how to get additional properties into the different part classes is there...

I think it would be rather easy to generalize for more filter / data like this (motherboards, as a case in point):

[damageboy]

I think I have it mostly working, here's a sample ipython session with a hacked up API that reads from a local web-server and supports the new property:

IPython 7.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pcpartpicker                                                                                                                                                                                                                                                                 

In [2]: from functional import seq                                                                                                                                                                                                                                                          

In [3]: a = pcpartpicker.API()                                                                                                                                                                                                                                                              

In [4]: mem = a.retrieve("memory")["memory"]                                                                                                                                                                                                                                                

In [5]: seq(mem).first()                                                                                                                                                                                                                                                                    
Out[5]: Memory(brand='ADATA', model='AD2S800B2G5-R 2 GB', module_type='DDR2', speed=ClockSpeed(cycles=800000000), form_factor='200-pin SODIMM', number_of_modules=1, module_size=Bytes(total=2000000000), price_per_gb=<Money: 0.00 USD>, color='Green', cas_timing=5, error_correction='Non-ECC / Unbuffered', price=<Money: 0.00 USD>)

In [6]: seq(mem).group_by(lambda m: m.error_correction).map(lambda g: (g[0], len(g[1])))                                                                                                                                                                                                    
Out[6]: [('Non-ECC / Unbuffered', 5811), ('ECC / Unbuffered', 154), ('ECC / Registered', 386), ('Non-ECC / Registered', 5)]

[damageboy]

Would you like me to submit a PR?

I have roughly the following changes:

• Use multiprocessing to speed up scraping
• Include a hacky error_correction property in parts.py

[JonathanVusich]

Yes, please do. :)