search

JonathanVusich / pcpartpicker-scraper

Web scraper for the PCPartPicker website
10 stars 6 forks source link

Host raw json data on docs/ #4

Open damageboy opened 5 years ago

damageboy commented 5 years ago

I think it would make sense to host the json files directly on GH / docs/ folder since the traffic would use gzip encoding anyway (and keep downloads small/fast)

That would allow to import the json files directly for usage in google sheets...

I've made an experiment out of this and seems to work well, example URL: memory.json

image

Thoughts?

damageboy commented 5 years ago

ping?

JonathanVusich commented 5 years ago

Sorry for the delay, I was AFK on vacation for a while. To answer your question, yes, that sounds like a good idea. I just did the compression in order to try and minimize the amount of bandwidth used. But it sounds like that is not even necessary now. I don't have time to do this RN, but you are free to make the necessary changes. :)

JonathanVusich commented 5 years ago

Closing this issue as the requested change has been made.

leo60228 commented 5 years ago

@JonathanVusich I don't see it. Did you maybe forget to push?

JonathanVusich commented 5 years ago

The raw JSON is now being hosted, just within an HTML file. It will take some more re-working to host the raw JSON files.

damageboy commented 5 years ago

If you don't mind I will submit a PR for at.least generation both the raw JSON as well as the current packaging approach.

The raw JSON via https very useful for embedding into pivot tables on Google sheets ;)

On Thu, 31 Oct 2019 at 18:20 Jonathan Vusich notifications@github.com wrote:

The raw JSON is now being hosted, just within an HTML file. It will take some more re-working to host the raw JSON files.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/JonathanVusich/pcpartpicker-scraper/issues/4?email_source=notifications&email_token=AAA6WIVRPDMM2MGGC42Q3ALQRMAWLA5CNFSM4IEISE62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECYMFRI#issuecomment-548455109, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA6WIUMMLKFXMCLYXXLGBDQRMAWLANCNFSM4IEISE6Q .

-- 

Shechter.
JonathanVusich commented 5 years ago

Sounds good to me. 👍 I won't merge it though until an update to the API is made.

JonathanVusich commented 4 years ago

@damageboy Feel free to submit the PR at any time. 😄

damageboy commented 4 years ago

I actually started working on this, but the entire parser needs an update to chase changes on their side I suppose.

JonathanVusich commented 4 years ago

Yes, I can handle that part once you submit your changes. I don't think it will be very tricky.