search

mnayal / basic-scraper-engine

Other
0 stars 0 forks source link

Misc improvements #3

Closed cavandavid closed 1 year ago

cavandavid commented 1 year ago
  1. Not sure why you had to use cheshire to convert response string to clojure map and then later do walk/keywordize-keys. When you could have just used :as json, and avoided the calls to cheshire/decode and walk/keywordize-keys 
    (:body (http/get "https://shopee.sg/api/v4/recommend/recommend" {:accept :json :as :json :query-params {"bundle" "category_landing_page", "catid" "11012453", "limit" 60, "offset" 60}}))
>>
{:bff_meta nil, :error 0, :error_msg nil, :data {:update_time 1682602641, :ve

This introduced two extra processing of the response body(One during cheshire/decode, other during walk)

  1. There is no historicity, Each time a new CSV gets created erasing previous data
mnayal commented 1 year ago

You are right, I should have used decoder from clj-http for json. I was thinking about not parsing and keywordizing the whole response body as most of it is not needed but yes in any way I would need to decode to json from the response string.

I have added date-time stamp to the generated csv files for historicity. Good catch!