search

yogesh-desai / WebCrawlerTokopedia

It is a web crawler and scrapper for https://www.Tokopedia.com. The project scrape the product-ID, product URL and product videos present under the product images present at right bottom of the page.
18 stars 8 forks source link

Reuse buffers using sync.Pool #4

Open GaruGaru opened 5 years ago

GaruGaru commented 5 years ago

Reference #3 Using the go stdlib object pool instead of creating new buffer instances should reduce memory usage and improve GC latency

yogesh-desai commented 5 years ago

Hi @GaruGaru Could you please compare previous working and your changes in terms of benchmarking? Like how the memory usage, GC latency is being better? That will help a lot Thank you.

GaruGaru commented 5 years ago

Hi @yogesh-desai Sadly i didn't manage to run the application property because chromedp package just deprecated its package chromedp/runner in the latest version and also the package is now pointing to a new repo ( https://github.com/chromedp/chromedp ). In order to run the application i need the version of chromedp that you have locally, also a properly versioned packaged manager (such as go modules or dep) may be needed to avoid this kind of reproducibility issues

Thank you for your time !