search

platonai / exotic-amazon

A complete solution to crawl amazon at scale completely and accurately.
143 stars 46 forks source link

The program is stuck on the dubug message #9

Open wfh1300 opened 1 year ago

wfh1300 commented 1 year ago

The web page is stuck. And The terminal occasionally prints debug messages. DEBUG a.p.s.r.a.schedule.ScentRestMonitor - Try executing top N tasks ... f50a9223a68608e876a32ebc4f0e67e

platonai commented 1 year ago

It means that all tasks have been completed.

wfh1300 commented 1 year ago

But there are only 2,249 ASINs, which is obviously not the entire amazon data. And the json folder is just 239MB.

platonai commented 1 year ago

Of course, our program will not crawl the entire amazon.com by default, which would be a very huge project. To start a real world data project, you need to make some simple configuration changes. Usually, each user has some special business logic, which requires a little code modifications to meet the business needs.