Open brunopace opened 5 years ago
Given that writing (crawled data) to disk (before writer reads it) might be inefficiently consuming resources, it would be necessary to guarantee that we monitor the bottlenecks properly. It may be somewhere unexpected!
Read into https://docs.python.org/3/library/profile.html
Description
Given that writing (crawled data) to disk (before writer reads it) might be inefficiently consuming resources, it would be necessary to guarantee that we monitor the bottlenecks properly. It may be somewhere unexpected!
Comments
Read into https://docs.python.org/3/library/profile.html