Closed curita closed 2 months ago
@curita
Skipping the coverage of certain patterns may not solve the problem, considering that for data not well structured, we may get keys unexpected from our targets. Regarding the stats limit in Scrapy Cloud, maybe changing what HubStorageStatsCollector uploads as stats is a more reliable alternative and is specific to the platform, so we don't introduce new Zyte specific code to Spidermon.
The alternative of SPIDERMON_LIST_FIELDS_COVERAGE_LEVELS
looks promising in my point of view. We can set the "depth" where we want to check the coverage, so with an item like:
{
"field_1": "some_value",
"field_2": "some_value",
"field_3": {
"field_3_1": "some_value",
"field_3_2": "some_value",
}
}
If SPIDERMON_LIST_FIELDS_COVERAGE_LEVELS = 1
, so the stats will be like:
{
"spidermon_field_coverage/dict/field1/": "some_value",
"spidermon_field_coverage/dict/field2/": "other_value",
"spidermon_field_coverage/dict/field3/": "other_value",
}
If SPIDERMON_LIST_FIELDS_COVERAGE_LEVELS = 2
, so the stats will be like:
{
"spidermon_field_coverage/dict/field1/": "some_value",
"spidermon_field_coverage/dict/field2/": "other_value",
"spidermon_field_coverage/dict/field3/": "other_value",
"spidermon_field_coverage/dict/field3/field_3_1": "other_value",
"spidermon_field_coverage/dict/field3/field_3_2": "other_value",
}
What do you think???
I like the depth approach the most 👍 Mauricio implemented SPIDERMON_DICT_FIELDS_COVERAGE_LEVELS
as a separate setting in his PR, though both SPIDERMON_LIST_FIELDS_COVERAGE_LEVELS
and SPIDERMON_DICT_FIELDS_COVERAGE_LEVELS
could be unified into a single setting if we consider so.
Background
Currently, the coverage monitor tracks and reports the coverage of all fields, including nested fields (i.e., keys inside dictionary values specifically). It follows all nested field levels.
This sometimes isn't desired, especially for dictionaries that have non-standardized keys. The coverage tracking can stay the same if there's a fixed schema for those dictionaries or too many nested levels. This is particularly troublesome when running spiders in ScrapyCloud, as there's a hard limit on the stats's storage size.
Alternatives
SPIDERMON_LIST_FIELDS_COVERAGE_LEVELS
, but for dictionaries.<key: field count>
pair could be stored as a separate entry in a ScrapyCloud collection to avoid triggering its size limitation.All of them could be viable and coexist as they address different parts of the problem.
Let me know what you think!