xC0uNt3r7hr34t
commented
1 year ago I suspect the 'test_nested_process_search_pq' test failed due to the change to the domain name and it seems to be coded to use url.address in the test file. The second test, 'test_get_query_text_handles_same_field_dv' failed due to the removal of this combined_query logic since dv queries are now handled differently and no longer merged in this way. The third test, 'test_get_query_text_handles_different_fields_dv' seems to have failed for similar reasons due to the change in logic around combined queries for DV
rc-csmith
Changes include:
Add error processing to evenly split up queries that are approaching the maximum query limit of characters. An approach of using 75% of the character limit is used in order to account for any additional characters outside of the terms themselves and if any additional columns are added to outputs in the future. This may need to be adjusted later if major changes to columns outputs are made. multi-threading was enabled for all query types. When DV is being used the logic was moved to handle kicking off the requests every 60 seconds in order to avoid race conditions of the logic from queued up queries in the concurrent.futures method. processing was added to group queries for both PQ and DV by tags. For PQ, this increases the efficiency as before all queries ran 1 at a time. For DV, this can decrease efficiency due to not bundling queries in groups of 10, but it greatly improves the output results with more easily identifying which result is associated with which tag. These changes primarily affect the use of definition files.
Additional thoughts on DV processing is to add an option to parse this way vs letting DV queries run more efficiently and bulking queries together, where the output isn't as clear to what tag the result is a match for. I'm in favor of better output, but due to the rate limits for DV, it does significantly slow down queries for definition files
Resolves #114 & #110