The FastURLFilter implementation currently in SC does not use the same format as the one in Nutch. In order to keep the filtering logic as close as possible when running in production at CommonCrawl, we will add a compatible implementation which can also refresh its data and load it from S3. By default, it will load the file from the resources in the jar.
The FastURLFilter implementation currently in SC does not use the same format as the one in Nutch. In order to keep the filtering logic as close as possible when running in production at CommonCrawl, we will add a compatible implementation which can also refresh its data and load it from S3. By default, it will load the file from the resources in the jar.