crawler-commons / url-frontier

API definition, resources and reference implementation of URL Frontiers
Apache License 2.0
46 stars 12 forks source link

Set an explicit limit to the size of a key to 255 chars #27

Closed jnioche closed 3 years ago

jnioche commented 3 years ago

This is the max length that a fully qualified domain name can have. Any URL sent to the Frontier with an explicit or computed key greater than the limit should be ignored. This would prevent a situation where resources would get exhausted if very large keys were used