Fixes #425 by creating Matchers that throw RuntimeExceptions on timeout and a TimeoutablePathRule that extends PathRule that uses them.
The default behaviour of the system is not to use them, however, it can be enabled via RobotstxtConfig.
NOTE: The code for the timeoutable Matches is based on this stackoverflow answer and it decreases performance of regexp. The ideal thing should be to include a native efficient and timeoutable regex library, but this is a valid workaround
Fixes #425 by creating Matchers that throw RuntimeExceptions on timeout and a TimeoutablePathRule that extends PathRule that uses them.
The default behaviour of the system is not to use them, however, it can be enabled via RobotstxtConfig.
NOTE: The code for the timeoutable Matches is based on this stackoverflow answer and it decreases performance of regexp. The ideal thing should be to include a native efficient and timeoutable regex library, but this is a valid workaround