Provide starting point URL/s for the crawler [Feature/issue 83]

salsadigitalauorg / merlin-framework

Merlin - migration framework

GNU General Public License v3.0

16 stars 3 forks source link

Provide starting point URL/s for the crawler [Feature/issue 83] #94

Closed nickgeorgiou closed 5 years ago

nickgeorgiou commented 5 years ago

Starting URLs for the crawler can be provided with the urls option in the crawler config. You can provide the urls option as an array (one or more URLs) or a string (single URL).

stooit commented 5 years ago

Merging for immediate need; @nickgeorgiou tests in a separate PR please 🙏