As default the /robots.txt is used, and this file allows everything.
However there is also an /robots-dev.txt which should be used via
apache rewrite url to disallow indexing. Thus, deploying this repo
defaults to allow indexing, but empowers disallowing by adjusting
the http server config.
See lobid/lodmill#349.
As default the /robots.txt is used, and this file allows everything. However there is also an /robots-dev.txt which should be used via apache rewrite url to disallow indexing. Thus, deploying this repo defaults to allow indexing, but empowers disallowing by adjusting the http server config.