add sitemap functionality - Githubissues

rero / sonar

SONAR - Swiss Open Access Repository

https://www.rero.ch/products/sonar

GNU Affero General Public License v3.0

12 stars 11 forks source link

add sitemap functionality #866

Closed Garfield-fr closed 2 years ago

Garfield-fr commented 2 years ago

Adds a new field serverName on organisation resource.
Implements sitemap generation for global and dedicated organisations.
Implements generation of the file robot.txt dynamically.
Closes #798.

Co-Authored-by: Bertrand Zuchuat bertrand.zuchuat@rero.ch

⚠️ ES update mapping ⚠️ Production

make the flag True on INVENIO_SONAR_APP_PRODUCTION_STATE
set INVENIO_SONAR_APP_SITEMAP_FOLDER_PATH

How to test?

poetry shell
./script/bootstrap
./script/setup --clear-data

Sitemap

invenio oaiharvester harvest -n rerodoc -q -a max=200 -f 2021-10-01
invenio sitemap generate -d sonar.rero.ch (for global view)
invenio sitemap generate -d folia.unifr.ch (for global view)
Global sitemap: https://localhost:5000/global/sitemap.xml
Organisation sitemap: https://localhost:5000/unifr/sitemap.xml (if you are some documents)

Robot.txt

by default: disallow indexation

https://localhost:5000/robot.txt
Change constant SONAR_APP_PRODUCTION_STATE to True
https://localhost:5000/robot.txt (allow indexation and show sitemap link)

Code review check list

[ ] Commit message template compliance.
[ ] Commit message without typos.
[ ] File names.
[ ] Functions names.
[ ] Functions docstrings.
[ ] Unnecessary commited files?
[ ] Extracted translations?

pronguen commented 2 years ago

[x] NPR/18.08: The file should be named robots.txt instead of robot.txt ZUB: Done -> https://github.com/rero/sonar/pull/866/files#diff-cac4306f677b3df8e88dec04e6b30c2b57e2fcd3c8ed35a24066b097f08f2821R70