Running the crawler is not adding new data neither:
python -m iyp.crawlers.citizenlab.urldb
Additional context
I'm not quite sure, but it seems that github behave differently than before to web scraping. I think we should use the github api, it is more reliable that reading the html from github. See iyp/crawlers/inetintel/as_org.py for an example.
Describe the bug The citizen lab crawler is not anymore pushing data to the database.
To Reproduce In recent dumps (https://exp1.iijlab.net/wip/iyp/dumps/2023/07/22/iyp-2023-07-22.dump) the following query gives no result:
Running the crawler is not adding new data neither:
Additional context I'm not quite sure, but it seems that github behave differently than before to web scraping. I think we should use the github api, it is more reliable that reading the html from github. See iyp/crawlers/inetintel/as_org.py for an example.