StackExchange Watcher does not correctly grabs new dumps

[benoit74]

Due to changes in StackExchange processes, our watcher does not grab new dumps properly.

Originally, all dumps where pushed to https://archive.org/details/stackexchange

Latest dumps have dedicated URLs now:

• https://archive.org/details/stackexchange_20240630
• https://archive.org/details/stackexchange_20240630_revised
• https://archive.org/details/stackexchange_20240930

Looks like we can search for these new identifiers with this URL: https://archive.org/services/search/beta/page_production/?user_query=subject:%22Stack%20Exchange%20Data%20Dump%22%20creator:%22Stack%20Exchange,%20Inc.%22&hits_per_page=1&page=1&sort=date:desc&aggregations=false&client_url=https://archive.org/search?query=subject%3A%22Stack+Exchange+Data+Dump%22+creator%3A%22Stack+Exchange%2C+Inc.%22

[benoit74]

This is blocked by the fact that Sites.xml is not provided anymore in the datadumps.

I've opened an issue upstream: https://meta.stackexchange.com/questions/404002/sites-xml-is-not-present-anymore-in-stackexchange-data-dumps

[benoit74]

I've pushed current changes (not complete at all, but already capable to retrieve the most recent dump) to https://github.com/openzim/zimfarm/tree/new_dumps_url ; waiting for SE answer on upstream issue