search

lokal-profil / BatchUploadTools

An attempt of unifying the various tools for batch uploads
MIT License
4 stars 2 forks source link

[WIP]Restructure listscraping #34

Closed lokal-profil closed 6 years ago

lokal-profil commented 7 years ago

Rebuild the listscraper to use the MappingList class.

This is a breaking change. To reflect this method names have changed and the version has been bumped.

There are still several open @todos related to internalising more of the unusual settings so as to not have them passed around as arguments. There are also some @todos related to reducing the number of needed settings in general.

Task: T180675

lokal-profil commented 7 years ago

@Vesihiisi Not critical that review happens now. Mainly wanted to flag it before its merged and breaks anything relying on listscraper in master.

lokal-profil commented 7 years ago

Rebased on top of 0.1.9

lokal-profil commented 7 years ago

Rebased on top of 0.1.10

lokal-profil commented 7 years ago

A known issue. For multi section list pages. The harvest treats these as one list and so deals badly with the same entry appearing in multiple lists. The safest solution is to not mix different lists on a page.

lokal-profil commented 7 years ago

Rebased to include #37

lokal-profil commented 7 years ago

Rebased on #38