Open stevieflow opened 3 years ago
The comments are useful, but I have a feeling @markbrough said it’s a bit trickier to scrape.
I didn't really look into it but I assume it should be fairly straightforward - I think it was just that it would significantly increase the size of the files. But maybe that is fine?
It will add to the file size, I agree - but will significantly improve the search function
(note - the searching of this codelist seems a liitle shaky atm, maybe because of it's size...)
Yup I agree with both of these points! Searching is very slow at the moment on large lists, and will likely be substantially slower if we add in the comments field. Perhaps there are improvements to the search function that we could make -- there is already a ticket for this: https://github.com/codeforIATI/codelists/issues/72
I just realised the comments already get scraped – I guess @markbrough already did that work! https://codeforiati.org/humanitarian-emergency-codelists/glide-emergencies.csv
So yeah, the file size is the only issue here (and therefore the search).
Re: https://github.com/codeforIATI/humanitarian-emergency-codelists
Coudl the comment field also be imported? This gives a useful description across the emergencies. For example 32 comments reference "ebola", whereas the titles do not