search

codeforIATI / Unofficial-Codelists

🥉Third party codelists that are not tracked by IATI
MIT License
0 stars 1 forks source link

Get comments from GLIDE list #189

Open stevieflow opened 3 years ago

stevieflow commented 3 years ago

Re: https://github.com/codeforIATI/humanitarian-emergency-codelists

Coudl the comment field also be imported? This gives a useful description across the emergencies. For example 32 comments reference "ebola", whereas the titles do not

andylolz commented 3 years ago

The comments are useful, but I have a feeling @markbrough said it’s a bit trickier to scrape.

markbrough commented 3 years ago

I didn't really look into it but I assume it should be fairly straightforward - I think it was just that it would significantly increase the size of the files. But maybe that is fine?

stevieflow commented 3 years ago

It will add to the file size, I agree - but will significantly improve the search function

(note - the searching of this codelist seems a liitle shaky atm, maybe because of it's size...)

markbrough commented 3 years ago

Yup I agree with both of these points! Searching is very slow at the moment on large lists, and will likely be substantially slower if we add in the comments field. Perhaps there are improvements to the search function that we could make -- there is already a ticket for this: https://github.com/codeforIATI/codelists/issues/72

andylolz commented 3 years ago

I just realised the comments already get scraped – I guess @markbrough already did that work! https://codeforiati.org/humanitarian-emergency-codelists/glide-emergencies.csv

So yeah, the file size is the only issue here (and therefore the search).