codeforIATI / iatikit

🐨 A toolkit for using IATI data
https://iatikit.readthedocs.io
MIT License
6 stars 0 forks source link

Chaining together `where`s that use the same keyword has unexpected results #45

Closed andylolz closed 5 years ago

andylolz commented 5 years ago

It’s possible to chain together multiple where filters, like this:

acts = iatikit.data().publishers.get('cafod').activities

cap_acts = acts.where(title__contains='capacity stengthening')
len(cap_acts)  # 8 activities have titles that mention "capacity stengthening"

lr_cap_acts = cap_acts.where(xpath='recipient-country/@code="LR"')
len(lr_cap_acts)  # 2 of the above activities list Liberia as a recipient country

It *ought to be possible* to chain together multiple where filters that use the same keyword. So for instance, multiple xpath queries.

However, this *isn’t* currently possible, and the behaviour is WEIRD. This is a bug!

johnadamsDFID commented 5 years ago

Yep, here's my example. Constructing a complex xpath query seems to be the only solution

dfid = iatikit.data().publishers.find(name='dfid')

dfid_wb = dfid.activities.where(xpath='participating-org/@ref="44000"')
dfid_wb.count() #1179

dfid_wb_h1 = dfid_wb.where(xpath='@hierarchy=1')
dfid_wb_h1.count() #5264

dfid_wb = dfid.activities.where(xpath='@hierarchy="1" and participating-org/@ref="44000"')
dfid_wb.count() #527
andylolz commented 5 years ago

Thanks @johnadamsDFID! Exactly right – that’s the only workaround currently :(

This is very annoying and also very fixable, so it’s next up.

johnadamsDFID commented 5 years ago

Yep, took me quite a while of head scratching to work out why the numbers were going up!

andylolz commented 5 years ago

Ha! Right, yes. Sorry about that!

andylolz commented 5 years ago

Fixed in b41f5d3. (This will be part of the next release.)

andylolz commented 5 years ago

@johnadamsDFID with the fix in place, your example now gives the expected result:

dfid = iatikit.data().publishers.find(name='dfid')

dfid_wb = dfid.activities.where(xpath='participating-org/@ref="44000"')
dfid_wb.count() #1179

dfid_wb_h1 = dfid_wb.where(xpath='@hierarchy=1')
dfid_wb_h1.count() #527