Missing Taxonomy data

[akariv]

In the current OS API, each dimension has a taxonomy field, see for example here (https://openspending.org/api/2/aggregate?drilldown=region%7Ccofog3%7Ccofog2%7Ccofog1&cut=year%3A2010&dataset=ukgov-finances-cra)

{
  "drilldown": [
    {
      "region": {
        "taxonomy": "cra-region", 
        "html_url": "https://openspending.org/ukgov-finances-cra/region/not-identifiable", 
        "id": 14, 
        "name": "not-identifiable", 
        "label": "NOT IDENTIFIABLE"
      }, 
      "amount": 42529187000.0, 
      "cofog2": {
        "name": "01.7", 
        "color": "#9900cc", 
        "taxonomy": "cofog-2", 

both cra-region and cofog-2 are values which are not present in the migrated datapackage.json and seem to be needed by users of the data.

[pudo]

I don't think the taxonomy thing is actually still used anywhere! If I recall correctly some of the old API would fake it by using dimension names, but I can't imagine where it'd be actively used.

[akariv]

That's what I thought too, but I stumbled upon it when trying to set-up the current wheredoesmymoneygo.org website with the new API server... It started working only after I added this property to the returned results (as hard coded values for now)

On Thu, Dec 31, 2015 at 11:23 AM Friedrich Lindenberg < notifications@github.com> wrote:

I don't think the taxonomy thing is actually still used anywhere! If I recall correctly some of the old API would fake it by using dimension names, but I can't imagine where it'd be actively used.

— Reply to this email directly or view it on GitHub https://github.com/openspending/openspending-migrate/issues/9#issuecomment-168154670 .

[akariv]

@danfowler any idea where this field is coming from?

[danfowler]

@akariv Doing some research. There are only 18 public datasets (all created between late 2011 and early 2012) with attributes that have taxonomy information in the database. For 3, the attribute name is the taxonomy; for 3 others, the taxonomy is unknown; for the rest, some assorted prefixing logic. I wonder if the only consumer for this is the old OpenSpending JS bubbletree and geo code. Where it exists, the taxonomy is stored (along with the other metadata) as JSON in the data field of a given dataset in the database. I could theoretically bring this forward, but seeing as how taxonomy was given up long ago perhaps it's better to update wheredoesmymoneygo.org as appropriate?

@pudo do I have this right?

https://openspending.org/ada 2011-12-09

attribute	taxonomy
region	region
country	country
from	from
to	to

https://openspending.org/afd 2011-12-09

attribute	taxonomy
social-impact	social-impact
agency	agency
country	country
responsible-structure	responsible-structure
responsible-agency	responsible-agency
is-co-financed	is-co-financed
funding-type	funding-type
status	status
aim	sector
environmental-impact	environmental-impact

https://openspending.org/uy-credito-presupuestal 2011-12-09

attribute	taxonomy
organismo	organismo
area_programatica	area-programatica
programa	programa

https://openspending.org/city-of-redacre-spending 2011-12-09

attribute	taxonomy
barsacct	unknown
category	unknown
object	unknown
priority	unknown
description	unknown
from	unknown
to	unknown

https://openspending.org/city-of-springfield-budget 2011-12-09

attribute	taxonomy
deptgroup	unknown
from	unknown
to	unknown

https://openspending.org/city-of-whiteacre-spending 2011-12-09

attribute	taxonomy
deptgroup	unknown
accounttype	unknown
from	unknown
to	unknown
object	unknown

https://openspending.org/budget-city-huettenberg 2012-01-06

attribute	taxonomy
hauptproduktbereich	lkgiessen-level-1
produkt	lkgiessen-level-4
produktgruppe	lkgiessen-level-3
produktbereich	lkgiessen-level-2

https://openspending.org/budget-city-lich 2012-01-08

attribute	taxonomy
produkt	lich-level-3
produktgruppe	lich-level-2
produktbereich	lich-level-1

https://openspending.org/budget_city_giessen 2011-12-09

attribute	taxonomy
amt	giessen-level-3
produkt	giessen-level-2
produktbereich	giessen-level-1

https://openspending.org/budget_city_linden 2012-02-11

attribute	taxonomy
abteilung	linden-level-2
amt	linden-level-3
bereich	linden-level-1

https://openspending.org/budget_county_giessen 2011-12-09

attribute	taxonomy
produkt	lkgiessen-level-3
produktgruppe	lkgiessen-level-2
produktbereich	lkgiessen-level-1

https://openspending.org/frankfurt-budget 2011-12-09

attribute	taxonomy
haushaltsstelle	ffm-level-3
produktgruppe	ffm-level-2
produktbereich	ffm-level-1

https://openspending.org/iati 2011-12-09

attribute	taxonomy
tied_status	iati-tied-status
identifier	iati-identifier
recipient_region	iati-region
sector	iati-sector
finance_type	iati-finance-type
transaction_type	iati-tx-type
status	iati-status
original_currency	iati-currency
flow_type	iati-flow-type
aid_type	iati-aid-type
recipient_country	iati-country

https://openspending.org/it-regional-accounts 2011-12-09

attribute	taxonomy
function	itb-function

https://openspending.org/mkk_budget 2011-12-09

attribute	taxonomy
fachbereich	mkk-level-2
konto	mkk-level-4
produkt	mkk-level-3
dezernat	mkk-level-1

https://openspending.org/ugandabudget 2011-12-09

attribute	taxonomy
sector_objective	uganda.swg.sector_objective
swg	uganda.swg
subsector	uganda.sector.subsector
peap_area	uganda.peap.objective.area
gou_vote	uganda.gou_vote
spending_source_type	uganda.spending_source_type
programme	uganda.programme
peap_pillar	uganda.peap
sector	uganda.sector
peap_objective	uganda.peap.objective

https://openspending.org/uk-barnet-budget 2011-12-09

attribute	taxonomy
level1	barnet-level-1
level2	barnet-level-2
level3	barnet-level-3

https://openspending.org/ukgov-finances-cra 2011-12-09

attribute	taxonomy
region	cra-region
pog	cra-pog
hmt1	cra-hmt-level1
hmt2	cra-hmt-level2
cofog1	cofog-1
cofog2	cofog-2
cofog3	cofog-3
cg_lg_or_pc	cra-cg_lg_or_pc
cap_or_cur	cra-cap_or_cur

[akariv]

Well, given that this is only necessary for backward compatibility, I think I can have these values hard-coded for these specific datasets. I really don't think that we need to find any 'generic' solution here (@pwalsh what do you reckon?).

On Tue, Jan 5, 2016 at 10:53 PM, Daniel Fowler notifications@github.com wrote:

@akariv https://github.com/akariv Doing some research. There are only 18 public datasets (all created between late 2011 and early 2012) with attributes that have taxonomy information in the database. For 3, the attribute name is the taxonomy; for 3 others, the taxonomy is unknown; for the rest, some assorted prefixing logic. I wonder if the only consumer for this is the old OpenSpending JS https://github.com/openspending-archive/openspendingjs/search?utf8=%E2%9C%93&q=taxonomy bubbletree and geo code. Where it exists, the taxonomy is stored (along with the other metadata) as JSON in the data field of a given dataset in the database. I could theoretically bring this forward, but seeing as how taxonomy was given up long ago https://github.com/openspending/openspending/issues/170#issuecomment-2674225 perhaps it's better to update wheredoesmymoneygo.org as appropriate?

@pudo https://github.com/pudo do I have this right?

https://openspending.org/ada 2011-12-09 attribute taxonomy region region country country from from to to

https://openspending.org/afd 2011-12-09 attribute taxonomy social-impact social-impact agency agency country country responsible-structure responsible-structure responsible-agency responsible-agency is-co-financed is-co-financed funding-type funding-type status status aim sector environmental-impact environmental-impact

https://openspending.org/uy-credito-presupuestal 2011-12-09 attribute taxonomy organismo organismo area_programatica area-programatica programa programa

https://openspending.org/city-of-redacre-spending 2011-12-09 attribute taxonomy barsacct unknown category unknown object unknown priority unknown description unknown from unknown to unknown

https://openspending.org/city-of-springfield-budget 2011-12-09 attribute taxonomy deptgroup unknown from unknown to unknown

https://openspending.org/city-of-whiteacre-spending 2011-12-09 attribute taxonomy deptgroup unknown accounttype unknown from unknown to unknown object unknown

https://openspending.org/budget-city-huettenberg 2012-01-06 attribute taxonomy hauptproduktbereich lkgiessen-level-1 produkt lkgiessen-level-4 produktgruppe lkgiessen-level-3 produktbereich lkgiessen-level-2

https://openspending.org/budget-city-lich 2012-01-08 attribute taxonomy produkt lich-level-3 produktgruppe lich-level-2 produktbereich lich-level-1

https://openspending.org/budget_city_giessen 2011-12-09 attribute taxonomy amt giessen-level-3 produkt giessen-level-2 produktbereich giessen-level-1

https://openspending.org/budget_city_linden 2012-02-11 attribute taxonomy abteilung linden-level-2 amt linden-level-3 bereich linden-level-1

https://openspending.org/budget_county_giessen 2011-12-09 attribute taxonomy produkt lkgiessen-level-3 produktgruppe lkgiessen-level-2 produktbereich lkgiessen-level-1

https://openspending.org/frankfurt-budget 2011-12-09 attribute taxonomy haushaltsstelle ffm-level-3 produktgruppe ffm-level-2 produktbereich ffm-level-1

https://openspending.org/iati 2011-12-09 attribute taxonomy tied_status iati-tied-status identifier iati-identifier recipient_region iati-region sector iati-sector finance_type iati-finance-type transaction_type iati-tx-type status iati-status original_currency iati-currency flow_type iati-flow-type aid_type iati-aid-type recipient_country iati-country

https://openspending.org/it-regional-accounts 2011-12-09 attribute taxonomy function itb-function

https://openspending.org/mkk_budget 2011-12-09 attribute taxonomy fachbereich mkk-level-2 konto mkk-level-4 produkt mkk-level-3 dezernat mkk-level-1

https://openspending.org/ugandabudget 2011-12-09 attribute taxonomy sector_objective uganda.swg.sector_objective swg uganda.swg subsector uganda.sector.subsector peap_area uganda.peap.objective.area gou_vote uganda.gou_vote spending_source_type uganda.spending_source_type programme uganda.programme peap_pillar uganda.peap sector uganda.sector peap_objective uganda.peap.objective

https://openspending.org/uk-barnet-budget 2011-12-09 attribute taxonomy level1 barnet-level-1 level2 barnet-level-2 level3 barnet-level-3

https://openspending.org/ukgov-finances-cra 2011-12-09 attribute taxonomy region cra-region pog cra-pog hmt1 cra-hmt-level1 hmt2 cra-hmt-level2 cofog1 cofog-1 cofog2 cofog-2 cofog3 cofog-3 cg_lg_or_pc cra-cg_lg_or_pc cap_or_cur cra-cap_or_cur

— Reply to this email directly or view it on GitHub https://github.com/openspending/openspending-migrate/issues/9#issuecomment-169130621 .

[pwalsh]

@akariv agree.

[akariv]

Cool - so @danfowler thanks a lot for digging up these values! I think we can close this issue now.