Various notes from trying to implement

[pudo]

Just for fun, I'm trying to hack up a FDP->babbage->FDP translator based on Dan's code. As some feedback, here's a sort of stream of consciousness of the issues that come up during this implementation effort:

Problems

• source is a weird name for the column thing. It's a really, really overloaded term. Why not field?
• The term field is overloaded between resources and dimensions (both are often used in the same code, I'd guess). Proposing attribute (ref #40).
• Why is the model called mapping, when mapping columns is only a small part of what it does? This is a mistake we already made in OpenSpending, now is the time to correct it.
• Allowing the primaryKey on a dimension to be a list seems a bit too flexible to me, what's the use case? How would I stuff that into an aggregation call, or generate a URL?
• When describing dimensions, the spec mentions both id and code fields. Which of these is better? How do they relate to primaryKey?
• At the same time, I still can't tell how to determine label fields for a dimension.
• Is there no difference between the user-facing name and the column name of a dimension, measure or dimension field?
• What is supposed to happen if the model includes columns from multiple resources? I can't find instructions on how to join them in the TDP spec (maybe too stupid?).

  Opinions

• How did phase and direction end up as attributes on the measure? While I agree that source budget data with multiple measures is quite common, I cannot see a clear difference between these two attributes and other dimensions. I want to break them down the same way I would break down a sector or a supplier, rather than by fiddling with multiple measures. (ref #91 - the discussion there confuses the living daylight out of me)
• There are also more types of phases and directions (the last dataset I added to OHa has directions "revenue", "expenditure", "internal paid", "internal received", "balance").
• Making everything a list makes sense in JTS (to preserve order), but just makes life a little harder for everything in the model (ref #90, #54).
• Proposing that code sheets separate keys with a dot is very cofog-specific, many taxonomies just use string-numbers (i.e. 0, 01, 011 for three levels).

[danfowler]

The term field is overloaded between resources and dimensions (both are often used in the same code, I'd guess). Proposing attribute (ref #40).

Thanks @pudo. I made a similar point here in a long discussion about restructuring the dimension object to support hierarchies. A given dimension field potentially, though not always, maps to multiple source fields.

[rufuspollock]

@danfowler can we make an issue for e.g. dimension fields as object rather than array?

[danfowler]

@rgrp I can, but I think there are actually (at least) two issues here:

1. Dimension fields as objects, forcing each name to be unique.
2. Dimension fields as objects that are each a collection of source CSV fields. Currently dimension fields map directly to source CSV fields.

[danfowler]

@rgrp I made the issue: https://github.com/openspending/fiscal-data-package/issues/94

[danfowler]

There are also more types of phases and directions (the last dataset I added to OHa has directions "revenue", "expenditure", "internal paid", "internal received", "balance").

@pudo would you class these as "directions"?

[pudo]

@danfowler I'm not actually sure, seems hacky to me. The city treasurer of Hildesheim apparently thinks they are directions, though. They live in a multi-dimensional universe.

[pwalsh]

I think much of this has been addressed. The main thing for me is that now that we use attributes in the logical model, I think we should support the proposed change @pudo suggests of source to field. It fits the semantics of the spec perfectly (example: an attributes maps to a field or a constant).

[danfowler]

@pwalsh +1

I see what you're saying and it makes sense.

In the physical model (the resources array), each resource has a schema and each schema has a fields array. If we make this change, we are mapping/modeling an element of the fields array to a field in the mapping/model.

[rufuspollock]

@pwalsh @danfowler can we get a dedicated issue for this as per (developing) standard practice.

[pwalsh]

@rgrp https://github.com/openspending/fiscal-data-package/issues/114

@pudo can we close this catch-all, or, is there something else worth pulling out to a distinct issue at this stage?

[pwalsh]

FIXED.