FRED.stlouisfed.org (coding Financial-Economic Units)

KevinCuddeback commented 4 years ago

The St Louis Federal Reserve sponsors an extensive set of data feeds (792,000) that are (1) poorly indexed (they say they don't have a list of all their feeds) and poorly unitized (seem to be "cut-and-pasted" from the prose title, rather than systematically logged.

I'm proposing for myself (and would like help with) 1) To map all 765,000 data feeds (most US Federal sources, like Census, BLS, and financial regulators) 2) To map the long-prose and short-prose units used in all the feeds 3) To code them so that "everyone" in the QUDT community would have a systematic way of understanding when FRED says "mil. of $" as the units that it is expressed in standard QUDT form in some kind of cross-walk

I have begun at https://fred.stlouisfed.org/tags/series

with the 167 currently-active feeds (there are hundreds of others that are infrequent or ancient (cotton mill production in New England in the early 1800s, for example)

https://fred.stlouisfed.org/release?rid=9

jhodgesatmb commented 4 years ago

Fred,

First, I think this is great!

Can you elaborate on what you mean by ‘feed’! I think of a feed as something streamed or real-time but that doesn’t make sense in this context.

Are these, without looking at them yet, domain specific?

Jack Hodges, Ph.D. Arbor Studios

On May 31, 2020, at 4:34 PM, KevinCuddeback notifications@github.com wrote:

The St Louis Federal Reserve sponsors an extensive set of data feeds (792,000) that are (1) poorly indexed (they say they don't have a list of all their feeds) and poorly unitized (seem to be "cut-and-pasted" from the prose title, rather than systematically logged.

I'm proposing for myself (and would like help with)

To map all 765,000 data feeds (most US Federal sources, like Census, BLS, and financial regulators) To map the long-prose and short-prose units used in all the feeds To code them so that "everyone" in the QUDT community would have a systematic way of understanding when FRED says "mil. of $" as the units that it is expressed in standard QUDT form in some kind of cross-walk I have begun at https://fred.stlouisfed.org/tags/series

with the 167 currently-active feeds (there are hundreds of others that are infrequent or ancient (cotton mill production in New England in the early 1800s, for example)

https://fred.stlouisfed.org/release?rid=9

https://fred.stlouisfed.org/release?rid=10 https://fred.stlouisfed.org/release?rid=11 https://fred.stlouisfed.org/release?rid=13 https://fred.stlouisfed.org/release?rid=14 https://fred.stlouisfed.org/release?rid=15 https://fred.stlouisfed.org/release?rid=17 https://fred.stlouisfed.org/release?rid=18 https://fred.stlouisfed.org/release?rid=19 https://fred.stlouisfed.org/release?rid=20 https://fred.stlouisfed.org/release?rid=21 https://fred.stlouisfed.org/release?rid=22 https://fred.stlouisfed.org/release?rid=25 https://fred.stlouisfed.org/release?rid=27 https://fred.stlouisfed.org/release?rid=46 https://fred.stlouisfed.org/release?rid=47 https://fred.stlouisfed.org/release?rid=49 https://fred.stlouisfed.org/release?rid=50 https://fred.stlouisfed.org/release?rid=51 https://fred.stlouisfed.org/release?rid=52 https://fred.stlouisfed.org/release?rid=53 https://fred.stlouisfed.org/release?rid=54 https://fred.stlouisfed.org/release?rid=61 https://fred.stlouisfed.org/release?rid=63 https://fred.stlouisfed.org/release?rid=70 https://fred.stlouisfed.org/release?rid=71 https://fred.stlouisfed.org/release?rid=72 https://fred.stlouisfed.org/release?rid=86 https://fred.stlouisfed.org/release?rid=91 https://fred.stlouisfed.org/release?rid=92 https://fred.stlouisfed.org/release?rid=93 https://fred.stlouisfed.org/release?rid=95 https://fred.stlouisfed.org/release?rid=97 https://fred.stlouisfed.org/release?rid=100 https://fred.stlouisfed.org/release?rid=101 https://fred.stlouisfed.org/release?rid=109 https://fred.stlouisfed.org/release?rid=110 https://fred.stlouisfed.org/release?rid=112 https://fred.stlouisfed.org/release?rid=113 https://fred.stlouisfed.org/release?rid=116 https://fred.stlouisfed.org/release?rid=122 https://fred.stlouisfed.org/release?rid=140 https://fred.stlouisfed.org/release?rid=148 https://fred.stlouisfed.org/release?rid=151 https://fred.stlouisfed.org/release?rid=169 https://fred.stlouisfed.org/release?rid=171 https://fred.stlouisfed.org/release?rid=175 https://fred.stlouisfed.org/release?rid=179 https://fred.stlouisfed.org/release?rid=180 https://fred.stlouisfed.org/release?rid=183 https://fred.stlouisfed.org/release?rid=185 https://fred.stlouisfed.org/release?rid=187 https://fred.stlouisfed.org/release?rid=188 https://fred.stlouisfed.org/release?rid=189 https://fred.stlouisfed.org/release?rid=190 https://fred.stlouisfed.org/release?rid=192 https://fred.stlouisfed.org/release?rid=193 https://fred.stlouisfed.org/release?rid=194 https://fred.stlouisfed.org/release?rid=196 https://fred.stlouisfed.org/release?rid=197 https://fred.stlouisfed.org/release?rid=198 https://fred.stlouisfed.org/release?rid=199 https://fred.stlouisfed.org/release?rid=200 https://fred.stlouisfed.org/release?rid=206 https://fred.stlouisfed.org/release?rid=208 https://fred.stlouisfed.org/release?rid=209 https://fred.stlouisfed.org/release?rid=212 https://fred.stlouisfed.org/release?rid=219 https://fred.stlouisfed.org/release?rid=221 https://fred.stlouisfed.org/release?rid=228 https://fred.stlouisfed.org/release?rid=229 https://fred.stlouisfed.org/release?rid=238 https://fred.stlouisfed.org/release?rid=239 https://fred.stlouisfed.org/release?rid=242 https://fred.stlouisfed.org/release?rid=245 https://fred.stlouisfed.org/release?rid=251 https://fred.stlouisfed.org/release?rid=253 https://fred.stlouisfed.org/release?rid=256 https://fred.stlouisfed.org/release?rid=261 https://fred.stlouisfed.org/release?rid=263 https://fred.stlouisfed.org/release?rid=266 https://fred.stlouisfed.org/release?rid=267 https://fred.stlouisfed.org/release?rid=268 https://fred.stlouisfed.org/release?rid=269 https://fred.stlouisfed.org/release?rid=271 https://fred.stlouisfed.org/release?rid=274 https://fred.stlouisfed.org/release?rid=279 https://fred.stlouisfed.org/release?rid=281 https://fred.stlouisfed.org/release?rid=282 https://fred.stlouisfed.org/release?rid=283 https://fred.stlouisfed.org/release?rid=287 https://fred.stlouisfed.org/release?rid=288 https://fred.stlouisfed.org/release?rid=289 https://fred.stlouisfed.org/release?rid=290 https://fred.stlouisfed.org/release?rid=291 https://fred.stlouisfed.org/release?rid=296 https://fred.stlouisfed.org/release?rid=301 https://fred.stlouisfed.org/release?rid=304 https://fred.stlouisfed.org/release?rid=305 https://fred.stlouisfed.org/release?rid=308 https://fred.stlouisfed.org/release?rid=313 https://fred.stlouisfed.org/release?rid=315 https://fred.stlouisfed.org/release?rid=317 https://fred.stlouisfed.org/release?rid=319 https://fred.stlouisfed.org/release?rid=320 https://fred.stlouisfed.org/release?rid=321 https://fred.stlouisfed.org/release?rid=322 https://fred.stlouisfed.org/release?rid=323 https://fred.stlouisfed.org/release?rid=325 https://fred.stlouisfed.org/release?rid=326 https://fred.stlouisfed.org/release?rid=327 https://fred.stlouisfed.org/release?rid=328 https://fred.stlouisfed.org/release?rid=331 https://fred.stlouisfed.org/release?rid=332 https://fred.stlouisfed.org/release?rid=333 https://fred.stlouisfed.org/release?rid=334 https://fred.stlouisfed.org/release?rid=336 https://fred.stlouisfed.org/release?rid=339 https://fred.stlouisfed.org/release?rid=341 https://fred.stlouisfed.org/release?rid=342 https://fred.stlouisfed.org/release?rid=343 https://fred.stlouisfed.org/release?rid=345 https://fred.stlouisfed.org/release?rid=351 https://fred.stlouisfed.org/release?rid=352 https://fred.stlouisfed.org/release?rid=353 https://fred.stlouisfed.org/release?rid=355 https://fred.stlouisfed.org/release?rid=357 https://fred.stlouisfed.org/release?rid=359 https://fred.stlouisfed.org/release?rid=362 https://fred.stlouisfed.org/release?rid=363 https://fred.stlouisfed.org/release?rid=364 https://fred.stlouisfed.org/release?rid=372 https://fred.stlouisfed.org/release?rid=373 https://fred.stlouisfed.org/release?rid=374 https://fred.stlouisfed.org/release?rid=375 https://fred.stlouisfed.org/release?rid=376 https://fred.stlouisfed.org/release?rid=377 https://fred.stlouisfed.org/release?rid=378 https://fred.stlouisfed.org/release?rid=379 https://fred.stlouisfed.org/release?rid=383 https://fred.stlouisfed.org/release?rid=386 https://fred.stlouisfed.org/release?rid=391 https://fred.stlouisfed.org/release?rid=397 https://fred.stlouisfed.org/release?rid=400 https://fred.stlouisfed.org/release?rid=403 https://fred.stlouisfed.org/release?rid=427 https://fred.stlouisfed.org/release?rid=434 https://fred.stlouisfed.org/release?rid=435 https://fred.stlouisfed.org/release?rid=436 https://fred.stlouisfed.org/release?rid=439 https://fred.stlouisfed.org/release?rid=440 https://fred.stlouisfed.org/release?rid=441 https://fred.stlouisfed.org/release?rid=442 https://fred.stlouisfed.org/release?rid=444 https://fred.stlouisfed.org/release?rid=445 https://fred.stlouisfed.org/release?rid=446 https://fred.stlouisfed.org/release?rid=449 https://fred.stlouisfed.org/release?rid=450 https://fred.stlouisfed.org/release?rid=454 https://fred.stlouisfed.org/release?rid=456 https://fred.stlouisfed.org/release?rid=459 https://fred.stlouisfed.org/release?rid=462 https://fred.stlouisfed.org/release?rid=463 https://fred.stlouisfed.org/release?rid=465 https://fred.stlouisfed.org/release?rid=468 https://fred.stlouisfed.org/release?rid=469 https://fred.stlouisfed.org/release?rid=470

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

steveraysteveray commented 4 years ago

I agree with Jack. Very interesting. Is this just variants of US dollars, or international in scope?

KevinCuddeback commented 4 years ago

FRED (the data APIs of the St Louis Federal Reserve that re-publish economic-financial-demographic statistics) has a strong USA bias (the mixed blessing that when they say $ it 99% of the time can be assumed to be USD), but at the same time does (re)publish economic statistics from around the world.

National Accounts for Japan: https://fred.stlouisfed.org/release?rid=269 which, as you see, has units stated in prose, like "Billions of Chained 2011 Yen"

European Central bank assets https://fred.stlouisfed.org/release?rid=271 which also has units stated in prose "Millions of Euros"

KevinCuddeback commented 4 years ago

FRED is (for me) "the world of prose-described economic units" -- a complete statement of "the problem"

FRED also has great/horrifying/specific examples of how Cryptocurrencies have the potential to break existing ISO 4217 currency codes (as also described here: https://en.wikipedia.org/wiki/ISO_4217#Cryptocurrencies), since FRED republishes Coinbase's higher-volume trading stats:

https://fred.stlouisfed.org/release?rid=441

KevinCuddeback commented 4 years ago

FRED has collected the world of open and lightly-licensed economic data in a single source. They are feeds in the sense that you can grab big blocks via their API (json or xml).

"Worse" they don't have a single catalog of what they have (they tell me) but just a searchable repository of the world's open economic data.

The units of measure challenge is that, while they have a world of census, bank, & financial-markets data, they have been systematic-but-not-standardized in (essentially) tearing their units of measure in long-form and short-form prose from other people's "chart legend" text.

https://fred.stlouisfed.org/tags/series

qudt / qudt-public-repo

FRED.stlouisfed.org (coding Financial-Economic Units) #130

https://fred.stlouisfed.org/release?rid=9