load data

[feomike]

load all data into postgress

[feomike]

Transmittal Sheet Update all TS data from 1990-2014 is loaded in Postgres (script forthcoming). there where some issues

• 2004 and 2005 had non UTF-8 issues. they were easy to solve by opening the file in textwrangler and re-openning as utf-8 and saving.
• 1992 and 1993 download files are totally hosed; had 100% binary characters in them. I downloaded and used the 1992 and 1993 ultimate files in their stead, which did work
• 1990 and 1991 had sporadic non-utf-8 characters which couldn't be solved with re-openning and saving the file as utf-8 compatable. i had to hand edit an search for issues

[feomike]

Panel Update all Panel data from 1992 - 2014 is loaded and in Postgres (script forthcoming).
*1990 and 1991 has a mis-aligned fixed field file. the issue is mostly noticed when importing the agency_code field, which should be an integer (single digit), but ends up w/ some characters. it is only the 3rd field in the file, so that is foo bar. I am not sure exactly how to proceed, but since we have TS for 1990 and 1991, am considering just moving ahead w/o the panel for those 2 years.

[feomike]

2004 - 20010 LAR has been loaded from the national archives data; 2011 - 2014 LAR had previously been loaded with data directly from the FFIEC. NOTE FFIEC table format differs from the National Archives. this format is scripted along with Panel and TS (also different from FFIEC direct download). script forth coming.

[feomike]

found minor error in as_of_date field for 2004-2011. the data is fixed field, and this field is defined as an integer. when converting it to a number in sql, i was formatting it as only a single digit (e.g. '9') not four digits (e.g. '9999'), so the year date was incorrect. i am fixing this.

[feomike]

1990-2004 LAR is now loaded. 1990-1992 had a ton of non-ascii characters in it and took forever to deal with. what a hassle. but this task is done.