build a dataset that integrates better with other sources of data

[barbh1307]

• standardize RequestingAgency names
• add ORI and other data (see LEAIC)
• explore ways to consolidate records
• best format to store/retrieve data

[barbh1307]

is kaggle an option?

[barbh1307]

see https://towardsdatascience.com/the-best-format-to-save-pandas-data-414dca023e0d

QUESTION: which columns are categorical? NOTE: none of these are evenly distributed OriginatingFile: categories will grow each month, so dates will change off these three files

DISP_AllStatesAndTerritories_mmddyyyy.xlsx[ALL]
DISP_Shipments_Territories_mmddyyyy_mmddyyyy.xlsx[SHIPMENTS]
DISP_Shipments_Territories_mmddyyyy_mmddyyyy.xlsx[CANCELLATIONS]

StateAbbreviation: categorized by state; there are 59 possibilities based on

US Postal Service Publication 28 https://pe.usps.com/text/pub28/28apb.htm
(see Check_DISP_AllStatesAndTerritories.ipynb and Check_DISP_Shipments_Cancellations.ipynb

Item_FSG: categorized by federal supply group; there are 100 possibilites, not evenly distributed

see https://en.wikipedia.org/wiki/Federal_Stock_Number#External_links
and https://en.wikipedia.org/wiki/List_of_NATO_Supply_Classification_Groups to start tracking these down

AgencyType: rough guess of types of agencies based on station names, currently 13 categories

TO EXPLORE: feather/parquet? should we save with more catgories?

[barbh1307]

Two possible sources for ORI data:

• agencies.zip downloaded 2020-09-29
• Law Enforcement Agency Identifier Crosswalk filename ICPSR_35158-V2 downloaded 2020-10-04
  An explanation of ORI can be found here.

[barbh1307]

build a new repo called create_DLA_LESO_dataset for this work