Exploratory data analysis

[denkii]

Prompt 2 datasets

• [x] account_000.txt
  • Expecting 23 cols, but line 858618 contains text after processing all cols.
  • Manually skipped in pandas and re-exported for R
• [x] activity_000.txt
• [ ] activityattributes_000.txt
  • Only 3 columns instead of 5 as per data dictionary
  • value and processed_date seem to be missing
• [x] activitytype_000.txt
• [x] activitytypeattributes_000.txt
• [x] product_000.txt
• [x] scans_000.txt
• [x] sqlcontact_v2.csv
• [ ] tickettransaction_000.txt
  • pandas loads fine, but errors with r
  • load in pandas and re-export for r

[heytheredli]

python command to remove all invalid rows for accounts_000.txt:

account_000 = pd.read_csv("account_000.txt", delimiter="|", skiprows = [858617,1041673,1149339,1217483, 1486057, 1486058, 1732138, 1744129, 1868116, 2007689, 2038301, 2082738, 2311444, 2358626, 2460007, 2460008, 2545168, 2572593, 2751962, 2878542, 2878543, 2892853], low_memory=False)