Create table(s) allowing query for uniqueness attributes for each dataset

[zzeppozz]

Start by comparing species and occurrence counts between all datasets. For each species in a dataset,

• is that the largest occ_count
• is that the only species
• what percentage/count of datasets contain that species

Create a dataframe/table with species rows and dataset columns, occ_count values Create on S3 with pandas, save in S3 (too big for redshift)

[zzeppozz]

Use a combination of pandas for stacked data and scipy.sparse for sparse aggregate data, download/upload to s3, read and write locally,

[zzeppozz]

Added queries, and tests to compare stacked data to sparse matrix