- Visit our [single-page web application](https://allotax.vercel.app/) to try it out online ([:octocat: github](https://github.com/jstonge/allotax))!
- Use [py-allotax](https://github.com/carterwward/py-allotax) to use allotaxonometry at scale ([:octocat: github](https://github.com/carterwward/py-allotax))
`Alloxonometer.js` is a lightweigt Javascript package for combining pairs of Zipfian ranked lists of components and provide [divergence metrics](https://compstorylab.org/allotaxonometry/papers/rank-turbulence-divergence/). The library performs the divergence calculations among two systems, which will typically be consumed by the following 2 tools to visualize results.
## Use cases and install
- Explore systems and formulate research questions: we recommend starting with the web app...
- Running multiple or larger scale analyses: we recommend using the py-allotax, which details its install instructions, details the required data format, and provides examples in its repo.
## CSV input
The allotaxonometer expects 2 tables in the following form:
| | types | counts | totalunique | probs |
|----|---------|----------|---------------|---------|
| 0 | John | 8502 | 1161 | 0.0766 |
| 1 | William | 7494 | 1161 | 0.0675 |
| 2 | James | 5097 | 1161 | 0.0459 |
| 3 | George | 4458 | 1161 | 0.0402 |
| 4 | Charles | 4061 | 1161 | 0.0366 |
## Paper data
#### Babynames data
The original babyname dataset for boys and girls can be found on the [catalog.data.gov](https://catalog.data.gov/dataset?tags=baby-names) website. But we use the dataset [here](http://pdodds.w3.uvm.edu/permanent-share/pocs-babynames.zip) to replicate the original paper. You can find a 5-years aggregated version used in the `Observable` version in `data/`. The original dataset includes each year from 1880–2018, which have 5 or more applications. You can convert the original folder into the formatted `.json` file using R with the following command:
```R
read_and_write_babyname_dat <- function(fname, gender) {
d <- readr::read_csv(fname,
col_names = c("types", "gender", "counts"),
col_select = c("types", "counts"),
col_types = c("c", "i"))
d$probs <- d$counts / sum(d$counts)
d$total_unique <- nrow(d)
return(d)
}
# You need to be in the folder above `data/`, which is the unzip folder contained in
# http://pdodds.w3.uvm.edu/permanent-share/pocs-babynames.zip
purrr::map(
list.files("data/", pattern = "names-boys*"),
~read_and_write_babyname_dat(paste("data", .x, sep = "/"), "boys")
)
purrr::map(
list.files("data/", pattern = "names-girls*"),
~read_and_write_babyname_dat(paste("data", .x, sep = "/"), "girls")
)
```
#### Twitter data
We access the Twitter data from the Comptuational Story Lab [storywrangling](https://gitlab.com/compstorylab/storywrangling)' API. Unfortunately, the API only work when you are connected on the University of Vermont's VPN. Follow the instructions [here](https://www.uvm.edu/it/kb/article/install-cisco-vpn/) to get the VPN working. Once this is done, run the following lines from the command line:
```shell
git clone https://gitlab.com/compstorylab/storywrangling.git
cd storywrangling
pip install -e .
```
Then from `python` you can get the top ngram count with rank data for any given day with the following:
```python
from storywrangling import Storywrangler
from datetime import datetime
import json
from pathlib import Path
def get_ngram(yr, month, day, fname=False):
storywrangler = Storywrangler()
ngram_zipf = storywrangler.get_zipf_dist(
date=datetime(yr, month, day),
lang="en", ngrams="1grams",
max_rank=10000, rt=False
).reset_index()\
.rename(columns={
"ngram":"types", "count":"counts", "count_no_rt":"counts_no_rt",
"rank":"rank", "rank_no_rt":"rank_no_rt", "freq":"probs", "freq_no_rt":"probs_no_rt"
})\
.dropna()\
.assign(totalunique = lambda x: x.shape[0])\
.loc[:, ["types", "counts", "totalunique", "probs"]]\
.to_dict(orient="index")
ngram_zipf = { f"{yr}_{month}_{day}": [_ for _ in ngram_zipf.values()] }
if fname:
if Path(fname).exists():
with open(fname) as f:
old_dat = json.load(f)
ngram_zipf.update(old_dat)
with open(fname, 'w') as f:
json.dump(ngram_zipf, f)
else:
return ngram_zipf
```
Note that this solution is a bit clunky. At some point we would prefer to have a sql DB that we can interact with.
#### Species Abundance Data
We access the species abundance data from https://datadryad.org/stash/dataset/doi:10.15146/5xcp-0d46, downloading the full dataset, unzipping it, and then loading bci.tree\