voxel51 / eta

ETA: Extensible Toolkit for Analytics
https://voxel51.com
Apache License 2.0
29 stars 13 forks source link

replace ndjson, patoollib with non-glp code #590

Closed nebulae closed 1 year ago

nebulae commented 1 year ago

changelog

replaced ndjson with jsonlines replaced patool with rarfile and py7zr

to note - jsonlines and rarfile are both python 3.6+, whereas the replaced patool and ndjson both support python 2.7.

ndjson test

import eta.core.serial as etas

import fiftyone as fo
import fiftyone.zoo as foz

ndjson_path = "/tmp/samples.ndjson"

dataset = foz.load_zoo_dataset("cifar10", split="test", max_samples=10)
objs = list(s.to_dict() for s in dataset)

etas.write_ndjson(objs, ndjson_path)

assert etas.read_ndjson(ndjson_path) == objs

with open(ndjson_path, "r") as f:
    assert etas.load_ndjson(f.read()) == objs

with open(ndjson_path, "rb") as f:
    assert etas.load_ndjson(f.read()) == objs

RAR test

import eta.core.utils as etau
import eta.core.web as etaw

url = "http://serre-lab.clps.brown.edu/wp-content/uploads/2013/10/test_train_splits.rar"
rar_path = "/tmp/test_train_splits.rar"

etaw.download_file(url, path=rar_path)

etau.extract_archive(rar_path)
ls -lah /tmp/