LMDB traversal cli - Githubissues

IntelLabs / matsciml

Open MatSci ML Toolkit is a framework for prototyping and scaling out deep learning models for materials discovery supporting widely used materials science datasets, and built on top of PyTorch Lightning, the Deep Graph Library, and PyTorch Geometric.

MIT License

155 stars 25 forks source link

LMDB traversal cli #301

Closed laserkelvin closed 1 month ago

laserkelvin commented 1 month ago

This PR adds a big QoL oriented CLI, which provides some high level functionality for inspecting LMDB datasets.

Adds a matsciml.datasets.lmdb_cli module, which houses a click-based interface with multiple commands that perform various LMDB inspection tasks
Updates pyproject.toml to install lmdb_cli as a "script", which allows you to access the CLI after installing matsciml simply by running lmdb_cli in the command line.
Accompanying documentaton

laserkelvin commented 1 month ago

So window size is used by the running average, so as you're iterating through the dataset it will do (by default) a running average of properties based on 10 of the last samples. It's different from just capping the number of samples to go through, because you might want to sweep through the data and look for outliers.