Closed reshinthadithyan closed 1 year ago
I'm having a hard time running the pytests. It keeps throwing a stats
module is not installed. Do I need to run things from a certain folder or install the module somehow? @reshinthadithyan
I'm having a hard time running the pytests. It keeps throwing a
stats
module is not installed. Do I need to run things from a certain folder or install the module somehow? @reshinthadithyan @ncoop57 I've fixed all the stats-related test bugs. But the existing ones are from codepile dataset subsets. I guess we should go ahead merge it. Given the fact that it isworking
or I can make a separate branch to work with. What do you think ?
STATS MODULE
The module acts as a single point of entry for all the stats related functions to run in large scale datasets. For our own terminology we see there are potentially two types of statistics,
Metadata based
- EDA components like distribution to be pivoted against the metadata. This is the most common type of stats that we see in all the dataset.Example : In stackexchange, we would like to find the distribution of datapoints amongst the different sites under different sites.
Document based
- NLP components like wordcloud, topic-modelling, language-detection, etc. These are the stats that are more specific to the document text and are necessarily applicable to all the datasets.Basic Structure
The whole module is split into two stages,
src/stat_module.py
- This script providesGetMeta
function and aStatistics
class. They extract the meta data and run the stats respectively.The
GetMeta
function is used to extract the metadata from the dataset and formats it to a json file to see the fields of the datasets. TheStatistics
module needs to be overidden inmap_fn
andmap_batch_fn
inorder to apply them to the stats of the dataset.src/viz.py
- This script provides theVisualize
class which is used to visualize the stats extracted from theStatistics
class.Various EDA components
src/utils.py
contains of the following selected EDA classes which are used to extract the stats from the dataset. Anystat-request
should be a class which has two functions,map_fn
- This function is used to extract the stats from the dataset. It takes in therow
of the dataset and returns the stats.map_batch_fn
- This function is used to aggregate the stats from themap_fn
and returns the aggregated stats over a batch.These two functions are based off the batching feature in the processing functions of the huggingface datasets library.