SE data dump processor #1

[vanga]

• Convert xml to json/parquet and stores them in temp_dir (parquet is preferred)
• Column names are modified to remove "@" at the beginning for convenience
• Transforms "Tags" column of posts table into a list
• Uses pandas "merge", "groupby" and "apply" features do one-one and one-many joins. one-many relationship leads to a nested list of dicts.
• Currently stores the output in temp_dir, we can add another arg for StackExchangeDataset to take outpu_dir also as input
• Does unzipping and xml-parquet conversion only if necessary, skips if the files are present in temp_dir
• Only limited relations and columns are being included right now.

[vanga]

1

[reshinthadithyan]

Hello Pradeep. Thanks for the PR. PRs should go to working branch not the main branch.

[flowpoint]

afaict, pyproject.toml is the recommended way of specifying python packaging information i think we should use it the requirements can be specified int the pyproject.toml file, so we dont need the requirements.txt