Closed flowpoint closed 1 year ago
Might be better to split the PR into two, one for stack exchange (since that one needs more work) and one for the CLI and dataset config. Unless you wanna hold out this PR in a draft until the stackexchange is better tested
Sure, this is a draft. I am going to clean up and split the pr.
@flowpoint do you think it is okay to go ahead and merge this in so ppl can start using the interfaces and abstract classes?
@ncoop57 i hope this pr is now fixed to just updating the core classes, the stackexchange script will be its own pr then.
Looks good to me. Merging the PR.
I wasn't happy with passing the same args to every implementation, so moved those into a config where we can easily store options for constructing the datasets.
This also cleaned up the cli a little. Additionally i added the extraction code for StackExchange.
Not tested yet, but too early for serious testing. These implementations should still be considered as spikes.
commit msg: …onfig, add Stackexchange comment, user, post StackExchangeDoc, add 7zip extraction to StackExchangeProcessor